That is a fair point, this is a similar issue that Libre-boot went through a few years back. Yes, you try to stick clear of binary blobs as much as possible but at a certain point you just run out of hardware that meets that criteria.
I think GrapheneOS focuses on privacy and security, not liberation. I think their pragmatic and narrow-minded approach is valid, it's important not to conflate their scope with related issues they are unable/unwilling to tackle.
Personally, I think a usable pure Linux phone is required to weaken the desktop vs. mobile distinction and break the lock-in. This would additionally empower the desktop platform, confirm it as baseline.