Afaik, VoIP is just the easiest onramp to spoofing numbers.

Anyone with BGP-equivalent access to the phone network can spoof numbers, even if they're coming from a landline. Might even be able to when you have a business landline terminated in a PBX.

i.e. the phone network backend is built on trust.