Yeah it's really not, it's about trying to keep the housing market about housing the local population rather than allowing in investor funds from overseas which warps it, changes what gets built at what price, and drives people into rental rather than ownership.
If you don't have the right to live somewhere, why should you be competing in the housing market and pricing out the local population?
Particularly if the nature of your competition is to keep properties mostly empty while speculating on price (see, for example, a lot of the high-end apartment development in London).
It's not xenophobia, it's a fundamental disagreement in what housing is actually for.
Not everyone agrees that property rights are absolute or supreme. Some of us think markets should be set up to help people, not the other way around, particularly where human needs like housing are involved.
It's certainly not clear that a preference for housing people over capital is 'xenophobic'.
