I'm not here to defend unnecessary zoning restrictions, but your argument here is that the right to not be affected by the externalities of someone else's decisions should be limited to the uber-wealthy?.
It's completely reasonable to believe that a given person should be capable of quiet enjoyment of their property without having to purchase everything remotely close to it.
If you want to live in a very desirable area and you also want to have very few neighbors, you should have to pay for that luxury. Right now that luxury is effectively subsidized for a few people who are “grandfathered in”, thanks to the market distortion created by zoning.
It's completely reasonable to believe that a given person should be capable of quiet enjoyment of their property without having to purchase everything remotely close to it.