It will start in cities with people declining to purchase new cars since they can just take a cheap self-driving car everywhere. It's true that at sufficiently low population density it would not makes sense to rely on a taxi service even if most people were using it, but I think that is only in very rural areas. 90% of the Americans live in an region where, if they all used a self-driving taxi service, the experience would be better than everyone owning a car. The hard economic benefits of not having to worry about maintenance, parking, or the up-front capital costs at purchase (huge!) will steam roll the romantic aspects of owning a personal car.
I'm including Uber and Lyft in "taxi", i.e., driving service on demand. Replacing a personal car with Uber and Lyft is too expensive for most people, and the basic economic reason is that a human driver must be paid. But in a few years self-driving taxis will be cheaper than taxis with human drivers. At that point, a larger and larger share of the population will stop buying cars and will just take self-driving taxis.