It's because self-driving cars have alternatives, like automated trains or streetcars, that are safer and less resource intensive. Self-driving cars aren't just competing against human-driven cars. Unless you think cars are the only form of allowed infrastructure.
If a train or streetcar isn’t an effective alternative to a human-driven car (as is the case in the overwhelming majority of car trips in the US), how are they an effective alternative to a self-driving car?