Human drivers "can" be way safer; they aren't always. There's likely some balance where overall, self driving is statistically safer than some group of suboptimal drivers. It remains to be seen but there's always hope.
Unless the unsafe parts of self driving only apply to previously unsafe drivers it will still struggle to take off.
Not every human driver has the same risk, but every self driving car will. (Or it will be based on which car you are in rather than how safe you are.) In other words, relatively safe human drivers could actually see their risk levels go up in a self driving car, even it if it statistically safer than all human drivers.
