There's a good reason for this. It's because the human can be interrogated into what was going through their mind whereas many ML models cannot. That means we can't ascertain if the ML accident is part of a latent issue that may rear its ugly head again (or in a slightly different manner) or just a one-off. That is the original point: a theory-of-mind is important to risk management. That means we will struggle to mitigate the risk if we don't "hyperanalyze" it.
You're missing the context. The AI didn't actually do anything unexpected, unless you expected it to try and drive through a downed branch. The AI behaved exactly as it should. The unexpected part was when the car behind the AI didn't see the branch and, therefore, didn't expect the AI car in front to stop. Unexpected doesn't mean wrong.
Cars can do unexpected things for good reasons, as the AI did in this case.
I'm taking in a larger context. I think just reading the three cited examples is an incorrect approach. For one, Waymo isn't sharing "all" their data, they've already been highlighted for bad practices in terms of only sharing the data from when their Waymo team decided was a bad decision. That's not necessarily objective, and can also lead to perverse incentives to obfuscate. So we don't have a great set of data to work with, because the data sharing requirements have not been well-defined or standardized. Secondly, if you look at reports of other accidents, you can see where AV developers have heinously poor practices as it relates to safety-critical software. Delaying actions as a mitigation for nuisance braking is really, really bad idea when you are delaying a potentially safety critical action. I'm not saying Waymo is bad in this regard, but we know other AV developers are and, when you combine that with the lack of confidence in the data and the previous questionable decisions around transparency, it should raise some questions.
There's a good reason for this. It's because the human can be interrogated into what was going through their mind whereas many ML models cannot. That means we can't ascertain if the ML accident is part of a latent issue that may rear its ugly head again (or in a slightly different manner) or just a one-off. That is the original point: a theory-of-mind is important to risk management. That means we will struggle to mitigate the risk if we don't "hyperanalyze" it.