No. I am not saying it’s the “industry” as much as the context of risk. That's why it doesn't matter if the analogy is Tesla or some other safety-critical manufacturer. To be clearer: how many F9 launches have carried humans?

Now go look at the history of Shuttle for the equivalent number of launches at that risk level. Would you claim they are equivalent in terms of human-rated safety?

If not, it’s only because you have the benefit of knowing the long-tail probabilities with the Shuttle.

>It's the same industry. Building the same type of product.

By extension of your logic, Starship should then already have the same launch reliability as F9. So either this is an example of a low-probability event, or your logic is flawed.

There were 135 space shuttle launches, of which 2 failed. There have been 162 launches of the F9 block 5 with 0 failures. Why do you think we have more knowledge of the long-tail probability of the Shuttle than F9?

True, few of those F9 missions were crewed, but that's the point. There's no difference between a crewed and a noncrewed F9 launch vehicle, so there's no reason to think the presence of humans would change the risk. So, you get to accumulate most of that reliability data without putting people at risk doing so.

>Why do you think we have more knowledge of the long-tail probability of the Shuttle than F9?

Because the nature of the two programs was fundamentally different. The Shuttle was a product contract, while CCP is a service contract. On the former, the govt has much more control, and will detail more rigorous acceptance criteria. This generally gives a much higher pedigree on quality control. On the latter, they take a much more hands off approach and have limited insight.

As an analogy, imagine you are making a big bet on acquiring a software company. One company gives you their source code, shows you all their most recent static analysis, unit test results, allows you to interview their programmers etc. The other company allows you none of that, but gives you a chance to play around on their website to see for yourself. Both systems seem to work when you try the end product, but which do you have higher confidence in?

At the end of the day, "reliability" is just a measure of how much confidence we have that a product will do what we ask of it, when we ask.

Are you asserting that we gain better insight into the reliability of a system by thinking about it deeply rather than by observing it perform its function? Because I don't believe that for a minute.

I'm not saying you can get by without thinking, but it's difficult for humans to estimate the reliability of a complicated system. Reality, though, has no problem doing it.

Plus, I think your analogy is flawed. NASA surely has a more hands-off approach on the CCP than on the Shuttle, but to say it's hands-off is misleading. They do have a lot of access.

No, I'm not saying by "thinking about it" (although that has its place). Everything I listed is a form of testing. But there's a distinction between iterative testing at a lower level, and end-to-end testing. Again, both have their place.

Take the example of the F9 strut failure. They could have tested the material outside of the final test configuration and saved themselves a lot of trouble. They chose to forego that testing, and instead 'tested ' it as part of part of their launch configuration. (I put it in quotes because it's not clear to me that this was a conscious testing decision).

There’s also a difference between “we’re not completely sure of the fundamental principles, but our testing indicates it works” and “our testing indicates it works and we have a solid understanding why”. The latter allows you to know the limits of your application much more readily. The risk in the former is that you don’t know what you don’t know, so you can never be wholly sure if you’re good or lucky. And luck can be fickle. And this is also where rapid iteration can lead to issues: the more you change, the less sure you can be about whether your results are attributable to luck.

>Plus, I think your analogy is flawed. NASA surely has a more hands-off approach on the CCP than on the Shuttle, but to say it's hands-off is misleading. They do have a lot of access.

They have many engineers who want more access and are effectively told to back off because it's not their place in this type of contract. So I'm not saying they have no access, I'm saying they have very limited access by comparison. It would have been better if I made the analogy that they get the results to a small select number of tests, but not all.