Your first paragraph is exactly how I feel about nuclear weapons, to put it into context. I don’t think the logical conclusion from that viewpoint is that nuclear weapons aren’t that dangerous so we should just move ahead.
I feel obligated to point out that nobody has argued for nuking datacenters; the most radical AI existential-safety advocates have argued for is "have a ban on advanced training programs, enforced with escalating measures from economic sanctions and embargos to, yes, war and bombing datacenters". Not that anybody is optimistic on that idea working.
I think it has been empirically demonstrated that lapses in regards to the control and use of nuclear weapons can occur without the destruction of humanity.
(I am not an AI doomer, nor do I feel that nuclear weapons are not dangerous/should be less controlled)
