In a sense, the neural network structure is the "hardware" of the LLM; and the weights are the "software". But rather than explicitly writing a program, as we do with normal computers, we use the magic of gradient descent to summon a program from the mathematical ether.

Put that way, it should be clearer why the AI doomers are so worried: if you don't know how it works, how do you know it doesn't have malign, or at least incompatible, intentions? Understanding how these "summoned" programs work is critical to trusting them; which is a major reason why Anthropic has been investing so much time in this research.