> This doesn't seem to happen very often in classical programming, does it?

Not really, no. The only counterexample I can think of is chess programs (before they started using ML/AI themselves), where the search tree was so deep that it was generally impossible to explain "why" a program made a given move, even though every part of it had been programmed conventionally by hand.

But I don't think it's particularly unusual for technology in general. Humans could make fires for thousands of years before we could explain how they work.