This article addresses the misconception that arises when someone lacks a clear understanding of the underlying mathematics of neural networks and mistakenly believes they are a magical solution capable of solving every problem. While neural networks are powerful tools, using them effectively requires knowledge and experience to determine when they are appropriate and when alternative approaches are better suited.
This does not apply to PINNs though. They were used and investigated by people deeply knowledgeable about numerics and neural networks, they just totally failed to live up to expectation.
"they just totally failed to live up to expectation"
Because the expectation was too high. If you are aiming for precision, neural networks might not be the best solution for you. That is why generative AI works so well, it doesn’t need to be extremely precise. On the other hand you don't see people use neural networks in system control for cricital processes.
I think that while the mathematics of neural networks are clearly completely understood we do not really understand why neural networks behave the way that they do when combined with large amounts of real world data.
In particular the ability of auto regressive transformer based networks to produce sequences speech while being immutable still shocks me whenever I think about it. Of course, this says as much about what we think of ourselves and other humans as it does about the matrices. I also think that the weather forcasting networks are quite shocking, the compression that they have achieved in modeling the physical system that produces weather is frankly.... wrong... but it obviously does actually work.
You can represent many things with numbers and build an algorithm that does stuff. ML techniques are formulas where some specifics constants are not known yet, so you go through a training phase to find them.
While combinations of words are infinite, only some makes sense. So there’s a lot of reccurent patterns there. When you take a huge datasets like most of the internet and digital documents. I would be more surprised if the trained model where incapable of producing correct texts as both the it’s overfitted to the grammar and the lexicon. And I believe it’s overfitted to general conversation patterns.
There is a lot of retrieval in the behaviours of LLM's, but I find it hard to characterize it as overfitted. For example, ask ChatGPT to respond to your questions with grammatically incorrect answers.
