> I tried a great deal of ML approaches, only to find recently that by better accounting for the optical specificities of the problem I was trying to solve
I want to point out, that any serious machine learning researcher is not oblivious to this, despite the deep learning boom suggesting to the contrary. Modern methods have shown that we are capable of building predictors with surprisingly complex representations, that can solve large-scale downstream tasks. i.e. our models are "flexible" enough.
The next challenge is whether they favor the "right kind" of solutions. For instance, Convolutional Neural Networks (CNNs) are architecturally just sparse version of Fully-Connected Neural Networks. Why is it then that CNNs perform far better on images? A key reason is that "inductive biases" afforded by MLP aren't strongly favored towards images. Another instance of this is the covariance functions used is Gaussian Processes - the Squared Exponential Kernel is very flexible and can in principle fit anything possible. Nevertheless, if the problem has specific structures, say periodicity, one better use the Periodic Kernel because it's inductive biases rightly align with the kind of solutions we expect.
> The lesson is : physics beats AI.
As a consequence, the single biggest reason physics would beat a generic AI in the short-term is precisely due to our ability to explicitly provide inductive biases that align with our expectations from the physical system.
We haven't found the secret sauce for every possible system in the universe. I don't think we can, either. But what we can do is devise ways to "control" such inductive biases we can encode in machine learning systems, which align with our expectations of the way the system should behave.
I want to point out, that any serious machine learning researcher is not oblivious to this, despite the deep learning boom suggesting to the contrary. Modern methods have shown that we are capable of building predictors with surprisingly complex representations, that can solve large-scale downstream tasks. i.e. our models are "flexible" enough.
The next challenge is whether they favor the "right kind" of solutions. For instance, Convolutional Neural Networks (CNNs) are architecturally just sparse version of Fully-Connected Neural Networks. Why is it then that CNNs perform far better on images? A key reason is that "inductive biases" afforded by MLP aren't strongly favored towards images. Another instance of this is the covariance functions used is Gaussian Processes - the Squared Exponential Kernel is very flexible and can in principle fit anything possible. Nevertheless, if the problem has specific structures, say periodicity, one better use the Periodic Kernel because it's inductive biases rightly align with the kind of solutions we expect.
> The lesson is : physics beats AI.
As a consequence, the single biggest reason physics would beat a generic AI in the short-term is precisely due to our ability to explicitly provide inductive biases that align with our expectations from the physical system.
We haven't found the secret sauce for every possible system in the universe. I don't think we can, either. But what we can do is devise ways to "control" such inductive biases we can encode in machine learning systems, which align with our expectations of the way the system should behave.