I'm failing to connect the anecdotes with the conclusion. He's claiming the the ML is scaling well but then gives the data on how GPT-3 is expensively brute-forcing its way to "success".
To me it just seems like what supercomputing is to normal computing: It makes the computationally expensive stuff do-able in a reasonable amount of time, or gives diminishing returns on existing algorithms. But it doesn't magic in any real advancements.
The problem in AI/ML and the concept of "AI winter" to me was always the barrier of the fact that we're just doing predictions, with no deep meaning or comprehension of features. The layman thinks there's magic, but there's not, so when that truth is revealed there will be problems. There's nothing intelligent about our Artificial Intelligence; we're just doing statistics with big data and added sprinkles. OpenAI just proved they could do statistics with even bigger data and more expensive sprinkles.
Has their work really shown we can get past that core problem? Personally, I don't see it.
Well, we don’t really know. I think that’s one element missing from most DL analysis: our understanding of the brain is incredibly limited. And ANNs are incredibly crude imitations of actual NNs. So we’ve built massive, extremely crude approximations of a system we don’t really understand.
As is often the case, the truth is somewhere in the middle. It’s almost certain that we won’t reach AGI without a fundamental breakthrough on soft/wet-ware but it’s also nearly certain that even with the best algorithms we will need to efficiently harness and coordinate massive compute power, as we’re learning to do now.
But thats the point, our brain is clearly a lot more than just a big table of probabilities. You only need to look at the absolutely insane volume of data and training time that these models need. How much time does it take a for human to understand a concept like "love" and what volume of training data is required? Computers would just regurgitate quotes from poetry or novels about love without any real understanding after billions of hours of training time and ingesting every document on the internet. A human can understand love in a tiny fraction of the time and with a tiny fraction of the volume of information processed and they also understand it in a more fundamental way and can articulate that in a way these models cannot.
You might argue back, well the human brain has pre-trained neural networks with billions of hours of training time. Well, that isn't really the case. We don't start off with some pre-existing memory of what "love" means, or what "Physics" is, or trillions of bytes of data. All we have is a capacity to learn which is highly efficient, a conscious mind which is aware of itself, and certain fundamental drives driven by our bodies and instincts. If you have a human child and give it zero input information it will never learn a language or be capable at all in any sense of the term. So we become incredibly capable based on a tiny fraction of input data fed into us after birth.
The way the human brain and mind works is deeply tied in to the experience of having a body, knowing we are mortal, and having fundamental drives such as a drive to survive, eat, drink, keep ourselves safe, and also a drive to be social, find a mate, and procreate. I would argue that we will never be able to have a computer/algorithm that thinks like we do unless it also has drives like we do and a body like we do, since so much of our process of thinking is tied in to having a body, our awareness of mortality, and our basic human drives and experience.
Obviously the above is contrived and abstracted, but you get my point. If I took a little bit of time, I can schematically map every word to makes up the definition of love, and how they interact. Then I can associate 3D, real world graphical observations to each of those words and then love as a concept holistically (as humans do, we're not just confined to text data, we observe extremely rich 3D visual data, and audio data, and touch data, etc...). There's no reason to believe a massive "correlation machine" can't do the same thing with the right algorithms, enough compute power, and multimodal inputs. Furthermore, we can make the correlation machine even better by specializing parts of the hardware for certain tasks, just like the brain.
And it still wouldn't be the same. Again, our notion of love is tied into having consciousness, which machines would not have. We still don't even understand what consciousness is, how to define it, or how it is generated in the brain. While machines are not consciousness they could never "understand" or "experience" what we call "love" because love again is tied up with our experience of consciousness, the idea of mortality, and having a physical presence in the world.
What is consciousness, and what unique non-tautological properties does it give us?
I know of the general idea of consciousness, but I can’t boil it down to first principles. Self-awareness, on the other hand, is more tangible. AI would seem capable of internal cognition, reflection on past experiences, etc...They might not have the desire or need for such reflection, but they would certainly have the ability.
That's possibly true at the 'implementation' layer, but human learning is quite different, or at elast has some aspects that are vastly different. Human manuals don't contain millions of tagged examples, they contain explanations and a few dozen examples.
So sure, when we learn how to walk or see or read, we may be in this mode of simply discerning patterns in large amounts of data. But when we learn maths or programming, we are using a completely different kind of learning.
That comparison is like comparing gardening my backyard to agricultural farming. Technically the core concept is similar, but I think we both know that the scale and depth makes them not at all the same.
There's a continuity of small changes between your backyard and industrial farming. Just like there's a continuity of small changes between you and your cat.
For instance, there is no continuity of small changes in height that will bring you from climbing up incrementally larger trees to climbing to the moon.
The question is whether gpt-3 vs human intelligence is more like climbing a tree vs climbing a mountain or more like climbing a tree vs building a rocket.
I'm becoming increasingly convinced that Artificial Intelligence will fail not because we can't do the "artificial" part but because there is so little we do that is "intelligent".
> the fact that we're just doing predictions, with no deep meaning or comprehension of features
To me it just seems like what supercomputing is to normal computing: It makes the computationally expensive stuff do-able in a reasonable amount of time, or gives diminishing returns on existing algorithms. But it doesn't magic in any real advancements.
The problem in AI/ML and the concept of "AI winter" to me was always the barrier of the fact that we're just doing predictions, with no deep meaning or comprehension of features. The layman thinks there's magic, but there's not, so when that truth is revealed there will be problems. There's nothing intelligent about our Artificial Intelligence; we're just doing statistics with big data and added sprinkles. OpenAI just proved they could do statistics with even bigger data and more expensive sprinkles.
Has their work really shown we can get past that core problem? Personally, I don't see it.