>> Most of the progress has been the result of people trying to model what's
going on in our heads: convnets modeling our vision system, rnns modeling
feedback loops, reinforcement learning modeling sparse reward signals,
attention based models modeling, well, attention.
The problem with all those advances is that they all happened more than 20
years ago. My comment discusses the state of machine learning research right
now, which is that there are very few new ideas and the majority of the field
doesn't have a clear direction.
Note also that the advances you describe were not "guided by theory". They
were inspired by ideas about how the mind works. But, finding ideas to try
is not the scientific process I describe above. And just because you're
inspired by an idea doesn't mean your work is in any way a proof or disproof
of that idea. For example, CNNs were not created in an effort to demonsrate
the accuracy of a certain model of the visual cortex. In fact, Yan LeCun is on
record saying that deep learning is nothing like the brain:
Yann LeCun: My least favorite description [of deep learning] is, “It works
just like the brain.” I don’t like people saying this because, while Deep
Learning gets an inspiration from biology, it’s very, very far from what the
brain actually does.
>> Using your analogy, GPT-3 is more like if you devised an algorithm which
produces n + k of pi digits after processing n pi digits - without knowing
anything about how to compute pi, or what pi is.
That's not a good example. GPT-3 can't actually do this. In fact, no
technology we know of can do this for a k sufficiently large and with accuracy
better than chance. Personally I find GPT-3's text generation very
underwhelming and not anywhere near the magickal guessing machine your example
seems to describe.
The problem with all those advances is that they all happened more than 20 years ago. My comment discusses the state of machine learning research right now, which is that there are very few new ideas and the majority of the field doesn't have a clear direction.
Note also that the advances you describe were not "guided by theory". They were inspired by ideas about how the mind works. But, finding ideas to try is not the scientific process I describe above. And just because you're inspired by an idea doesn't mean your work is in any way a proof or disproof of that idea. For example, CNNs were not created in an effort to demonsrate the accuracy of a certain model of the visual cortex. In fact, Yan LeCun is on record saying that deep learning is nothing like the brain:
Yann LeCun: My least favorite description [of deep learning] is, “It works just like the brain.” I don’t like people saying this because, while Deep Learning gets an inspiration from biology, it’s very, very far from what the brain actually does.
https://spectrum.ieee.org/automaton/artificial-intelligence/...
>> Using your analogy, GPT-3 is more like if you devised an algorithm which produces n + k of pi digits after processing n pi digits - without knowing anything about how to compute pi, or what pi is.
That's not a good example. GPT-3 can't actually do this. In fact, no technology we know of can do this for a k sufficiently large and with accuracy better than chance. Personally I find GPT-3's text generation very underwhelming and not anywhere near the magickal guessing machine your example seems to describe.