The issue with AI isn’t that it simply gets things wrong — as is frequently pointed out, so do humans. The issue is that it gets things wrong in a way that comes out of nowhere and doesn’t even have a post-rationalised explanation.
The big claim about AI systems (especially LLMs) is that they can generalise, but in reality the ‘zone of possible generalisation’ is quite small. They overfit their training data and when presented with input out of distribution they choke. The only reason anyone is amazed by the power of LLMs is because the training set is unimaginably huge.
In fifty years we’ll have systems that make this stuff look as much like ‘AI’ as, say, Djikstra’s algorithm does now.
I first noticed this with Watson (IBM's language processing system) when it played Jeopardy!: when it was right, it was spot on and usually faster than the human contestants; but when it was wrong, it was way, way off base.
Part of that has to do with the fact that language is not the same for an LLM as it is for a person. If I say to you the sentence "The cat sat on the mat", that will evoke a picture, at the very least an abstract sketch, in your mind based on prior experience of cats, mats, and the sitting thereupon. Even aphantasic people will be able to map utterances to aspects of their experience in ways that allow them to judge whether something makes sense. A phrase like "colorless green dreams sleep furiously" is arrant nonsense to just about everybody.
But LLMs have no experiences. Utterances are tokens with statistical information about how they relate to one another. Nodes in a graph with weighted edges or something. If you say to an LLM "Explain to me how colorless green dreams can sleep furiously", it might respond with "Certainly! Dreams come in a variety of colors, including green and colorless..."
I've always found Searle's argument in the Chinese Room thought experiment fascinating, if wrong; my traditional response to it was "the man in the room does not understand Chinese, but the algorithm he's running might". I've been revisiting this thought experiment recently, and think Searle may have been less wrong than I'd first guessed. At a minimum, we can say that we do not yet have an algorithm that can understand Chinese (or English) the way we understand Chinese (or English).
The issue with AI isn’t that it simply gets things wrong — as is frequently pointed out, so do humans. The issue is that it gets things wrong in a way that comes out of nowhere and doesn’t even have a post-rationalised explanation.
The big claim about AI systems (especially LLMs) is that they can generalise, but in reality the ‘zone of possible generalisation’ is quite small. They overfit their training data and when presented with input out of distribution they choke. The only reason anyone is amazed by the power of LLMs is because the training set is unimaginably huge.
In fifty years we’ll have systems that make this stuff look as much like ‘AI’ as, say, Djikstra’s algorithm does now.