I'm not sure. If you see what they're doing with feedback already in code generation. The LLM makes a "hallucination", generates the wrong idea, then tests its code only to find out it doesn't compile. And goes on to change its idea, and try again.
We seem to be talking past one another. All I was talking about was the facts of how these systems perform, without any reverence about it at all.
But to your point, I do see a lot of people very emotionally and psychologically committed to pointing out how deeply magical humans are, and how impossible we are to replicate in silicon. We have a religion about ourselves; we truly do have main character syndrome. It's why we mistakenly thought the earth was at the center of the universe for eons. But even with that disproved, our self-importance remains boundless.
> I do see a lot of people very emotionally and psychologically committed to pointing out how deeply magical humans are, and how impossible we are to replicate in silicon.
This a straw man, the question isn't if this is possible or not (this is an open question), it's about whether or not we are already here, and the answer is pretty straightforward: no we aren't. (And the current technology isn't going to bring us anywhere near that)
Some machines, maybe. But attention-based LLMs aren't these machines.