I'm sure you're already familiar with the ELIZA effect [0], but you should be a bit skeptical of what you are seeing with your eyes, especially when it comes to language. Humans have an incredible weakness to be tricked by language.
You should be doubly skeptically ever since RLHF has become standard as the model has literally been optimized to give you answers you find most pleasing.
The best way to measure of course is with evaluations, and I have done professional LLM model evaluation work for about 2 years. I've seen (and written) tons of evals and they both impress me and inform my skepticism about the limitations of LLMs. I've also seen countless times where people are convinced "with their eyes" they've found a prompt trick that improves the results, only to be shown that this doesn't pan out when run on a full eval suite.
As an aside: What's fascinating is that it seems our visual system is much more skeptical, an eyeball being slightly off created by a diffusion model will immediately set off alarms where enough clever word play from an LLM will make us drop our guard.
We get around this a bit when using it to write code since we have unit tests and can verify that it's making correct changes and adhering to an architecture. It has truly become much more capable in the last year. This technology is so flexible that it can be used in ways no eval will ever touch and still perform well. You can't just rely on what the labs say about it, you have to USE it.
Interesting observation about the visual system. Truth be told, we get the visual feedback about the world at a much higher data rat AND the visual about the world is usually much higher correlated with reality, whereas the language is a virtual byproduct of cognition and communication.
No one understands how LLMs work. But some people manage to delude themselves into thinking that they do.
One key thing that people prefer not to think about is that LLMs aren't created by humans. They are created by an inhuman optimization algorithm that humans have learned to invoke and feed with data and computation.
Humans have a say in what it does and how, but "a say" is about the extent of it. The rest is a black box - incomprehensible products of a poorly understood mathematical process. The kind of thing you have to research just to get some small glimpses of how it does what it does.
Expecting those humans to understand how LLMs work is a bit like expecting a woman to know how humans work because she made a human once.
But my opinions about what LLMs can do are based on... what LLMs can do. What I can see them doing. With my eyes.
The right answer to the question "What can LLMs do?" is... looking... at what LLMs can do.