Not OP, but here is my observations:
The llm are uniformly dumb and not "understaging" across all spectrum of topics. It is counter-intuitive. By asking llm to simply blab ("write a story about ..") you notice it:
- mixes up pronouns (who is "you" or "he")
- cannot keep track of what is where.
- continuously plugs it's guidance slant ("lets cook dinner, Bob! It is paramount to strive for safety and cooperation while doing it!")
— language style is all over the place, comically so.
— when asked about the text it just generated, is able to give valid critique to itself (i.e. having that "insight" does not help the generation)
Journalists may have shallow understanding of topic, but they do not start referring to a person they write about as "me" halfway through.
- mixes up pronouns (who is "you" or "he")
- cannot keep track of what is where.
- continuously plugs it's guidance slant ("lets cook dinner, Bob! It is paramount to strive for safety and cooperation while doing it!")
— language style is all over the place, comically so.
— when asked about the text it just generated, is able to give valid critique to itself (i.e. having that "insight" does not help the generation)
Journalists may have shallow understanding of topic, but they do not start referring to a person they write about as "me" halfway through.
LLM is uniformly dumb