How about this alternative challenge: ask it to write a poem in IPA (pronunciation language). I’d be surprised if this has ever been done pre-LLM, yet it excels at weird tasks like this.

You could probably just ask it to come up with 100 tasks to prove it’s not a stochastic parrot.

Yeah, I think "stochastic parrot" is a crappy phrase that obscures the mechanics of the LLM. Of course the LLM is capable of producing novel outputs, for some definition of novel. My only position here is that we can take any apparently magical outputs of the thing and, based on an understanding of how LLMs work, understand how they were likely produced. I think that sort of literacy will take us a long way.