On one hand, we don't have any idea what consciousness is or how it happens. For all we know, putting a ton of numbers onto a graphics card and doing matrix math on them is enough to make it.
On the other hand, this really feels like getting freaked out about seeing a realistic photo of a person for the first time, because it looks so much like a person, or hearing a recording of someone speaking for the first time because it sounds like they're really there. They're reproductions of a person, but they are not the person. Likewise, LLMs seem to me to be reproductions of thought, but they are not actually thought.
Reproductions of the product of thought, more like it.
I assume pretty much everyone here knows the gist of how LLMs work? "Based on these previous tokens, predict the next token, then recurse." The result is fascinating and often useful. I'm even willing to admit the possibility that human verbal output is the result of a somewhat similar process, though I doubt it.
But somehow, even highly educated/accomplished people in the field start talking about consciousness and get all spun up about how the model output some text supposedly telling you about its feelings or how it's going to kill everyone or whatever. Even though some basic undergraduate-level[0] philosophy of mind, or just common human experience, feels like it should be enough to poke holes in this.
[0] Not that I care that much for academic philosophy, but it does feel like it gives you some basic shit-from-shinola filters useful here...
I'm a functionalist, to me a complete "reproductions of the product of thought", as you beautifully put it, is enough to prove consciousness. LLMs are not there yet, though.
On the other hand, this really feels like getting freaked out about seeing a realistic photo of a person for the first time, because it looks so much like a person, or hearing a recording of someone speaking for the first time because it sounds like they're really there. They're reproductions of a person, but they are not the person. Likewise, LLMs seem to me to be reproductions of thought, but they are not actually thought.