If you tediously work out the LLM math by hand, is the pen and paper conscious too?

Consciousness is not computation. You need something else.

The pen and paper are not the actual substrate of entropy reduction, so not really.

Consciousness is what it "feels like" when a part of the universe is engaged in local entropy reduction. You heard it here first, folks!

This comment here is pure gold. I love it.

On the flip side: If you do that, YOU are conscious and intelligent.

Would it mean that the machine that did the computation became conscious when it did it?

What is consciousness?

Even if they do, it can only be transiently during the inference process. Unlike a brain that is constantly undergoing dynamic electrochemical processes, an LLM is just an inert pile of data except when the model is being executed.