At first I was shocked to hear this, but then I realized it's basically a substitution cipher.

It's not even a cipher, it's just an encoding. Nobody would be surprised if chatgpt would be able to also understand e.g. utf-16.

What's interesting here is that base64 does not have the exact same byte sequence (such as a single space) between tokens. So I still think for it to understand base64 is a bit surprising.

It's the same for every trigram... maybe that makes it easier for it? It could probably memorise the most common trigrams.

If that's the case it would be less accurate for encoding random letters than English.

Yeah, still a bit strange given that GPT4 doesn’t even see individual characters but only tokens.