One of the reasons is because to ChatGPT, each word is a token. It only knows ho... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		cultureswitch on May 20, 2023 \| parent \| context \| favorite \| on: Don Knuth plays with ChatGPT One of the reasons is because to ChatGPT, each word is a token. It only knows how many letters are in a word because it has been trained to know that about each word individually.

rvnx on May 20, 2023 [–]

You can see it by yourself here if you are interested: https://platform.openai.com/tokenizer

The tokens are usually not matching the length of a word

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact