Tokenisation

In chapter 1 the machine sometimes paused for a moment in the middle of a word. Here is why. The last two chapters lied to you, a little, for the sake of simplicity. Models do not predict the next word. They predict the next token, and a token is usually a piece of a word.

Why not words? Think about what "put a probability on every possible next word" would actually mean. English alone has something like a million words once you count the variants: run, runs, ran, running, runner, runners. Add names, brands, slang, typos, code, and every other language, and the list never ends. A model would need a number for each one, at every step, and it would meet words it had never seen and have nothing to say.

So go to the other extreme: predict the next letter. Now the list is tiny, a few dozen symbols, and nothing is ever missing. But a single letter carries almost no meaning. "t" tells you very little. You have to predict several letters in a row before anything resembling a thought appears, and the model has to hold a much longer history to keep track of what it is in the middle of spelling.

The trick scientists settled on sits in between: cut text into sub-words. Common words stay whole, so "the" and "chased" are one token each. Less common ones are split into pieces that recur across the language: "unbelievable" becomes something like "un" + "believ" + "able", and "tokenisation" becomes "token" + "isation". Names, rare words, and other languages break into smaller fragments still, sometimes down to single characters. The pieces are chosen automatically by counting which chunks appear most often in a large body of text, so the vocabulary is a few tens of thousands of fragments that between them can spell anything. Every chunk is large enough to mean something, and nothing is ever missing from the list.

This matters in practice. Costs and limits are counted in tokens, not words. A model's occasional trouble with spelling, counting letters, or arithmetic often traces back to the fact that it never sees letters or digits directly, only the fragments it was given. From here on, when this course says "the next word", read "the next token".