Tokenisation explained: why a model cannot count the letters in strawberry

Ask a capable model how many times the letter r appears in strawberry and it will often get it wrong. The reason is not intelligence. It is that the model never saw the letters.

Tokenisation is the step that converts your text into the units a model actually processes, and it quietly explains a surprising number of otherwise baffling behaviours.

What a token is

Not a word and not a character. A token is a chunk of text learned from data, typically a common word, a word fragment, or a piece of punctuation.

The dominant method came from machine translation. Sennrich and colleagues adapted byte pair encoding to represent rare and unseen words as sequences of subword units, which solved the problem of vocabularies that could never cover every word.

Strawberry might be two tokens. The model is being asked to count letters inside symbols it cannot see inside.

Later work made the process language-agnostic, with SentencePiece treating raw text as a stream and removing the assumption that words are separated by spaces, which matters enormously for languages that do not do that.

What this explains

Odd behaviour	Tokenisation cause
Miscounting letters	Letters are inside tokens, not visible units
Struggling with reversed words	Requires character access it does not have
Poor arithmetic on long numbers	Digits get grouped inconsistently into tokens
Some languages cost more	Fewer characters per token outside English
Rhyming and puns are hit and miss	Sound is not represented at all

The cost row has real consequences. Text in languages underrepresented in the training corpus splits into more tokens, so the same paragraph costs more to process and consumes more of the context window.

That is a fairness issue as much as a pricing one, and it is invisible unless you are the one paying for it.

The same effect shows up with code, where indentation, punctuation and identifiers fragment heavily. A code file often costs more tokens than a prose document of the same length.

Why nobody just uses characters

The obvious fix is to feed the model raw characters, which would solve counting immediately. It is not done, for a reason that comes down to arithmetic.

Attention cost grows with the square of sequence length, a property of the architecture set out in Attention Is All You Need. Characters would make sequences roughly four times longer, and roughly sixteen times more expensive to attend over.

Words are the opposite extreme and fail differently, since a fixed vocabulary cannot contain every name, typo or new term. Subwords are the compromise that made large models affordable.

How the vocabulary gets built

The token list isn’t designed by anyone. It’s learned, by counting.

Byte pair encoding starts with individual characters and repeatedly merges whichever adjacent pair appears most often, until it hits a target vocabulary size. Common words end up as single tokens and rare ones stay split.

That’s why the vocabulary is a fossil of the training corpus. A word that was common in the data is cheap to process; one that wasn’t gets fragmented, and the model handles fragments less confidently.

It also means tokenisers differ between models, so token counts aren’t portable. The same document can cost noticeably more with one provider than another before any price difference is applied.

Scale didn’t remove the issue either. GPT-3 was trained on 300 billion tokens with a subword vocabulary, as the few-shot learning paper describes, and the counting problem survived every subsequent increase in size.

Tokens are the unit of everything else

Once you see tokens as the real currency, several other parts of the field line up.

Training budgets are counted in them. DeepMind’s Chinchilla work found that “for every doubling of model size the number of training tokens should also be doubled”, which is why data supply became a constraint.

Your bill is counted in them too, and prices span $6 to $30 per million output tokens across frontier providers.

Context windows are measured in them as well, which is why a stated window is a slightly slippery number. A million tokens is not a million words, and how many words it actually holds depends on your language and your content.

Our explainer on what a large window changes covers the practical side of that, including why capacity and usable attention are different things.

Working with it rather than against it

Don’t ask a model to do character-level work. Counting, reversing, checking spelling letter by letter and constructing acrostics are all tasks it is structurally bad at.

Give it a tool instead. Three lines of code count letters perfectly, and an agent that can call that function will always beat a model asked to do it from memory, which our guide to building agents covers.

For long numbers, separate the digits or hand the arithmetic to a calculator. The failure is in how the number was chunked, not in the model’s grasp of addition.

And when you estimate cost, count tokens rather than words. A rough rule for English is about four characters per token, and anything with code, tables or non-English text will run higher than you expect.

What would change this

Byte-level and character-aware architectures exist and are getting better, and they would remove this entire class of failure if the efficiency gap closed.

There’s a broader lesson worth taking from it, though. When a model fails at something that looks trivially easy, the cause is usually a representation choice rather than a gap in reasoning, and knowing which one you’re looking at decides whether prompting can help.

Position is the other representation problem worth knowing about. Stanford’s Lost in the Middle found accuracy degrades for information in the middle of a long input, which is a property of how tokens are attended to rather than of comprehension.

Until then the strawberry question stays a useful demonstration rather than a real weakness. It reveals the plumbing, and the plumbing is a compromise that made everything else possible.