Context windows explained: what a million tokens actually changes

Context windows went from a few thousand tokens to over a million in a handful of years. It’s the most quoted number in every model announcement and one of the most misunderstood, because the advertised window and the usable one are different figures.

Knowing where the gap sits changes how you write prompts and what you can safely build.

What the window physically is

The context window is everything the model can see at once: your instructions, the conversation so far, any documents you pasted, and the response being generated. All of it competes for the same budget.

It’s measured in tokens rather than words. A token averages roughly four characters of English, so a thousand tokens is about 750 words. Code, unusual names and non-English text tokenise less efficiently, sometimes much less.

A chatbot appears to remember your conversation only because the entire history is resent with every message. Nothing persists outside the window.

Why it was hard to make bigger

The constraint comes from the architecture. In a standard transformer, described in Attention Is All You Need, every token attends to every other token.

That means computation grows with the square of the sequence length. Double the context and you roughly quadruple the work, which is why early models capped out where they did.

Getting to a million tokens took a stack of engineering: attention implementations that avoid materialising the full attention matrix, positional encodings that extrapolate beyond training length, and sparse patterns where not every token attends to every other.

Meta’s Llama 3 flagship, “a dense Transformer with 405B parameters and a context window of up to 128K tokens”, shows where a conventional dense design landed.

Sparse architectures took a different route to the same goal. The Switch Transformer work showed models that “select different parameters for each incoming example”, keeping compute per token flat while capacity grows, which is the same instinct applied to width rather than length.

Advertised is not usable

Here’s the part that matters practically. A model advertised at a million tokens accepts a million tokens without error. Whether it reliably uses information from the middle of that million is a separate question, and the answer is often no.

Nelson Liu and colleagues at Stanford measured this in Lost in the Middle, finding that “performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts”.

Task	How it holds up at length
Find one fact at the start or end	Reliable
Find one fact in the middle	Measurably worse
Combine several facts scattered throughout	Degrades considerably faster
Summarise the whole thing faithfully	Hardest of all

That last row is the one to watch, because most benchmarks test the first. A model that finds one needle at 500,000 tokens may still fail to relate three needles at 100,000.

Three things that follow for your prompts

Put instructions at the end. If you’re pasting a long document, the document goes first and the question after it, because an instruction buried above 50,000 words sits in the weakest position available.

Don’t pad the context because you can. Filling a window with marginally relevant material makes the relevant material harder to find, and you pay for every token. More context is not free accuracy, a point our prompting guide develops further.

Test at your real length. A prompt that works beautifully at 5,000 tokens tells you nothing about its behaviour at 300,000, so if production is long then your evaluations have to be long too.

What long context genuinely replaces

It eliminates real work. Analysing a single large document used to require chunking it, embedding the chunks and retrieving the relevant ones. Now you paste the document.

Retrieval can’t fail if there’s no retrieval step, and the model sees how the parts relate rather than a handful of fragments. Our piece on when RAG is worth the complexity covers the other side of that trade.

Prompt caching amplifies the effect. Most providers let you cache a stable prefix, so repeated queries against the same document pay full price once, which changes the economics of pasting everything.

What it doesn’t replace

It’s not a knowledge base. If your corpus is ten million documents, no window holds it, and you still need retrieval to select candidates.

It’s not persistent memory either. Every request starts empty, so anything the model should know across sessions has to be stored by you and resent.

That distinction trips up a lot of agent designs. A long window makes any single task richer, and it does nothing to help a system remember what it learned yesterday, which is a separate problem needing separate storage.

And it isn’t free. You’re billed per input token and latency scales with length, so at current token prices a million-token request is expensive enough to need a specific justification.

Why the numbers kept climbing anyway

Context length became a marketing axis because it’s a single comparable number, and single comparable numbers sell.

It also rode the same compute curve as everything else. Epoch AI tracks frontier training compute growing 5x per year since 2020, and longer contexts are one of the things that abundance bought.

Reasoning models add pressure from another direction, since a model that thinks before answering consumes window space with its own working, as our piece on what thinking tokens buy explains.

How to read the next announcement

The headline number isn’t the useful part. Ask what the benchmark measured, single-fact retrieval or reasoning across scattered facts, and whether performance was reported across the full range or only at the extremes.

Ask what it costs per token at length, and what latency looks like when the window is actually full.

Be sceptical of the evaluation itself, too. Benchmarks are selected by the company shipping the model, and contamination is widespread enough that researchers now argue benchmarks should be built contamination-resistant by design.

Window sizes are quoted in tokens, and a token is a stranger unit than it looks. Our explainer on how tokenisation works covers why a million tokens is not a million words.

What would change this picture is an architecture where attention cost stops growing with sequence length, since that’s the constraint everything else works around. Until then, the window size is a ceiling rather than a promise, and what matters is how much of it the model can genuinely think with.