Me, and the AI thingy

What a Context Window Actually Is (and What It Costs)

A context window is the finite set of tokens a model reads in one pass to produce its next token. Your instructions, the open files, the running conversation, every line of tool output: all of it, held in one bounded space with a fixed capacity. Nothing outside the window exists to the model. Nothing inside it is free.

The setting is agents: coding assistants and the tools built around them. Most of the mechanics hold for any use of a large language model, but the pressures are sharpest when an agent is filling the window on its own. Three facts do most of the work, and they all follow from one mechanic: the model reads the whole window, start to finish, on every single step.

TL;DR. A token is about 4 characters of English, so 100 tokens is roughly 75 words, and code runs denser. What fills the window: system prompt, tool definitions, loaded files, the conversation, and tool output. Attention cost grows with the square of length, so doubling the context roughly quadruples the work per step. A stable prefix is cheap to reuse (cached reads run 50 to 90% off), but change it early and you pay full price again. And capacity is not reliability: models get measurably less accurate long before the window is full.

Anatomy of a context window: a single bounded bar filled by a gray standing load (system prompt, tool definitions, files and skills) and an orange flood (conversation, tool output), with free space shrinking toward a fixed capacity.

Tokens: the unit you are actually spending

The model does not read characters or words. It reads tokens: short chunks of text, each mapped to a number. For English prose, one token is about 4 characters, and 100 tokens come to roughly 75 words (OpenAI, What are tokens and how to count them). “Tokenization” is nothing more than the step that cuts your text into those chunks before the model sees it.

Two practical notes. First, the count is model-specific, so treat 4 characters as a rule of thumb, not a law. Second, code is denser than prose. Indentation, punctuation, and camelCase identifiers all fragment into many small tokens, so a file of source costs noticeably more tokens per character than the same length of English. A budget you sized on prose will run out early on code.

What goes into the window

The window is not just your prompt. It is a budget, and five things spend it.

The standing load is what sits there before the real work starts: the system prompt, the tool definitions the model is allowed to call, and any files or skills you loaded up front. You set it once. It is on the board for the whole session whether the current step needs it or not.

The flood is what accumulates while the agent runs: the conversation so far, and above all the tool output. A single test run or a directory listing can pour hundreds of lines into the window in one step. The standing load you choose deliberately. The flood arrives on its own, and it is usually where a window goes from lean to bloated.

The window is finite, and full is not the failure you fear

Every window has a hard ceiling, and that ceiling has moved a lot. The original Transformers trained on 512 to 2,048 tokens (Liu et al., 2024). GPT-4 shipped at 8K, with a 32K variant. Claude reached 100K in 2023 and 200K in 2024. By 2026 the frontier sits at 1M tokens on Claude, GPT-4.1, and Gemini 2.5 Pro, and 2M on Gemini 1.5 Pro (vendor documentation, 2025 to 2026).

Log-scale timeline of maximum context windows: 512 to 2K tokens for original Transformers in 2017, 8K to 32K for GPT-4 in 2023, 100K then 200K for Claude, and 1M to 2M by 2025 to 2026. Each step is about an order of magnitude.

Running out of room is the boring failure. You hit the ceiling, tokens get evicted or the call errors, and you know exactly what happened. The failure worth understanding arrives much earlier, well below the limit, and it is quiet. Hold that thought. The last two sections are about it.

Prefill and decode: two phases, two bills

Producing a response happens in two phases, and they cost differently.

Prefill reads your whole prompt in one parallel pass. Every input token is processed together, which saturates the GPU’s math units. Prefill is compute-bound.

Decode then generates the answer one token at a time. Each new token requires its own pass that re-reads the entire context built so far, and each pass is mostly the machine waiting on memory rather than doing math. Decode is memory-bandwidth-bound, and because it is sequential, it dominates the latency you actually feel (SPAD, arXiv 2025; and the standard prefill/decode literature).

Two panels: prefill reads the whole prompt in one parallel pass and is compute-bound; decode emits one token per pass, each pass re-reading a context that grows by one token, and is memory-bandwidth-bound.

The point for a window budget: the tokens you load are not paid once. The prompt is prefilled, and then every token the model writes is another pass over the whole thing.

Why a longer window costs more on every step

Here is the mechanic under all of it. Attention, the operation that lets each token look at every other token, scales with the square of the sequence length. Self-attention is O(n²) in the number of tokens (Vaswani et al., Attention Is All You Need, 2017). Liu et al. put the same fact plainly: Transformers “require memory and compute that increases quadratically in sequence length” (2024).

Quadratic is the word that matters. Double the tokens in the window and you do not double the attention work. You roughly quadruple it. Ten times the tokens is a hundred times the work.

A convex curve of attention work per step against context length: at length n the cost is one unit, at length 2n it is four units. Doubling the context roughly quadruples the work.

And this is paid per step. A window that is twice as full is not just carrying twice the material. It is more expensive to read, on every token the model generates, and it gets more expensive faster the fuller it gets. That is why “just use the big window because you can” is not free even when it fits.

The cache: why a stable prefix is cheap

The counterweight is caching, and it is worth understanding because it changes how you should order a prompt.

While the model processes a prompt, it builds a KV cache: the intermediate state for every token, kept so it does not have to recompute earlier tokens on each new step. That cache grows linearly with length. As a rough sense of scale, a 70B model can spend on the order of a couple of megabytes of cache per token (illustrative; modern grouped-query attention cuts this several-fold). Linear, not quadratic, but not nothing.

The lever is prefix caching. If the front of your prompt is byte-for-byte identical to a recent call, the provider reuses the cached work instead of reprocessing it. Anthropic bills those cached reads at 10% of the input price, a 90% discount; OpenAI applies about 50% off automatically, both above roughly 1,024 tokens (Anthropic and OpenAI prompt-caching docs). The catch: the cache matches a prefix. Change one token early in the prompt and everything after the edit is invalidated and recomputed at full price.

Two rows: a stable prefix is reused from cache (a cache hit, billed about 90% cheaper), while changing a token early invalidates the cache from the edit onward, forcing a full-price recompute.

So put the stable material first: system prompt, tool definitions, the things that do not change. Put the volatile material last. It is the difference between paying for your standing load once or paying for it on every turn.

Attention is not uniform: the middle gets neglected

The model reads the whole window, but it does not weigh every part of it equally. Position matters, and not in the way you would hope.

Liu et al. measured this directly. Give a model 20 documents and ask a question whose answer is in exactly one of them, and accuracy depends heavily on where that document sits. It is highest when the answer is first or last, and it sags in the middle.

🔬 Position, measured. For GPT-3.5-Turbo the curve runs from about 75% when the answer is first, down to a trough near 53% in the middle, back up to 63% when it is last (Liu et al., Lost in the Middle, 2024).

U-shaped accuracy curve against the position of the answer among 20 documents: about 75% first, a trough near 53% in the middle, 63% last, all against a 56.1% closed-book baseline.

Look at the dashed line. When the answer sits in the middle, the model scores below its closed-book baseline of 56.1%, the score it gets with no documents at all. Buried mid-context, the right answer was worse than no answer. This is why trimming a long input to its head and its tail is not a crude hack. The head and the tail are the parts attention actually uses.

Capacity is not reliability

Now the failure I told you to hold. A window that is not full can still be a window that has gone bad.

🔬 Measured, not guessed. In 2025 a benchmark called NoLiMa put this to the test on 13 frontier models. The design was deliberately unkind: strip out the literal word-matches that make retrieval easy, so the model has to reason over the text instead of keyword-spotting. The result was consistent and unwelcome. At 32,000 tokens, 11 of the 13 models scored below half of their own short-context baseline, and GPT-4o alone fell from a near-perfect 99.3% to 69.7% (Modarressi et al., NoLiMa: Long-Context Evaluation Beyond Literal Matching, ICML 2025).

A second benchmark, RULER, found the same gap from the other direction. Of 17 models that all advertised a context of 32,000 tokens or more, only half still held up when actually pushed to that length (Hsieh et al., RULER, 2024). That is the whole reason the size of the window and the health of the window are two different things. Capacity tells you what fits. It says nothing about what the model can still reliably use. A bigger window raised the ceiling. It did not buy immunity. The discipline of keeping the window lean, on purpose, is a separate piece, a story for another day.

Hold on to the one mechanic. The model reads the whole window, start to finish, on every step. Cost, position, and rot all fall out of that one line. The window was never storage. It is the thing the model rereads to think, and you pay for the reread every time.

Sources

context-engineering primer cost
Newsletter

One email a month — what I'm working through, and the posts it produced. One click to leave.

Malo Couaran
Got a reaction, a correction, or a better idea? Reply by email.