Signum
Learn

How LLM context windows work, and why bigger is not always better

By Elliot Marsh, Editor · · 5 min read

Every large language model has a limit on how much text it can take into account at once. That limit is the context window. It decides how long a document you can paste in, how long a conversation can run before the model "forgets" the start, and how much an AI agent can keep in view while it works.

The short version

The context window is the maximum number of tokens a model can process in one go, counting both what you send (the input) and what it writes back (the output). Anything outside the window does not exist as far as the model is concerned.

Tokens, not words

Models do not read words or characters. They read tokens: chunks of text produced by a tokeniser. Common words are often one token; rarer words are split into several pieces; spaces and punctuation are tokens too. For English prose a useful rule of thumb is that one token is about three-quarters of a word, so 1,000 tokens is roughly 750 words. Other languages, code and numbers often need more tokens for the same content. OpenAI's online tokeniser tool lets you see exactly how a piece of text is split.

What "in context" really means

When you chat with a model, it does not have a memory that persists between messages. Each time you send a message, the application sends the whole conversation so far (plus any system instructions, uploaded files and tool results) back to the model as one long input. The model reads all of it and predicts the reply token by token.

So the context window is less like memory and more like a desk. Whatever is on the desk can be used. When the desk is full, something has to come off, and the application decides what: usually the oldest messages, or a summary replaces them.

Why there is a limit

Modern models are based on the Transformer architecture, introduced in the 2017 paper Attention Is All You Need. Its core mechanism, self-attention, lets every token look at every other token in the input to work out what is relevant. That is what makes these models good at using context, but in the standard form the work grows with the square of the input length: double the tokens and the attention computation roughly quadruples. Memory use grows too, because the model keeps intermediate values (the "key-value cache") for every token it has seen.

Researchers have found many ways to stretch the limit: more efficient attention implementations, position-encoding schemes such as rotary position embeddings (RoPE) that can be extended to longer lengths, and training on long documents. That is why windows have grown from a few thousand tokens in early models to hundreds of thousands, and in some cases millions.

Bigger windows have costs

A long context is not free.

  • Price. Most APIs charge per token, so a 200,000-token prompt costs far more than a 2,000-token one, every time you send it. Many providers now offer cheaper rates for "cached" input that repeats between calls.
  • Speed. The model has to process the whole input before it writes the first word of the answer, so long prompts take longer to start.
  • Output limits. The maximum length of the answer is usually much smaller than the input window.

Long context is not perfect recall

Being able to accept a million tokens does not mean using all of them equally well. The 2023 paper Lost in the Middle (Liu et al.) found that models were best at using information placed at the start or end of a long input and noticeably worse when the relevant passage sat in the middle. Newer models have improved, but practitioners still see quality drop as the context fills, especially when the task requires combining many scattered facts rather than finding one.

Simple "needle in a haystack" tests, where one odd sentence is hidden in a long document, are useful but flattering. Finding one planted sentence is easier than reasoning across 40 contracts.

Context windows and agents

For AI agents, the window is often the binding constraint. Every tool call adds its result to the context: a web page, a file, an error log. A long task can fill the window in minutes. Agent frameworks handle this by:

  • summarising earlier steps (compaction);
  • storing notes or files outside the model and reading them back when needed;
  • trimming tool outputs to the relevant parts.

Each of these risks losing a detail that turns out to matter, which is one reason long-running agents drift.

Context window or retrieval?

If your documents are larger than the window, or you want answers grounded in a large library, the usual approach is retrieval-augmented generation (RAG): search for the relevant passages first and put only those in the context. Even when everything would fit, retrieval can be cheaper and sometimes more accurate, because the model has less irrelevant text to wade through.

Practical tips

  1. Put instructions and the question where they will be noticed. For long inputs, state the task clearly, and repeat the key question after the long material.
  2. Cut what is irrelevant. Less text often means better answers and lower cost.
  3. Structure long inputs. Headings, labels and clear separators between documents help the model find things.
  4. Start fresh when a conversation drifts. A new chat with a short summary often beats a long, cluttered one.
  5. Check the real limit. Advertised windows include output tokens and system prompts; apps may also cap them lower than the underlying model.

The bottom line

A context window is the model's working space, measured in tokens. Bigger windows make new things possible, from reading whole codebases to running long agent tasks, but they cost more, run slower and still do not guarantee the model will use every detail. Treat the window as space to be managed, not filled.

Sources