Context window

A context window is how much text an AI model can take in at once: your question, its instructions, any documents you add, the conversation so far and its own answer. It is measured in tokens, small pieces of words. Whatever does not fit has to be left out or summarised, so with a long contract or a long chat, the AI can miss what it never got to see.

How big is a context window?

For Gemini models, Google puts a token at about 4 characters, with 100 tokens equal to about 60 to 80 English words (Google). According to its long-context guide, early generative models took 8,000 tokens at a time and later ones 32,000 or 128,000. Google calls Gemini the first model that accepted 1 million tokens, which it compares to eight average-length English novels. In September 2026, Anthropic lists a window of 1 million tokens for its newer Claude models and 200,000 for others, such as Claude Sonnet 4.5 (Anthropic).

What counts toward it?

Everything in a request counts: the system prompt, every message including documents and tool results, the tool definitions, and the answer the model writes, including any reasoning it does first. Anthropic calls the context window the model's "working memory", separate from the data it was trained on. Providers bill per token, so a fuller window also makes each request more expensive.

At the limit, the behaviour depends on the service. For Claude 4.5 models and newer, generation stops with a stop reason that says the window is full; earlier Claude models return an error when the request cannot fit. Applications with long conversations therefore drop old messages, summarise them or fetch only the relevant passages with RAG, strategies Google lists for models with smaller windows. See also context compression.

Is a bigger window always better?

No, for two reasons. The attention mechanism in a transformer compares tokens with each other, so its cost per layer grows with the square of the input length (Vaswani et al., 2017): twice the context means about four times the attention computation.

Models also do not use a long context evenly. Liu et al. (2023) found that models answer best when the relevant passage sits at the start or the end of the input, and worst when it sits in the middle. In the worst case, GPT-3.5-Turbo answered questions about 20 or 30 documents less well than without any documents (56.1% correct). The RULER benchmark (2024) tested 17 models that all claim at least 32,000 tokens; only half of them still performed satisfactorily at that length. Anthropic calls this decline context rot.

Example

Suppose a webshop wants its assistant to answer from a product manual of 200,000 characters. At about 4 characters per token, that is roughly 50,000 tokens. It fits in a window of 200,000 tokens, but every question then carries those 50,000 tokens, and an answer buried in the middle is the hardest for the model to use. With RAG, the assistant sends only the few sections that match the question, which keeps each request small and puts the relevant text in front of the model.