Context windows explained

Every request you send to a large language model has a hard ceiling on how much text it can involve. That ceiling, the context window, shapes what you can paste in, how long a chat can run, what you pay and how reliable answers are. This guide explains how it works, converts tokens to words, pages and code with worked arithmetic, compares current limits and gives practical rules for long prompts.

What a context window is

A context window is the maximum number of tokens a model can process in one request. Tokens are the text chunks a model's tokenizer produces: common words are usually one token, while rarer words split into several.

The window is one shared budget for input and output. Your system prompt, the full conversation history, pasted documents, images, tool definitions and tool results all count, and so does the reply. Anthropic's and OpenAI's API documentation both state that the reasoning tokens produced by reasoning models count against the same window.

Most models also cap output separately, at a much lower number than the window. A model with a 1M-token window and a 128K output cap can read a very long prompt but cannot write a 500K-token reply in one call.

Here is a worked budget for a self-hosted model with a 131,072-token window, the "128K" size Meta specifies for Llama 3.1:

• System prompt and rules: 1,500 tokens
• Tool definitions: 2,500 tokens
• Conversation history: 20,000 tokens
• Retrieved documents: 90,000 tokens
• Reserved for the reply: 8,000 tokens

That totals 122,000 tokens, leaving about 9,000 spare. Each turn adds to the history, which is why long chats eventually hit the wall even when every message is short.

Converting tokens to words, pages, books and code

For English prose, OpenAI's help center puts one token at roughly four characters or three-quarters of a word, so 100 tokens is about 75 words. Google's Gemini documentation uses the same four-character figure and a range of 60 to 80 words per 100 tokens. Plan with 0.75 words per token and treat every result as an estimate.

The page, book and code columns below add three planning assumptions: a dense single-spaced page holds about 500 words, a typical novel runs about 90,000 words, and an average line of code costs about 10 tokens. For a 128K window the arithmetic is 128,000 × 0.75 = 96,000 words, then 96,000 ÷ 500 = 192 pages, 96,000 ÷ 90,000 ≈ 1.1 novels, and 128,000 ÷ 10 = 12,800 lines of code.

These figures cover the whole window, so subtract your system prompt and reply budget first, then adjust for these caveats:

• Language matters. The ratios above are for English. Other languages tokenize differently, and text in many non-Latin scripts can need noticeably more tokens for the same content.
• Code and data are denser. Indentation, symbols, long identifiers, JSON keys and numbers all cost tokens, so your codebase may land well above or below 10 tokens per line.
• Tokenizers differ. The same text produces different counts on different model families, and sometimes on different versions from one provider.
• Media has its own rates. Gemini's docs count an image up to 384 pixels on each side as 258 tokens, tile larger images into 768-pixel squares at 258 tokens each, and count audio at 32 tokens per second. One hour of audio is therefore 3,600 × 32 = 115,200 tokens.

The fastest way to stop guessing is to paste a real sample into the token counter and scale from there.

Context window sizes as of October 2026

The table lists published limits for current flagship models from major API providers, plus two widely used open-weight families, checked against each provider's documentation on October 10, 2026. Limits can differ by plan, region or cloud platform and change often, so confirm before you design around a number.

Sources: OpenAI models, Anthropic models overview, Gemini models, DeepSeek models and pricing, xAI models and Meta's Llama 4 announcement.

Two patterns stand out: about 1M tokens is now standard for most frontier APIs, and output caps remain far smaller than input limits. For open-weight models, the advertised figure is what the model was built for; a host may serve less, and your own GPUs may force you lower. Browse the LLM directory or compare models side by side for current specs.

What happens when you hit the limit

The result depends on where the overflow happens and what software sits between you and the model. Expect one of four behaviors.

• The request is rejected. If your input alone exceeds the window, APIs refuse it. Anthropic's API, for example, returns a 400 error saying the prompt is too long.
• The reply stops early. When output reaches the max output setting or the end of the window, generation stops mid-thought and the API reports a stop reason such as max_tokens or length. Anthropic's newer models use a dedicated model_context_window_exceeded reason. A JSON object cut off halfway will not parse, so check the stop reason before using any output.
• Old content is silently dropped. Many chat apps and frameworks trim the oldest messages, or the middle of a long paste, to make room. The model cannot tell you about text it never received.
• History is summarized. Some systems replace older turns with a summary, often called compaction, and OpenAI's Responses API offers it server-side. Summaries save space but lose detail, especially exact numbers, names and quotes.

Why a bigger window doesn't guarantee better answers

The advertised window tells you what fits. The effective context, meaning how much the model can actually use well, is often smaller and depends heavily on the task.

The best-known finding is "lost in the middle." In a 2023 study, Liu and colleagues found that models answered best when the relevant passage sat at the start or end of a long input, and noticeably worse when it sat in the middle. The practical rule: keep the material that matters most near the beginning or end, and put your question last.

Needle-in-a-haystack tests, which hide one fact in long filler text and ask the model to find it, flatter models because they reward simple lookup. The RULER benchmark tested 17 models in 2024: they scored nearly perfectly on the basic needle test, yet only half held up at 32K tokens on a broader mix of retrieval, tracing and aggregation tasks. The 2025 NoLiMa study removed literal word overlap between question and needle; at 32K tokens, 11 of 13 models fell below half their short-context scores, and GPT-4o dropped from 99.3% to 69.7%.

Chroma's 2025 context rot report found similar patterns across 18 models, even on simple tasks: reliability fell as input grew, and distracting near-matches made it worse. Anthropic's documentation uses the same term and warns that adding context does not automatically improve results.

The models in these studies are a generation or more old, so treat the numbers as historical. The lesson holds: test your own task at the length you plan to use.

Cost, latency and memory

You pay for every input token on every request. Most chat APIs are stateless, so each turn resends the whole history. If every turn adds 2,000 tokens, turn 10 sends 20,000 tokens, and the 10 turns together send 2,000 × (1 + 2 + … + 10) = 110,000 input tokens, not 20,000.

Long documents multiply quickly. At an illustrative $2.00 per million input tokens, a 400,000-token document costs 0.4 × $2.00 = $0.80 to send once. Ask 20 questions about it without caching and you pay $16.00 in input alone, before any output.

Some providers also charge more per token above a length threshold. As of October 2026, OpenAI bills GPT-6 family requests with more than 272K input tokens at 2× the input rate and 1.5× the output rate for the whole request, and Google's Gemini 3.1 Pro Preview moves from $2.00 to $4.00 per million input tokens above 200K. Anthropic prices the full 1M window of its current models at standard rates, except Haiku 5.5, which steps up above 100K tokens. Run your numbers through the LLM cost calculator and read LLM API pricing explained before committing to a long-context design.

Long prompts are also slower. The model must process every input token before the first output token appears, and standard attention work grows with the square of sequence length. Measure time to first token, a key part of latency, at your real prompt size: a design that feels instant at 5,000 tokens can drag at 500,000.

Prompt caching

Prompt caching lets a provider reuse the processed form of a prompt prefix it has seen recently, such as a long system prompt, tool definitions or a document. As of October 2026, OpenAI and Anthropic list cache writes at about 1.25× the normal input rate and reads at about 0.1×, with deeper read discounts on several current models. Gemini caches automatically on 2.5 and newer models once a prompt passes a minimum size, 4,096 tokens on Gemini 3.1 Pro and recent Flash models.

Rework the earlier example with caching. The first request writes the 400,000-token document at 1.25 × $2.00 = $2.50 per million, so $1.00. Each of the next 19 questions reads it at $0.20 per million, or $0.08, for $1.52 in total. You pay $2.52 instead of $16.00, a saving of about 84%, and cached requests generally start responding sooner.

• Caches match exact prefixes. Put stable content first and variable content, such as the user's question or a timestamp, last. One changed character early in the prompt invalidates everything after it.
• Caches expire. Anthropic's default lifetime is five minutes, refreshed on each hit, with a paid one-hour option. OpenAI's newer models keep a prefix eligible for 30 minutes after its last use.
• Short prompts don't qualify. Minimums range from 512 to 4,096 tokens depending on provider and model.

KV cache memory if you self-host

When you run an open model yourself, long context costs GPU memory. During generation the model stores key and value vectors for every token in context, in every layer, in what's called the KV cache. It grows linearly with context length and with the number of requests you serve at once.

Bytes per token = 2 (keys and values) × layers × KV heads × head dimension × bytes per value. Llama 3.1 8B has 32 layers, 8 KV heads and a head dimension of 128, per Meta's published architecture in the Llama 3 paper. At 16-bit precision that is 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB per token, so a full 131,072-token context needs 16 GiB of KV cache for a single request. That is about as much memory as the model's 16-bit weights.

Llama 3.1 70B has 80 layers with the same KV heads and head dimension, so it needs 320 KiB per token and 40 GiB for one full-length request. Four users at full length need 160 GiB for the cache alone.

To cope, cap the maximum context in your server settings, store the cache in 8-bit precision to halve it, and use a serving engine with paged memory management. The vLLM paper reported 2-4× higher throughput than earlier serving systems by cutting KV cache waste. The VRAM calculator runs this arithmetic for you.

Long context or RAG?

Retrieval-augmented generation (RAG) searches your documents and sends the model only the most relevant passages, while long context sends everything. Neither is always better, so decide per workload.

Many production systems combine both: retrieve generously, then let a long-context model reason across dozens of passages at once. See RAG explained for how to build the retrieval side.

Practical ways to fit more in

Common mistakes

• Treating the advertised window as usable capacity. A model that accepts 1M tokens may still reason poorly across 300,000 of them.
• Forgetting the output and reasoning budget. A reasoning model can spend thousands of tokens thinking before it writes a word of the answer.
• Using English ratios for code or other languages. Measure instead of assuming.
• Ignoring stop reasons. Truncated output looks fine until a parser or a user hits the missing half.
• Breaking the cache. A timestamp or user name at the top of the system prompt turns every request into a cache miss.
• Self-hosting at maximum context by default. Under concurrent load, the growing KV cache is usually what exhausts GPU memory.
• Comparing prices without long-context surcharges. Per-token rates can double past a threshold.

If you're designing a long-document pipeline and want someone to review the architecture, you can get hands-on help from LLM specialists.

Next steps

• See how much text fits in 8K, 128K and 1M tokens with the context window visualizer.
• Count the tokens in your own prompts and estimate per-request cost with the token counter.
• Decide how to bring your own data into the window with RAG explained.