Why do models get worse with more context?

Attention is not uniform across position. Models weight the start and end of a context more heavily than the middle, and the effect worsens as occupancy rises. A decisive fact buried mid-document is the one most likely to be missed.

Is long context ever the better choice?

Yes — for a single large document where chunking would sever the relationships an answer depends on. A contract, a long transcript, a file whose parts refer to each other. That is what the window is for.

Comparison

Long context vs RAG

Million-token windows made stuffing everything into the prompt possible. Why it performs worse than retrieval on both accuracy and cost, and when it is genuinely right.

The short answer

Retrieve rather than stuff, in almost every case. A tightly retrieved 20,000 token prompt regularly beats a padded 500,000 token one on accuracy, because recall degrades as a window fills — and it always beats it on cost, because long context is billed on every turn. Long context earns its place on single large documents where chunking would break cross-references.

At a glance

Long context compared with RAG
	Long context	RAG
Cost per request	Very high — everything, every time	Low — only what was retrieved
Accuracy as size grows	Degrades — context rot	Stable
Setup effort	None	Chunking, embedding, a vector store
Latency	High — prefill dominates	Lower
Cross-document reasoning	Strong	Limited to what was retrieved
Corpus size ceiling	The window	Effectively unlimited

When to choose which

Choose Long context when

One document that must be read wholeA contract, a codebase file, a transcript where an answer depends on relationships between distant sections. Chunking would sever exactly the connections that matter.
A prototype, before retrieval existsStuffing is the fastest way to find out whether the idea works at all. It is a legitimate first step and a poor final state.
The corpus is small and staticIf everything genuinely fits with room to spare and never changes, the engineering cost of retrieval may not be worth paying.

Choose RAG when

The corpus is larger than the windowThe obvious case. Retrieval is the only option once the material exceeds what fits.
Cost mattersThis is usually the deciding factor. Sending 200,000 tokens on every request rather than 3,000 is a factor of sixty on the input line of the bill.
You need citationsRetrieval knows which passage produced the answer. Stuffing does not, so an answer cannot be traced back to a source.
Accuracy matters on a large corpusRecall degrades as the window fills and is weakest in the middle. Retrieval puts a small amount of relevant material where attention is strongest.

What it costs either way

The cost difference is not marginal. At a typical input rate, 200,000 tokens per request across 100,000 requests a month is a five-figure monthly bill; 3,000 retrieved tokens over the same traffic is a fraction of it.

Long context is billed on every turn of a conversation, not once. A stuffed prompt in a chat product multiplies by the turn count as well as by the request count.

Retrieval adds an embedding cost that is almost always trivial by comparison — embedding a corpus once costs tens of dollars, and query embedding is a fraction of a cent.

Prefill latency scales with input length too, so stuffing costs time to first token as well as money.

The mistake people make

Treating a million-token window as a substitute for retrieval

The window makes stuffing possible; it does not make it advisable. Recall degrades as occupancy rises, and a fact in the middle of a very long context is the one most likely to be missed. Providers document this themselves — curating what goes into context matters as much as how much space exists. Using the window as a filing cabinet gets you worse answers at sixty times the price.

How to decide

1Measure the corpus. If it does not fit with substantial headroom, the decision is made.
2If it fits, price both: everything on every request against retrieved chunks on every request.
3Test accuracy on your hardest questions in both configurations. On a large corpus, retrieval usually wins that too.
4Reach for long context deliberately, for single documents where the relationships matter — not as the default.

Price it yourself

Frequently asked questions

Why do models get worse with more context?: Attention is not uniform across position. Models weight the start and end of a context more heavily than the middle, and the effect worsens as occupancy rises. A decisive fact buried mid-document is the one most likely to be missed.
Is long context ever the better choice?: Yes — for a single large document where chunking would sever the relationships an answer depends on. A contract, a long transcript, a file whose parts refer to each other. That is what the window is for.