Do you still need RAG now that models take a million tokens? Sometimes not. Putting whole documents in the prompt works well for one corpus of up to a few hundred thousand tokens that many queries reuse, especially with prompt caching. RAG — retrieving only the relevant chunks — remains one to two orders of magnitude less expensive per query, scales past any context window, keeps per-document permissions enforceable, and is less exposed to the accuracy loss that comes with very long inputs. For many products the best answer is a hybrid: retrieve broadly, then let a long-context model reason over a larger, relevant slice.
What changed
Claude 4.6 and later models (except Haiku 5.5) include a 1M-token context window at standard rates; Anthropic's page notes that a 900K-token request pays the same per-token rate as a 9K-token one1. Other vendors price long requests differently: OpenAI charges long-context rates on the whole request above 272K input tokens2, Gemini 3.1 Pro charges more above 200K3, and Claude Haiku 5.5 re-prices requests whose prompt exceeds 100,000 tokens1. So "fits in the window" and "costs the same per token" are separate questions, and the answer depends on the model.
The cost per query, worked through
Caching is what makes long context affordable at all: without it, the same library costs over fifty times as much per question as RAG on Sonnet 5.5. With it, long context is about three times the RAG cost there, and about twelve times on GPT-6 Sol, where the 272K line moves every request onto higher rates. Caches also expire: Anthropic's 5-minute cache needs traffic to stay warm, and each rewrite costs 1.25 times the input rate1. A library queried every few seconds stays cached; one queried a few times an hour pays the write repeatedly.
Accuracy: more context is not more attention
The argument for long context is that the model sees everything, so nothing is missed by retrieval. The argument against is that models do not use long inputs evenly. Chroma's July 2025 report tested 18 models from Anthropic, OpenAI, Google and Alibaba and found that "across all experiments, model performance consistently degrades with increasing input length", even on simple tasks, and that distractors related to the answer made it worse4. Newer models handle long inputs better than the ones tested, but the direction of the effect is the reason to measure rather than assume.
In practice:
- Long context avoids retrieval misses — the failure where the right passage was never retrieved.
- RAG avoids distraction — the failure where the right passage is present but buried among similar ones.
Which failure dominates depends on your corpus and questions, which is why the decision belongs to an evaluation on your own query set, not to a context-window number. Long context in production covers context rot and context management in depth.
The question teams forget: permissions
RAG can filter retrieval by what the asking user may see: a sales rep's question retrieves only documents the rep can open. Long context puts the whole corpus in front of the model for every question, so any user can, in principle, get the model to repeat a document they should not see. If your corpus has per-document access rules, long context over the whole corpus is a permission design you cannot enforce; at minimum, build the context per user from what they may access, which is retrieval by another name.
Freshness and size
- Corpus larger than the window (most company knowledge bases, all of a codebase's history, a year of tickets): retrieval is not optional.
- Corpus that changes daily: RAG updates one chunk; long context changes the cached prefix and pays the cache write again.
- One stable document or bundle (a contract, a manual, a book, one repository snapshot): long context is simplest and avoids building an index.
A decision guide
| Situation | Prefer |
|---|---|
| One stable corpus under ~200K tokens, many questions, cache stays warm | Long context with caching |
| Corpus over the window, or growing | RAG |
| Per-document access rules | RAG with permission filters |
| Low cost or low latency per query matters | RAG |
| Questions that need the whole document (summaries, cross-references) | Long context, or a hybrid |
| Retrieval keeps missing the right passage | Hybrid: retrieve 30–50 chunks, then a long-context model |
The hybrid is often the best of both: retrieval narrows a million-document corpus to a few dozen relevant chunks, and a model with a large window reasons over all of them, rather than over the top five. Measure recall and answer quality on your own questions before choosing; where the tokens go in a RAG pipeline helps size the retrieved context.
Latency is part of the cost
Every input token has to be processed before the first output token appears. A 4,000-token RAG prompt and a 400,000-token long-context prompt differ by a factor of a hundred in what the model reads first, and prompt caching reduces but does not remove that difference. For interactive products, measure time to first token on both designs with your real prompts; measuring time to first token shows how. Retrieval adds its own latency — an embedding call and a vector search — usually tens of milliseconds, which is small next to reading hundreds of thousands of tokens.
How to run the comparison on your own data
- Collect 100 to 300 real questions with known answers or source passages.
- Build the long-context variant: the corpus as one cached prefix, the question appended.
- Build the RAG variant: your chunking, embedding model and top-k, with the same answer prompt.
- Score both on answer correctness and on whether the cited passage is right, with a separate grading model checked against human review.
- Record cost and time to first token per question from the provider's usage fields, not from estimates.
- Repeat with a hybrid: a larger top-k (30–50 chunks) into the long-context model.
The winning design is usually obvious once the three columns — quality, cost per question and latency — sit side by side.
Measuring both with INFRO
Whichever design you choose, the cost differences above appear per request. INFRO records each request's input, cached and output tokens with an estimated provider cost, including embedding calls on POST /v1/embeddings, and POST /v1/estimates prices a request before you send it — useful for seeing which questions cross a 100K or 272K pricing line. Attribution by workflow and step lets you compare the retrieval, embedding and answer steps of a RAG pipeline against a long-context design on the same traffic.
Summary
Long context replaces retrieval for one stable, cacheable corpus that fits comfortably in the window. RAG wins on cost, scale, freshness and permissions, and is less exposed to accuracy loss on long inputs. Check each model's pricing thresholds, evaluate on your own questions, and consider retrieving broadly into a long-context model when neither alone is good enough.