# RAG vs long context in 2026: cost, accuracy and when to use each

URL: https://infro.io/blog/rag-vs-long-context-2026
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-10-20
Last modified: 2026-10-20
Last reviewed: 2026-10-20 (next review due 2026-11-19)
Topic: Agents and multimodal systems
Format: Guide

> Long context means putting whole documents in the prompt; RAG means retrieving only the relevant chunks. With prompt caching, long context can be practical for a single corpus up to a few hundred thousand tokens that many queries reuse, and it avoids retrieval misses. RAG stays cheaper per query by one to two orders of magnitude, scales past any window, keeps per-document permissions enforceable, and suffers less from accuracy loss as inputs grow, which Chroma's 2025 study of 18 models measured. Pricing thresholds matter too: OpenAI re-prices requests above 272K tokens and Claude Haiku 5.5 above 100K.

Key takeaways:
- Long context works best for one corpus of up to a few hundred thousand tokens, reused across many queries with prompt caching.
- RAG costs one to two orders of magnitude less per query, scales past any window, and keeps per-document permissions enforceable.
- Accuracy falls as input grows: Chroma's 2025 study of 18 models found performance degrades consistently with input length.
- Pricing thresholds change the answer: OpenAI's long-context rates apply above 272K input tokens, Claude Haiku 5.5's above 100K.
- A hybrid — retrieve broadly, then reason over a larger context — often beats both.

**Do you still need RAG now that models take a million tokens?** Sometimes not. Putting whole documents in the prompt works well for one corpus of up to a few hundred thousand tokens that many queries reuse, especially with prompt caching. **RAG** — retrieving only the relevant chunks — remains one to two orders of magnitude less expensive per query, scales past any context window, keeps per-document permissions enforceable, and is less exposed to the accuracy loss that comes with very long inputs. For many products the best answer is a hybrid: retrieve broadly, then let a long-context model reason over a larger, relevant slice.

## What changed

Claude 4.6 and later models (except Haiku 5.5) include a 1M-token context window at standard rates; Anthropic's page notes that a 900K-token request pays the same per-token rate as a 9K-token one[1]. Other vendors price long requests differently: OpenAI charges long-context rates on the whole request above 272K input tokens[2], Gemini 3.1 Pro charges more above 200K[3], and Claude Haiku 5.5 re-prices requests whose prompt exceeds 100,000 tokens[1]. So "fits in the window" and "costs the same per token" are separate questions, and the answer depends on the model.

## The cost per query, worked through

> **Example: A 400,000-token policy library, 500 questions a day**
> Each answer is 500 output tokens. Long context sends the whole library with every question; RAG sends 4,000 tokens of retrieved chunks plus a 500-token prompt.
> 
> **Claude Sonnet 5.5** ($2 input, $0.10 cache read, $10 output per 1M)[1]:
> 
> - Long context, uncached: 0.4M × $2 + 0.0005M × $10 = $0.805 per question, about $400 a day
> - Long context, cache hit: 0.4M × $0.10 + $0.005 = $0.045 per question, about $23 a day, plus cache writes whenever the cache expires
> - RAG: 0.0045M × $2 + $0.005 = $0.014 per question, about $7 a day, plus embedding and vector search
> 
> **GPT-6 Sol** above 272K tokens is on long-context rates: $4 input, $0.40 cached input, $15 output per 1M[2]:
> 
> - Long context, cache hit: 0.4M × $0.40 + 0.0005M × $15 = about $0.17 per question
> - RAG, on short-context rates: 0.0045M × $2 + 0.0005M × $10 = $0.014 per question

Caching is what makes long context affordable at all: without it, the same library costs over fifty times as much per question as RAG on Sonnet 5.5. With it, long context is about three times the RAG cost there, and about twelve times on GPT-6 Sol, where the 272K line moves every request onto higher rates. Caches also expire: Anthropic's 5-minute cache needs traffic to stay warm, and each rewrite costs 1.25 times the input rate[1]. A library queried every few seconds stays cached; one queried a few times an hour pays the write repeatedly.

## Accuracy: more context is not more attention

The argument for long context is that the model sees everything, so nothing is missed by retrieval. The argument against is that models do not use long inputs evenly. Chroma's July 2025 report tested 18 models from Anthropic, OpenAI, Google and Alibaba and found that "across all experiments, model performance consistently degrades with increasing input length", even on simple tasks, and that distractors related to the answer made it worse[4]. Newer models handle long inputs better than the ones tested, but the direction of the effect is the reason to measure rather than assume.

In practice:

- **Long context avoids retrieval misses** — the failure where the right passage was never retrieved.
- **RAG avoids distraction** — the failure where the right passage is present but buried among similar ones.

Which failure dominates depends on your corpus and questions, which is why the decision belongs to an evaluation on your own query set, not to a context-window number. [Long context in production](https://infro.io/blog/long-context-window-management) covers context rot and context management in depth.

## The question teams forget: permissions

RAG can filter retrieval by what the asking user may see: a sales rep's question retrieves only documents the rep can open. Long context puts the whole corpus in front of the model for every question, so any user can, in principle, get the model to repeat a document they should not see. If your corpus has per-document access rules, long context over the whole corpus is a permission design you cannot enforce; at minimum, build the context per user from what they may access, which is retrieval by another name.

## Freshness and size

- **Corpus larger than the window** (most company knowledge bases, all of a codebase's history, a year of tickets): retrieval is not optional.
- **Corpus that changes daily**: RAG updates one chunk; long context changes the cached prefix and pays the cache write again.
- **One stable document or bundle** (a contract, a manual, a book, one repository snapshot): long context is simplest and avoids building an index.

## A decision guide

| Situation | Prefer |
| --- | --- |
| One stable corpus under ~200K tokens, many questions, cache stays warm | Long context with caching |
| Corpus over the window, or growing | RAG |
| Per-document access rules | RAG with permission filters |
| Low cost or low latency per query matters | RAG |
| Questions that need the whole document (summaries, cross-references) | Long context, or a hybrid |
| Retrieval keeps missing the right passage | Hybrid: retrieve 30–50 chunks, then a long-context model |

The hybrid is often the best of both: retrieval narrows a million-document corpus to a few dozen relevant chunks, and a model with a large window reasons over all of them, rather than over the top five. Measure recall and answer quality on your own questions before choosing; [where the tokens go in a RAG pipeline](https://infro.io/blog/rag-token-accounting) helps size the retrieved context.

## Latency is part of the cost

Every input token has to be processed before the first output token appears. A 4,000-token RAG prompt and a 400,000-token long-context prompt differ by a factor of a hundred in what the model reads first, and prompt caching reduces but does not remove that difference. For interactive products, measure time to first token on both designs with your real prompts; [measuring time to first token](https://infro.io/blog/measure-time-to-first-token) shows how. Retrieval adds its own latency — an embedding call and a vector search — usually tens of milliseconds, which is small next to reading hundreds of thousands of tokens.

## How to run the comparison on your own data

1. **Collect 100 to 300 real questions** with known answers or source passages.
2. **Build the long-context variant:** the corpus as one cached prefix, the question appended.
3. **Build the RAG variant:** your chunking, embedding model and top-k, with the same answer prompt.
4. **Score both** on answer correctness and on whether the cited passage is right, with a separate grading model checked against human review.
5. **Record cost and time to first token per question** from the provider's usage fields, not from estimates.
6. **Repeat with a hybrid:** a larger top-k (30–50 chunks) into the long-context model.

The winning design is usually obvious once the three columns — quality, cost per question and latency — sit side by side.

## Measuring both with INFRO

Whichever design you choose, the cost differences above appear per request. INFRO records each request's input, cached and output tokens with an estimated provider cost, including embedding calls on `POST /v1/embeddings`, and `POST /v1/estimates` prices a request before you send it — useful for seeing which questions cross a 100K or 272K pricing line. Attribution by workflow and step lets you compare the retrieval, embedding and answer steps of a RAG pipeline against a long-context design on the same traffic.

## Summary

Long context replaces retrieval for one stable, cacheable corpus that fits comfortably in the window. RAG wins on cost, scale, freshness and permissions, and is less exposed to accuracy loss on long inputs. Check each model's pricing thresholds, evaluate on your own questions, and consider retrieving broadly into a long-context model when neither alone is good enough.

## Questions

### Do I still need RAG with a 1M-token context window?

Often yes. Long context can replace retrieval for a single corpus that fits comfortably in the window and is queried many times with caching. RAG remains the better choice when the corpus is larger than the window, changes often, has per-document access rules, or must be queried at low cost and latency.

### Is long context more accurate than RAG?

Not reliably. It avoids retrieval misses, but Chroma's 2025 study of 18 models found model performance consistently degrades as input length grows, even on simple tasks. A well-tuned retriever that puts the right few thousand tokens in front of the model can be more accurate than a million tokens of mostly irrelevant text.

### How much does a 1M-token prompt cost?

At standard rates in October 2026, a million uncached input tokens costs $2 on Claude Sonnet 5.5, $4 on Claude Opus 5.5, and $4 on GPT-6 Sol at its long-context rate, before output. Cached reads cost much less: $0.10 per million on Sonnet 5.5. Repeating that on every query adds up quickly.

## Sources

1. Pricing — Anthropic. https://platform.claude.com/docs/en/about-claude/pricing (accessed 2026-10-20)
2. Pricing — OpenAI. https://developers.openai.com/api/docs/pricing (accessed 2026-10-20)
3. Gemini Developer API pricing — Google. https://ai.google.dev/gemini-api/docs/pricing (accessed 2026-10-20)
4. Context Rot: How Increasing Input Tokens Impacts LLM Performance — Chroma. https://www.trychroma.com/research/context-rot (accessed 2026-10-20)