# Where the tokens go in a RAG pipeline

URL: https://infro.io/blog/rag-token-accounting
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-09-29
Last modified: 2026-09-29
Last reviewed: 2026-09-29 (next review due 2026-10-29)
Topic: Agents and multimodal systems
Format: Analysis

> In a typical retrieval-augmented answer, the generate call's input — retrieved chunks, conversation history, system prompt and tool definitions — dominates the model bill, while query embeddings are small and document embedding is a one-off per corpus version. Measure tokens per segment, order the prompt so the stable part can be cached, tune how many chunks you send against answer quality, and never report a model-call total as the cost of the whole pipeline.

Key takeaways:
- Most of a RAG answer's input is context you choose: chunks, history, system prompt and tools. The user's question is a rounding error.
- Embedding the query is small per request; re-embedding the corpus is a one-off cost per index version, and it belongs in a different budget line.
- Put stable content first and retrieved chunks after the history, so provider prompt caches can serve the prefix.
- More chunks cost more and can answer worse. Tune k against a quality metric, not by intuition.
- A gateway's run total covers model calls only. Vector database, reranking services and compute are separate lines.

Ask what a RAG answer costs and the usual reply is "one model call". It is at least three billable events — embed the question, search, generate — plus a one-off cost every time the corpus is re-indexed, and the generate call itself is mostly context your pipeline chose to send. The user's question is often the smallest thing in the prompt.

This analysis takes one answer apart, segment by segment, to show which parts dominate, which grow, and which a cost report tends to leave out.

## The billable steps

[Diagram: Who bills each step. Only the model calls reach a model gateway; the vector database and any non-chat reranker bill separately.]

**Document embeddings** are billed by input tokens; OpenAI's embeddings guide states that "requests are billed based on the number of tokens in the input"[1]. This is a corpus-sized, one-off cost per index version: you pay it when you first index and again whenever you change the embedding model, the chunking, or the documents. Vectors from different embedding models are not comparable, so a model change means re-embedding everything.

**Query embeddings** are billed the same way, per request, but a query is short. Per answer it is usually the smallest model-side line.

**Vector search** is not a model call at all. It is billed by your database vendor in queries, storage or compute, and no LLM gateway sees it.

**Reranking** depends on how you do it: a dedicated rerank endpoint bills per search or per token under its own terms; a chat model used as a reranker is an ordinary completion.

**Generation** is where the tokens are. Everything in the request counts: Anthropic's documentation lists the system prompt, every message including tool results and documents, and the tool definitions as counting toward the context window, with the output counting too[2]. OpenAI states that function definitions are "billed as input tokens"[3].

## One generate call, segment by segment

[Diagram: Illustrative input tokens for one answer, at 4 and at 12 retrieved chunks of 450 tokens. The question is 60 tokens either way; retrieved context is the segment that moves.]

> **Example: One answer at k = 4 and k = 12**
> Assume a 900-token system prompt, 600 tokens of tool definitions, 1,800 tokens of conversation history, chunks of 450 tokens, a 60-token question and a 500-token answer. Use illustrative rates of $2 per million input tokens, $0.20 per million cached input tokens and $10 per million output tokens.
> 
> At k = 4 the generate call sends 900 + 600 + 1,800 + 1,800 + 60 = 5,160 input tokens: $0.0103 of input and $0.0050 of output, about $15.32 per thousand answers.
> 
> At k = 12 it sends 8,760 input tokens: $0.0175 of input, the same $0.0050 of output, about $22.52 per thousand answers. Tripling k raised input by 70% and total by 47%.
> 
> If the prompt is ordered system, tools, history, then chunks and question, the first 3,300 tokens are identical to the previous turn of the same conversation and can be served from the provider's cache. At steady state that is 3,300 cached tokens ($0.00066) and 5,460 uncached ($0.0109): about $16.58 per thousand answers at k = 12. The first request of each prefix pays full price, or a premium to write the cache on providers that charge one.
> 
> A query embedding of about 20 tokens at an illustrative $0.02 per million costs $0.0000004 — invisible next to the generate call.

[Diagram: Illustrative estimated provider cost per thousand answers for the three configurations in the example above.]

Three things follow from the arithmetic.

**The question is noise; the context is the bill.** Optimising the user-facing prompt wording does almost nothing for cost. The levers are chunk count, chunk size, how much history you carry and how large your tool definitions are.

**Output is a big share per token and a small share of tokens.** Output rates are usually several times input rates, so a verbose answer format matters. Capping `max_tokens` and asking for concise answers is a real lever even though output is a tenth of the tokens.

**Order decides what caching can do.** Provider caches match on an exact prefix. OpenAI's guide says reuse "requires the entire rendered prefix to match" and advises putting stable instructions and reference material first[4]; Anthropic builds its cache prefix in the order tools, system, then messages[5]. Retrieved chunks change on every question, so anything placed after them cannot be reused. Put them after the history, not before it.

## Caching has thresholds and lifetimes

A cache that never hits is just a more complicated prompt. Two details decide whether a RAG prefix hits.

**Minimum length.** Prefixes below a minimum are not cached at all. OpenAI's current models require 1,024 visible input tokens before a prefix becomes cacheable[4]. Anthropic's minimum varies by model from 512 to 4,096 tokens, and a shorter prompt is processed without caching and without an error[5]. A 900-token system prompt alone may be below the line; system prompt plus tools plus history often is not.

**Lifetime.** Anthropic's default cache lives five minutes, refreshed each time it is read, and cache writes cost more than ordinary input — 1.25 times the base rate for the five-minute cache[5]. OpenAI's current models keep an entry for 30 minutes after its most recent use[4]. A support conversation with a reply every minute keeps its history warm; a nightly batch of unrelated questions shares only the system prompt and tools.

## More chunks is not free quality

Sending more retrieved text costs more on every request, and it does not reliably help. In "Lost in the Middle", Liu and colleagues found that performance is often highest when relevant information is at the beginning or end of the input, and "significantly degrades" when the model must use information in the middle of a long context[6]. A pipeline that pads the prompt with twelve chunks to be safe may pay 70% more input for answers that are no better, or worse.

Treat k, chunk size and reranking depth as parameters with a cost and a quality curve. Evaluate them on your own questions, measure cost per *correct* answer rather than cost per answer, and put the highest-ranked chunks at the edges of the retrieved block rather than in its middle.

## Measure tokens per segment

Provider usage tells you the total input tokens of a call. It does not tell you how many came from chunks versus history. To manage the segments you have to count them yourself, then reconcile against the provider's total.

```python
from dataclasses import dataclass

import tiktoken

# An approximation for segment shares only. Generation models may use other
# tokenizers; the provider-reported usage is the authoritative total.
enc = tiktoken.get_encoding("cl100k_base")


def n(text: str) -> int:
    return len(enc.encode(text))


@dataclass
class Segments:
    system: int
    tools: int
    history: int
    chunks: int
    question: int

    @property
    def total(self) -> int:
        return self.system + self.tools + self.history + self.chunks + self.question


def count_segments(system: str, tools_json: str, history: list[dict],
                   chunks: list[str], question: str) -> Segments:
    return Segments(
        system=n(system),
        tools=n(tools_json),
        history=sum(n(m["content"]) for m in history if isinstance(m.get("content"), str)),
        chunks=sum(n(c) for c in chunks),
        question=n(question),
    )


def attribute(seg: Segments, usage: dict) -> dict:
    """Scale local counts to the provider's reported prompt tokens."""
    reported = usage["prompt_tokens"]
    scale = reported / seg.total if seg.total else 0.0
    cached = (usage.get("prompt_tokens_details") or {}).get("cached_tokens", 0)
    return {
        "reported_prompt_tokens": reported,
        "cached_tokens": cached,
        "completion_tokens": usage["completion_tokens"],
        "share": {
            name: round(getattr(seg, name) * scale / reported, 3) if reported else 0.0
            for name in ("system", "tools", "history", "chunks", "question")
        },
        "overhead_tokens": max(reported - seg.total, 0),  # message framing, tool prompt
    }
```

Log the result of `attribute()` beside the request id for a sample of traffic, and chart the chunk and history shares over time. A chunk share that creeps up after a retrieval change, or a history share that grows without bound in long sessions, shows up there weeks before it shows up on an invoice. Where a provider offers a token-counting endpoint — Anthropic's accepts the same inputs as a message, including tools and documents, and returns an estimate[7] — use it to check the approximation for the model you actually call.

## Keeping RAG cost records in INFRO

INFRO records each model call routed through it with provider-reported token counts — it does not re-tokenize — an estimated provider cost in `usage.cost`, and `usage.prompt_tokens_details.cached_tokens` when the provider reports cache reads ([prompt caching](https://infro.io/docs/features/prompt-caching)). The estimate uses your negotiated rate when you have entered one, or the vendor's list rate; it is an estimate of what your provider will bill you, and your provider's invoice is authoritative.

Tag each call with its role so a pipeline reads as one run:

```json
{
  "model": "anthropic/claude-sonnet-5",
  "messages": ["…system, history, chunks, question…"],
  "max_tokens": 600,
  "metadata": {
    "trace_id": "ans_71c0e2",
    "workflow": "support-answer",
    "step": "generate",
    "feature": "help-center",
    "user_id": "cust_4821"
  }
}
```

`GET /v1/traces/{trace_id}` then returns the run's requests with estimated cost by step, and `GET /v1/usage?group_by=step` or `group_by=feature` aggregates across runs ([request traces](https://infro.io/docs/observability/traces), [cost attribution](https://infro.io/docs/features/attribution)).

Two limits of that picture are stated in the product itself. INFRO does not serve embeddings today, so embedding calls stay on your provider directly and are not in these records ([set up with an AI agent](https://infro.io/docs/agents)). And the run total says `"covers": "model_calls_through_infro"`: vector search, rerank services and your own compute are outside it, and `cost_is_partial` marks a total as a lower bound when any call was unpriced. Report RAG cost as model calls from the gateway plus embeddings, search and infrastructure from their own sources — never the first number alone.

## What to change first

1. **Measure segments** on a sample of real traffic before changing anything.
2. **Reorder the prompt**: system, tools, history, then chunks and question.
3. **Tune k and chunk size** against a quality metric, and try reranking to send fewer, better chunks.
4. **Bound history**: summarise or drop old turns once a session passes a token budget.
5. **Trim tool definitions** to the tools the step can actually use.
6. **Cap output** with `max_tokens` and an answer-length instruction.
7. **Separate the one-off**: book corpus re-embedding against the index version, not against queries.

## Questions

### How do I estimate the cost of a RAG query before building it?

Write down the token count of each prompt segment at your intended settings — system prompt, tool definitions, typical history, chunk size times k, question and expected output — multiply by your provider's rates for input, cached input and output, and add the per-query embedding call. Then replace the estimates with provider-reported usage from a pilot as soon as you have one.

## Sources

1. Vector embeddings — OpenAI. https://developers.openai.com/api/docs/guides/embeddings (accessed 2026-09-29)
2. Context windows — Anthropic. https://platform.claude.com/docs/en/build-with-claude/context-windows (accessed 2026-09-29)
3. Function calling — OpenAI. https://developers.openai.com/api/docs/guides/function-calling (accessed 2026-09-29)
4. Prompt caching — OpenAI. https://developers.openai.com/api/docs/guides/prompt-caching (accessed 2026-09-29)
5. Prompt caching — Anthropic. https://platform.claude.com/docs/en/build-with-claude/prompt-caching (accessed 2026-09-29)
6. Lost in the Middle: How Language Models Use Long Contexts — Liu et al., Transactions of the Association for Computational Linguistics. https://arxiv.org/abs/2307.03172 (accessed 2026-09-29)
7. Token counting — Anthropic. https://platform.claude.com/docs/en/build-with-claude/token-counting (accessed 2026-09-29)