Where the tokens go in a RAG pipeline

Ask what a RAG answer costs and the usual reply is "one model call". It is at least three billable events — embed the question, search, generate — plus a one-off cost every time the corpus is re-indexed, and the generate call itself is mostly context your pipeline chose to send. The user's question is often the smallest thing in the prompt.

This analysis takes one answer apart, segment by segment, to show which parts dominate, which grow, and which a cost report tends to leave out.

The billable steps

Document embeddings are billed by input tokens; OpenAI's embeddings guide states that "requests are billed based on the number of tokens in the input"¹. This is a corpus-sized, one-off cost per index version: you pay it when you first index and again whenever you change the embedding model, the chunking, or the documents. Vectors from different embedding models are not comparable, so a model change means re-embedding everything.

Query embeddings are billed the same way, per request, but a query is short. Per answer it is usually the smallest model-side line.

Vector search is not a model call at all. It is billed by your database vendor in queries, storage or compute, and no LLM gateway sees it.

Reranking depends on how you do it: a dedicated rerank endpoint bills per search or per token under its own terms; a chat model used as a reranker is an ordinary completion.

Generation is where the tokens are. Everything in the request counts: Anthropic's documentation lists the system prompt, every message including tool results and documents, and the tool definitions as counting toward the context window, with the output counting too². OpenAI states that function definitions are "billed as input tokens"³.

One generate call, segment by segment

Three things follow from the arithmetic.

The question is noise; the context is the bill. Optimising the user-facing prompt wording does almost nothing for cost. The levers are chunk count, chunk size, how much history you carry and how large your tool definitions are.

Output is a big share per token and a small share of tokens. Output rates are usually several times input rates, so a verbose answer format matters. Capping max_tokens and asking for concise answers is a real lever even though output is a tenth of the tokens.

Order decides what caching can do. Provider caches match on an exact prefix. OpenAI's guide says reuse "requires the entire rendered prefix to match" and advises putting stable instructions and reference material first⁴; Anthropic builds its cache prefix in the order tools, system, then messages⁵. Retrieved chunks change on every question, so anything placed after them cannot be reused. Put them after the history, not before it.

Caching has thresholds and lifetimes

A cache that never hits is just a more complicated prompt. Two details decide whether a RAG prefix hits.

Minimum length. Prefixes below a minimum are not cached at all. OpenAI's current models require 1,024 visible input tokens before a prefix becomes cacheable⁴. Anthropic's minimum varies by model from 512 to 4,096 tokens, and a shorter prompt is processed without caching and without an error⁵. A 900-token system prompt alone may be below the line; system prompt plus tools plus history often is not.

Lifetime. Anthropic's default cache lives five minutes, refreshed each time it is read, and cache writes cost more than ordinary input — 1.25 times the base rate for the five-minute cache⁵. OpenAI's current models keep an entry for 30 minutes after its most recent use⁴. A support conversation with a reply every minute keeps its history warm; a nightly batch of unrelated questions shares only the system prompt and tools.

More chunks is not free quality

Sending more retrieved text costs more on every request, and it does not reliably help. In "Lost in the Middle", Liu and colleagues found that performance is often highest when relevant information is at the beginning or end of the input, and "significantly degrades" when the model must use information in the middle of a long context⁶. A pipeline that pads the prompt with twelve chunks to be safe may pay 70% more input for answers that are no better, or worse.

Treat k, chunk size and reranking depth as parameters with a cost and a quality curve. Evaluate them on your own questions, measure cost per correct answer rather than cost per answer, and put the highest-ranked chunks at the edges of the retrieved block rather than in its middle.

Measure tokens per segment

Provider usage tells you the total input tokens of a call. It does not tell you how many came from chunks versus history. To manage the segments you have to count them yourself, then reconcile against the provider's total.

from dataclasses import dataclass

import tiktoken

# An approximation for segment shares only. Generation models may use other
# tokenizers; the provider-reported usage is the authoritative total.
enc = tiktoken.get_encoding("cl100k_base")


def n(text: str) -> int:
    return len(enc.encode(text))


@dataclass
class Segments:
    system: int
    tools: int
    history: int
    chunks: int
    question: int

    @property
    def total(self) -> int:
        return self.system + self.tools + self.history + self.chunks + self.question


def count_segments(system: str, tools_json: str, history: list[dict],
                   chunks: list[str], question: str) -> Segments:
    return Segments(
        system=n(system),
        tools=n(tools_json),
        history=sum(n(m["content"]) for m in history if isinstance(m.get("content"), str)),
        chunks=sum(n(c) for c in chunks),
        question=n(question),
    )


def attribute(seg: Segments, usage: dict) -> dict:
    """Scale local counts to the provider's reported prompt tokens."""
    reported = usage["prompt_tokens"]
    scale = reported / seg.total if seg.total else 0.0
    cached = (usage.get("prompt_tokens_details") or {}).get("cached_tokens", 0)
    return {
        "reported_prompt_tokens": reported,
        "cached_tokens": cached,
        "completion_tokens": usage["completion_tokens"],
        "share": {
            name: round(getattr(seg, name) * scale / reported, 3) if reported else 0.0
            for name in ("system", "tools", "history", "chunks", "question")
        },
        "overhead_tokens": max(reported - seg.total, 0),  # message framing, tool prompt
    }

Log the result of attribute() beside the request id for a sample of traffic, and chart the chunk and history shares over time. A chunk share that creeps up after a retrieval change, or a history share that grows without bound in long sessions, shows up there weeks before it shows up on an invoice. Where a provider offers a token-counting endpoint — Anthropic's accepts the same inputs as a message, including tools and documents, and returns an estimate⁷ — use it to check the approximation for the model you actually call.

Keeping RAG cost records in INFRO

INFRO records each model call routed through it with provider-reported token counts — it does not re-tokenize — an estimated provider cost in usage.cost, and usage.prompt_tokens_details.cached_tokens when the provider reports cache reads (prompt caching). The estimate uses your negotiated rate when you have entered one, or the vendor's list rate; it is an estimate of what your provider will bill you, and your provider's invoice is authoritative.

Tag each call with its role so a pipeline reads as one run:

{
  "model": "anthropic/claude-sonnet-5",
  "messages": ["…system, history, chunks, question…"],
  "max_tokens": 600,
  "metadata": {
    "trace_id": "ans_71c0e2",
    "workflow": "support-answer",
    "step": "generate",
    "feature": "help-center",
    "user_id": "cust_4821"
  }
}

GET /v1/traces/{trace_id} then returns the run's requests with estimated cost by step, and GET /v1/usage?group_by=step or group_by=feature aggregates across runs (request traces, cost attribution).

Two limits of that picture are stated in the product itself. INFRO does not serve embeddings today, so embedding calls stay on your provider directly and are not in these records (set up with an AI agent). And the run total says "covers": "model_calls_through_infro": vector search, rerank services and your own compute are outside it, and cost_is_partial marks a total as a lower bound when any call was unpriced. Report RAG cost as model calls from the gateway plus embeddings, search and infrastructure from their own sources — never the first number alone.

What to change first

1. Measure segments on a sample of real traffic before changing anything.
2. Reorder the prompt: system, tools, history, then chunks and question.
3. Tune k and chunk size against a quality metric, and try reranking to send fewer, better chunks.
4. Bound history: summarise or drop old turns once a session passes a token budget.
5. Trim tool definitions to the tools the step can actually use.
6. Cap output with max_tokens and an answer-length instruction.
7. Separate the one-off: book corpus re-embedding against the index version, not against queries.