Do you need a reranker? Testing Cohere Rerank 5 Pro vs Fast

Start with where your retrieval fails

A reranker is a second-stage scorer. A first stage (keyword, vector or hybrid search) returns a wide candidate set, and the reranker reads the query and each candidate together and reorders them. It cannot find a passage the first stage never returned. So the decision to add one rests on two numbers you can measure before spending anything: how often the gold passage is anywhere in the candidates, and how often it is in the few chunks you actually send to the model.

If recall at 100 candidates is low, fix chunking, the embedding model or the query before touching rerank. If recall at 100 is high and recall at 5 is much lower, the evidence is retrieved and then buried. That gap is what a reranker, a larger model, or a longer context would each try to cover in a different way. The test below prices all three on the same questions.

This article covers the test method. For where the tokens go in a pipeline, see Where the tokens go in a RAG pipeline; for the first-stage model choice, see Embedding models compared in 2026.

What Rerank 5 gives you to test

Cohere's Rerank 5 family has two tiers on the same endpoint and request format. rerank-v5.0-pro targets the highest ranking quality, complex or reasoning-heavy queries and long or domain-specific documents. rerank-v5.0-fast targets low-latency, high-throughput work such as high-volume search and agent loops. Cohere lists about 2B parameters for Pro and about 500M for Fast, a 32,768-token context shared by query and document, input as text only, and calibrated scores between 0 and 1.¹

Details that change how you design the test:

• Candidate volume. Up to 10,000 documents per request on v5; long documents are truncated to max_tokens_per_doc, which defaults to 4096.²
• Context Focus. Adding relevant_spans to the request returns the most relevant character spans within each document, with a threshold and a max_spans cap. It is off by default, and Cohere says it works best on targeted queries, not summarization, and splits tables, forms and code less cleanly than prose.¹
• No instructions. Rerank 5 does not support instructions that change how relevance is judged.¹
• No re-indexing. Rerank reads raw text at query time, so the corpus is untouched.³

Cohere's announcement lists API pricing of $0.05 per 1M tokens for Pro and $0.02 for Fast, and reports that Pro leads public benchmarks while Fast outperforms the previous-generation Rerank 4 Pro on its four headline benchmarks.⁴ Those benchmark scores use Cohere's own RCP-nDCG@10 methodology, so treat them as a reason to include both tiers in your test, not as a result for your data.

The five arms

Run every arm on the same labeled queries, with the same generator prompt and the same grading. Change one thing per arm.

Two optional variants run afterwards on the winner. One replaces "top 5" with a score threshold. The other enables Context Focus and sends spans instead of whole chunks. Keep them out of the first round so the tier comparison stays clean.

If you retrieve 100 candidates for arms B and C, also run arms A, D and E with the first stage widened to 100 as a recall check. That tells you whether the gain came from wider retrieval or from reranking.

Step 1: build the labeled set

You need queries with a known gold passage and a known correct answer. A few hundred is a workable start; fewer makes the confidence intervals too wide to separate Pro from Fast. Sample from real traffic, not from questions written by the people who built the index, and include the hard cases: multi-hop questions, near-duplicate documents, and queries with no answer in the corpus.

For each query record:

1. The gold passage ids.
2. A reference answer, or a rubric a grader can apply.
3. Whether the correct behavior is to abstain.

Then measure retrieval alone, before any generation: recall at 5, 20, 50 and 100 for the first stage, and recall at 5 after each reranker. This costs almost nothing and answers the first question. If reranking lifts recall at 5 to near your recall at 100, the reranker is doing its job. If recall at 100 is already low, no arm B or C result will rescue it.

Step 2: instrument every stage

Measure latency and spend per stage, per query, not end to end only. An end-to-end number cannot tell you whether the reranker's added round trip was paid back by a shorter prompt.

import time, requests

def timed(fn, *a, **kw):
    t = time.perf_counter()
    out = fn(*a, **kw)
    return out, (time.perf_counter() - t) * 1000

def rerank(query, docs, model, top_n, api_key):
    r = requests.post(
        "https://api.cohere.com/v2/rerank",
        headers={"Authorization": f"bearer {api_key}"},
        json={"model": model, "query": query,
              "documents": docs, "top_n": top_n},
        timeout=30,
    )
    r.raise_for_status()
    body = r.json()
    return body["results"], body.get("meta", {}).get("billed_units")

def run_query(q, arm, retrieve, rerank_fn, generate):
    row = {"query_id": q.id, "arm": arm.name}
    cands, row["retrieve_ms"] = timed(retrieve, q.text, arm.k_first)
    if arm.rerank_model:
        (ranked, billed), row["rerank_ms"] = timed(
            rerank_fn, q.text, [c.text for c in cands],
            arm.rerank_model, arm.k_final)
        row["rerank_billed"] = billed
        cands = [cands[r["index"]] for r in ranked]
    ans, row["generate_ms"] = timed(generate, q.text, cands[:arm.k_final])
    row["input_tokens"] = ans.usage.input_tokens
    row["output_tokens"] = ans.usage.output_tokens
    row["answer"] = ans.text
    return row

Store the generator's own reported token counts and, separately, whatever billed units the rerank response returns. For the generator, also record time to first token if you stream, since a shorter prompt mostly helps prefill. Measuring time to first token covers that metric.

Report p50 and p95 per stage. Averages hide the tail that users notice, and a reranker adds a network call whose tail you now own.

Step 3: compute cost per successful answer

Raw cost per query rewards the arm that answers worst. The metric to rank by is:

cost_per_successful_answer =
  sum(all stage costs over all queries, including retries and failures)
  / count(answers graded correct and grounded)

Stage costs are: first-stage retrieval (embedding of the query, vector search), rerank, generation input and output, and the grader if you count it in production. A wrong answer's cost still counts in the numerator. This is the same idea as cost per successful task, applied to one pipeline.

The marginal calculation is the Pro-versus-Fast decision. Compute extra cost per query divided by extra correct answers per query, and compare it with your cost of an error: a support escalation, a wrong figure in a report, a lost sale. If an error costs you more than the marginal number, take Pro. If it does not, take Fast. Segment the result: Cohere positions Pro for complex queries and long or domain-specific documents, so the gap may be concentrated in one query class and not spread evenly.¹ A routing rule that sends only that class to Pro is a third option you can test.

Step 4: read the latency stage by stage

Lay the stage latencies side by side.

Cohere's pages I read do not publish latency figures for either tier, so the only numbers that count are yours, from your network path and your document sizes. Also measure under concurrency: the 10,000-document limit is per request, and rate limits apply to the Cohere API, so a load test belongs here before you commit an agent loop that reranks on every step.¹²

Step 5: the two follow-up experiments

Score threshold. Cohere describes the calibrated scores as consistent across queries and domains, so one threshold can decide what reaches the LLM or trigger an "I don't know".⁴ Its best-practices guide cautions that a score is query dependent and that 0.91 is not twice as relevant as 0.04. It recommends picking 30-50 representative queries, pairing each with a borderline-relevant document, and using the scores to set the cutoff.² Validate that on your labeled set instead: sweep the threshold, plot correct-answer rate against abstain rate, and include your no-answer queries. A threshold that cuts prompts but abstains on answerable questions raises cost per successful answer.

Context Focus. Enable relevant_spans on the winning tier and send only the spans. Cohere says this cuts LLM token cost with minimal loss in answer quality; verify that on your own questions.⁴ Grade answers that need context outside the span, such as definitions two paragraphs above a figure. Cohere's own caveat is that broad requests need the full document.¹

Failure modes to look for

• Low first-stage recall. The reranker reorders what it is given. If the gold passage is not in the candidates, every rerank arm loses.
• Truncation. With the default max_tokens_per_doc of 4096, a long document is cut before scoring, so the relevant part may never be seen.² Chunk long documents yourself if recall drops on them.
• Structured data order. Cohere says YAML key order matters for semi-structured documents because long strings are truncated.² Put the identifying fields first.
• Unmatched grading. If the larger-model arm is graded by a different rubric than the others, the comparison is void. Use one grader for all arms.
• Benchmark transfer. Public scores come from the vendor's methodology; a gain on BEIR is not a gain on your tickets.
• Billing-unit surprise. See the warning above; reconcile the unit on a small paid run first.

A decision-model arm, if you want one

INFRO's POST /v1/rerank uses Cohere's request shape, but it is served by a decision model such as Jev or another decision model you connect, not by Cohere Rerank 5. Each passage is judged alone against the query and billed by your own provider account per input token, with a limit of 200 passages per request.⁶ That makes it a possible sixth arm if you want to compare a decision-model reranker with a dedicated one, on the same labeled set and the same cost formula. For the generation stage, per-request model, latency and estimated provider cost are recorded in the request log, queryable by project and key, which supplies the generate-stage columns in the tables above. The Cohere rerank call itself still has to be timed in your own code.

Decision rules

• Recall at 100 is high (your threshold) and recall at 5 is clearly lower: add a reranker.
• Recall at 100 is low: fix retrieval first; do not run the rerank arms yet.
• A rerank arm beats the larger-model and longer-context arms on cost per correct answer on the same grading.
• Choose Fast when the extra cost per additional correct answer for Pro exceeds your cost of an error.
• Choose Pro where the gain is concentrated, such as long, domain-specific or reasoning-heavy queries, and consider routing only those.
• Ship a threshold or Context Focus only after the sweep on your own no-answer queries.
• Re-run the test when you change the embedding model, chunking, the generator or the rerank tier, and when Cohere's rates change.