# Do you need a reranker? Testing Cohere Rerank 5 Pro vs Fast

URL: https://infro.io/blog/do-you-need-a-reranker-testing-cohere-rerank-5-pro-vs-fast
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-10-29
Last modified: 2026-10-29
Last reviewed: 2026-10-29 (next review due 2026-11-28)
Topic: Agents and multimodal systems
Format: Methodology

> Add a reranker only if your gold passage is usually in the candidate set but not in the few chunks you send to the model. Test five arms on one labeled set and compare cost per successful answer and latency per stage, then pick Pro or Fast on the measured margin.

Key takeaways:
- A reranker only helps when recall in the candidate set is high and recall in the top few chunks is low; measure both before buying anything.
- Compare five arms on one labeled query set: baseline, Rerank 5 Fast, Rerank 5 Pro, a larger generator, and a longer context.
- Rank arms by total spend across every stage divided by correct, grounded answers, and record latency for each stage separately.
- Pro versus Fast is a marginal question: price the extra cost per additional correct answer and compare it with what a wrong answer costs you.
- Calibrated scores and Context Focus are separate experiments from the tier choice; test thresholds and span extraction as their own arms.
- Cohere's benchmark and price figures are vendor-published, so confirm the billed units on your own responses.

## Start with where your retrieval fails

A reranker is a second-stage scorer. A first stage (keyword, vector or hybrid search) returns a wide candidate set, and the reranker reads the query and each candidate together and reorders them. It cannot find a passage the first stage never returned. So the decision to add one rests on two numbers you can measure before spending anything: how often the gold passage is anywhere in the candidates, and how often it is in the few chunks you actually send to the model.

If recall at 100 candidates is low, fix chunking, the embedding model or the query before touching rerank. If recall at 100 is high and recall at 5 is much lower, the evidence is retrieved and then buried. That gap is what a reranker, a larger model, or a longer context would each try to cover in a different way. The test below prices all three on the same questions.

This article covers the test method. For where the tokens go in a pipeline, see [Where the tokens go in a RAG pipeline](https://infro.io/blog/rag-token-accounting); for the first-stage model choice, see [Embedding models compared in 2026](https://infro.io/blog/embedding-models-compared-2026).

## What Rerank 5 gives you to test

Cohere's Rerank 5 family has two tiers on the same endpoint and request format. `rerank-v5.0-pro` targets the highest ranking quality, complex or reasoning-heavy queries and long or domain-specific documents. `rerank-v5.0-fast` targets low-latency, high-throughput work such as high-volume search and agent loops. Cohere lists about 2B parameters for Pro and about 500M for Fast, a 32,768-token context shared by query and document, input as text only, and calibrated scores between 0 and 1.[1]

Details that change how you design the test:

- **Candidate volume.** Up to 10,000 documents per request on v5; long documents are truncated to `max_tokens_per_doc`, which defaults to 4096.[2]
- **Context Focus.** Adding `relevant_spans` to the request returns the most relevant character spans within each document, with a threshold and a `max_spans` cap. It is off by default, and Cohere says it works best on targeted queries, not summarization, and splits tables, forms and code less cleanly than prose.[1]
- **No instructions.** Rerank 5 does not support instructions that change how relevance is judged.[1]
- **No re-indexing.** Rerank reads raw text at query time, so the corpus is untouched.[3]

Cohere's announcement lists API pricing of $0.05 per 1M tokens for Pro and $0.02 for Fast, and reports that Pro leads public benchmarks while Fast outperforms the previous-generation Rerank 4 Pro on its four headline benchmarks.[4] Those benchmark scores use Cohere's own RCP-nDCG@10 methodology, so treat them as a reason to include both tiers in your test, not as a result for your data.

> **Warning: Check the billing unit on your own account**
> Cohere's pricing explainer says rerank models are priced on the quantity of searches, while the Rerank 5 announcement quotes a per-token rate.[5][4] Do not assume which applies. Log the billed units the rerank response reports and reconcile them with your invoice before you trust any cost-per-answer figure.

## The five arms

Run every arm on the same labeled queries, with the same generator prompt and the same grading. Change one thing per arm.

| Arm | First stage | What goes to the generator | Generator |
| --- | --- | --- | --- |
| A. Baseline | Top 20 | 20 chunks | Current model |
| B. Fast rerank | Top 100 | Top 5 after rerank-v5.0-fast | Current model |
| C. Pro rerank | Top 100 | Top 5 after rerank-v5.0-pro | Current model |
| D. Larger model | Top 20 | 20 chunks | Next tier up |
| E. Longer context | Top 50 | 50 chunks | Current model |

Table: Arms for the rerank decision. The counts are starting points; use your own k.

Two optional variants run afterwards on the winner. One replaces "top 5" with a score threshold. The other enables Context Focus and sends spans instead of whole chunks. Keep them out of the first round so the tier comparison stays clean.

If you retrieve 100 candidates for arms B and C, also run arms A, D and E with the first stage widened to 100 as a recall check. That tells you whether the gain came from wider retrieval or from reranking.

## Step 1: build the labeled set

You need queries with a known gold passage and a known correct answer. A few hundred is a workable start; fewer makes the confidence intervals too wide to separate Pro from Fast. Sample from real traffic, not from questions written by the people who built the index, and include the hard cases: multi-hop questions, near-duplicate documents, and queries with no answer in the corpus.

For each query record:

1. The gold passage ids.
2. A reference answer, or a rubric a grader can apply.
3. Whether the correct behavior is to abstain.

Then measure retrieval alone, before any generation: recall at 5, 20, 50 and 100 for the first stage, and recall at 5 after each reranker. This costs almost nothing and answers the first question. If reranking lifts recall at 5 to near your recall at 100, the reranker is doing its job. If recall at 100 is already low, no arm B or C result will rescue it.

## Step 2: instrument every stage

Measure latency and spend per stage, per query, not end to end only. An end-to-end number cannot tell you whether the reranker's added round trip was paid back by a shorter prompt.

```python
import time, requests

def timed(fn, *a, **kw):
    t = time.perf_counter()
    out = fn(*a, **kw)
    return out, (time.perf_counter() - t) * 1000

def rerank(query, docs, model, top_n, api_key):
    r = requests.post(
        "https://api.cohere.com/v2/rerank",
        headers={"Authorization": f"bearer {api_key}"},
        json={"model": model, "query": query,
              "documents": docs, "top_n": top_n},
        timeout=30,
    )
    r.raise_for_status()
    body = r.json()
    return body["results"], body.get("meta", {}).get("billed_units")

def run_query(q, arm, retrieve, rerank_fn, generate):
    row = {"query_id": q.id, "arm": arm.name}
    cands, row["retrieve_ms"] = timed(retrieve, q.text, arm.k_first)
    if arm.rerank_model:
        (ranked, billed), row["rerank_ms"] = timed(
            rerank_fn, q.text, [c.text for c in cands],
            arm.rerank_model, arm.k_final)
        row["rerank_billed"] = billed
        cands = [cands[r["index"]] for r in ranked]
    ans, row["generate_ms"] = timed(generate, q.text, cands[:arm.k_final])
    row["input_tokens"] = ans.usage.input_tokens
    row["output_tokens"] = ans.usage.output_tokens
    row["answer"] = ans.text
    return row
```

Store the generator's own reported token counts and, separately, whatever billed units the rerank response returns. For the generator, also record time to first token if you stream, since a shorter prompt mostly helps prefill. [Measuring time to first token](https://infro.io/blog/measure-time-to-first-token) covers that metric.

Report p50 and p95 per stage. Averages hide the tail that users notice, and a reranker adds a network call whose tail you now own.

## Step 3: compute cost per successful answer

Raw cost per query rewards the arm that answers worst. The metric to rank by is:

```text
cost_per_successful_answer =
  sum(all stage costs over all queries, including retries and failures)
  / count(answers graded correct and grounded)
```

Stage costs are: first-stage retrieval (embedding of the query, vector search), rerank, generation input and output, and the grader if you count it in production. A wrong answer's cost still counts in the numerator. This is the same idea as [cost per successful task](https://infro.io/blog/cost-per-successful-task), applied to one pipeline.

> **Example: Illustrative arithmetic, not measured results**
> Every number here is made up to show the mechanics. Assume 400-token chunks, a generator input rate of $2.50 per 1M tokens for the current model and $10 per 1M for the larger one, output cost held equal across arms and left out, and rerank at the per-token list rates Cohere announced ($0.05 per 1M for Pro, $0.02 for Fast), applied to 100 candidates × 400 tokens = 40,000 tokens per query.
> 
> | Arm | Rerank cost | Generation input | Total per query | Correct | Cost per correct answer |
> | --- | --- | --- | --- | --- | --- |
> | A. Baseline (20 chunks) | $0 | $0.0200 | $0.0200 | 70% | $0.0286 |
> | B. Fast, top 5 | $0.0008 | $0.0050 | $0.0058 | 78% | $0.0074 |
> | C. Pro, top 5 | $0.0020 | $0.0050 | $0.0070 | 80% | $0.0088 |
> | D. Larger model | $0 | $0.0800 | $0.0800 | 74% | $0.1081 |
> | E. 50 chunks | $0 | $0.0500 | $0.0500 | 72% | $0.0694 |
> 
> In this made-up run the rerank arms win because they cut the prompt by 75% and lift accuracy. Pro costs $0.0012 more per query than Fast for two more points of accuracy, which is $0.06 per additional correct answer: (0.0070 − 0.0058) / (0.80 − 0.78). Whether that is worth paying depends on what a wrong answer costs you.

The marginal calculation is the Pro-versus-Fast decision. Compute extra cost per query divided by extra correct answers per query, and compare it with your cost of an error: a support escalation, a wrong figure in a report, a lost sale. If an error costs you more than the marginal number, take Pro. If it does not, take Fast. Segment the result: Cohere positions Pro for complex queries and long or domain-specific documents, so the gap may be concentrated in one query class and not spread evenly.[1] A routing rule that sends only that class to Pro is a third option you can test.

## Step 4: read the latency stage by stage

Lay the stage latencies side by side.

> **Example: Illustrative stage latencies, not measurements**
> | Arm | Retrieve p50 | Rerank p50 | Generate p50 | Total p50 |
> | --- | --- | --- | --- | --- |
> | A. Baseline | 40 ms | 0 | 900 ms | 940 ms |
> | B. Fast | 45 ms | 70 ms | 700 ms | 815 ms |
> | C. Pro | 45 ms | 150 ms | 700 ms | 895 ms |
> | D. Larger model | 40 ms | 0 | 1,400 ms | 1,440 ms |
> | E. 50 chunks | 50 ms | 0 | 1,300 ms | 1,350 ms |
> 
> These figures are invented. They show a pattern you should check, not one you should expect: a reranker's added call can be repaid by a shorter prompt, and the larger model and the longer context both add to the generate stage.

Cohere's pages I read do not publish latency figures for either tier, so the only numbers that count are yours, from your network path and your document sizes. Also measure under concurrency: the 10,000-document limit is per request, and rate limits apply to the Cohere API, so a load test belongs here before you commit an agent loop that reranks on every step.[1][2]

## Step 5: the two follow-up experiments

**Score threshold.** Cohere describes the calibrated scores as consistent across queries and domains, so one threshold can decide what reaches the LLM or trigger an "I don't know".[4] Its best-practices guide cautions that a score is query dependent and that 0.91 is not twice as relevant as 0.04. It recommends picking 30-50 representative queries, pairing each with a borderline-relevant document, and using the scores to set the cutoff.[2] Validate that on your labeled set instead: sweep the threshold, plot correct-answer rate against abstain rate, and include your no-answer queries. A threshold that cuts prompts but abstains on answerable questions raises cost per successful answer.

**Context Focus.** Enable `relevant_spans` on the winning tier and send only the spans. Cohere says this cuts LLM token cost with minimal loss in answer quality; verify that on your own questions.[4] Grade answers that need context outside the span, such as definitions two paragraphs above a figure. Cohere's own caveat is that broad requests need the full document.[1]

## Failure modes to look for

- **Low first-stage recall.** The reranker reorders what it is given. If the gold passage is not in the candidates, every rerank arm loses.
- **Truncation.** With the default `max_tokens_per_doc` of 4096, a long document is cut before scoring, so the relevant part may never be seen.[2] Chunk long documents yourself if recall drops on them.
- **Structured data order.** Cohere says YAML key order matters for semi-structured documents because long strings are truncated.[2] Put the identifying fields first.
- **Unmatched grading.** If the larger-model arm is graded by a different rubric than the others, the comparison is void. Use one grader for all arms.
- **Benchmark transfer.** Public scores come from the vendor's methodology; a gain on BEIR is not a gain on your tickets.
- **Billing-unit surprise.** See the warning above; reconcile the unit on a small paid run first.

## A decision-model arm, if you want one

INFRO's `POST /v1/rerank` uses Cohere's request shape, but it is served by a decision model such as Jev or another decision model you connect, not by Cohere Rerank 5. Each passage is judged alone against the query and billed by your own provider account per input token, with a limit of 200 passages per request.[6] That makes it a possible sixth arm if you want to compare a decision-model reranker with a dedicated one, on the same labeled set and the same cost formula. For the generation stage, per-request model, latency and estimated provider cost are recorded in the request log, queryable by project and key, which supplies the generate-stage columns in the tables above. The Cohere rerank call itself still has to be timed in your own code.

## Decision rules

- [ ] Recall at 100 is high (your threshold) and recall at 5 is clearly lower: add a reranker.
- [ ] Recall at 100 is low: fix retrieval first; do not run the rerank arms yet.
- [ ] A rerank arm beats the larger-model and longer-context arms on cost per correct answer on the same grading.
- [ ] Choose Fast when the extra cost per additional correct answer for Pro exceeds your cost of an error.
- [ ] Choose Pro where the gain is concentrated, such as long, domain-specific or reasoning-heavy queries, and consider routing only those.
- [ ] Ship a threshold or Context Focus only after the sweep on your own no-answer queries.
- [ ] Re-run the test when you change the embedding model, chunking, the generator or the rerank tier, and when Cohere's rates change.

## Questions

### Does upgrading to Rerank 5 require re-indexing?

No. Rerank reads raw text at query time, so moving from an earlier Rerank model means changing the model parameter to rerank-v5.0-pro or rerank-v5.0-fast; Context Focus stays off until you enable it.[3]

### Can I use the score as an "I don't know" threshold?

Cohere positions the calibrated scores for that, but its best-practices guide says to derive a threshold from 30-50 representative queries of your own and not to read the number as a proportion.[1][2]

## Sources

1. Cohere Rerank (model documentation) — Cohere. https://docs.cohere.com/docs/rerank (accessed 2026-10-29)
2. Best Practices for using Rerank — Cohere. https://docs.cohere.com/docs/reranking-best-practices (accessed 2026-10-29)
3. Announcing Cohere's Rerank 5 Models — Cohere. https://docs.cohere.com/changelog/rerank-v5 (accessed 2026-10-29)
4. Introducing Rerank 5: precise, efficient enterprise document retrieval — Cohere. https://cohere.com/blog/rerank-5 (accessed 2026-10-29)
5. How Does Cohere's Pricing Work? — Cohere. https://docs.cohere.com/docs/how-does-cohere-pricing-work (accessed 2026-10-29)
6. Rerank: API reference — INFRO. https://infro.io/docs/api/rerank (accessed 2026-10-29)