# Can an LLM judge be trusted? Calibrating graders against humans

URL: https://infro.io/blog/can-an-llm-judge-be-trusted-calibrating-graders-against-humans
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-10-31
Last modified: 2026-10-31
Last reviewed: 2026-10-31 (next review due 2027-04-29)
Topic: Evaluation and models
Format: Methodology

> An LLM judge can be trusted for a specific rubric, on a specific kind of output, to the degree that you have measured its agreement with human labels on that output. Sample humans in strata, report agreement with intervals, probe for bias, pin and re-check the judge, and gate a model switch on a confidence bound, not a raw average.

Key takeaways:
- A judge is trusted per rubric and per output type, only to the extent you have measured its agreement with human labels on that output.
- Sample humans in strata, label a share twice, and split labels into a tuning set and a held-out test set before you touch the judge prompt.
- Report chance-corrected agreement, false-pass and false-fail rates, and intervals; raw accuracy hides a lenient judge on a skewed dataset.
- Probe the judge for position, length, self-preference and format bias with controlled edits, and pin the judge to a dated model.
- Gate a model switch on a lower confidence bound of the paired difference, and re-check judge agreement on the candidate's own outputs.

## A judge is an instrument, so calibrate it

OpenAI's hosted Evals platform is closing: existing evals become read-only on 31 October 2026 and the dashboard and API are scheduled to shut down on 30 November 2026.[1] Teams that used model-graded checks as a regression gate are now writing their own LLM-as-judge graders. The danger in that migration is quiet. A rebuilt grader will produce numbers, the numbers will look plausible, and nobody will know whether they track what a person would decide.

The migration advice OpenAI gives is itself a warning: recreated LLM-as-a-judge graders should be validated before you rely on them, because scores will not be identical across systems.[1] Anthropic's evaluation guide lists LLM-based grading beside exact match and code grading as one way to automate, and treats a Likert-scale judge as suited to subjective qualities such as tone.[2] Neither tells you how far to trust the score. This method does.

The claim you are trying to earn is narrow: *for this rubric, on outputs like these, the judge agrees with trained humans at least this often, and its errors lean in a known direction.* Anything wider than that is a guess. The steps are:

1. Fix the rubric and the unit of judgement.
2. Sample items and collect human labels, with a share labelled twice.
3. Measure agreement and error direction, with intervals.
4. Probe for bias with controlled edits.
5. Set a gate that uses the measured error.
6. Pin the judge and watch for drift.

## Fix the rubric before you sample anything

Most disagreement between a judge and a human is really disagreement about the rubric. Before collecting labels, write the criterion so that two people could apply it without talking to each other. Anthropic's guidance on success criteria asks for criteria that are specific and measurable, and notes that even hazy qualities can be quantified when defined precisely.[2] Apply the same standard to each judge question.

Prefer a binary or small-ordinal verdict per criterion over a 1–10 score. Fine-grained scales produce low agreement among humans, which caps how well any judge can do. Split compound rubrics ("accurate, concise and polite") into one question per judge call, so you can see which dimension fails.

Include the hard cases on purpose. Anthropic's eval design notes name ambiguous test cases, where even humans would struggle to reach consensus, as a category to include.[2] Keep them, but tag them. If humans disagree with each other on an item, a judge disagreeing with one of them is not evidence of a bad judge.

Decide the unit of judgement as well. Pointwise grading (is this answer acceptable?) is simpler to calibrate than pairwise grading (which of A or B is better?), but pairwise is what a model switch usually needs. If you use pairwise, you must also test for position bias, covered below.

## Sample human labels so they represent the decision

A random sample of last week's traffic over-represents easy, common cases and under-represents the failures the gate exists to catch. Sample in strata instead.

- **By input type.** Intent, language, length bucket, tool use, customer tier; whatever your product splits on.
- **By judge verdict.** Draw items the judge passed and items it failed in comparable numbers. If 95% of items pass, a random sample holds few failures, and you cannot estimate the false-pass rate.
- **By producing model.** Include outputs from the current model, the candidate and, if you have one, a deliberately weaker model. The judge must discriminate across that range.
- **Near-threshold items.** If the judge returns a score or probability, oversample the middle of its range, where mistakes concentrate.

Then split the labelled set. Keep a **tuning set** for editing the judge prompt and a **held-out test set** that nobody looks at until the judge is frozen. Reusing the same items for both inflates agreement, just as it does for any model.

Label a share of items twice, by different people, to measure human-human agreement. That number is your ceiling. A judge that agrees with a single annotator 85% of the time when two annotators agree 86% of the time is performing at the limit of the rubric.

> **Example: Sample size, illustrative**
> With the normal approximation, the 95% interval on an agreement rate p from n items has half-width about 1.96 × √(p(1−p)/n). The figures below are arithmetic, not measurements of any judge.
> 
> | Expected agreement | Half-width | Items needed |
> | --- | --- | --- |
> | 90% | ±5 points | ~140 |
> | 90% | ±10 points | ~35 |
> | 80% | ±5 points | ~250 |
> 
> Per-slice estimates multiply these numbers. Under about 30 items in a slice, read the items instead of quoting a rate.

## Measure agreement and the direction of error

Raw accuracy misleads on skewed data. If 90% of answers are acceptable, a judge that passes everything is 90% "accurate". Report these together:

| Measure | What it tells you | Watch for |
| --- | --- | --- |
| Percent agreement | Plain match rate with the human majority label | High on skewed data even for a useless judge |
| Cohen's kappa (or Krippendorff's alpha for ordinal) | Agreement beyond chance | Sensitive to prevalence; read it with the confusion matrix |
| False-pass rate | Judge passes, humans fail | The costly error for a safety or quality gate |
| False-fail rate | Judge fails, humans pass | Wastes review time and blocks good candidates |
| Human-human agreement | Ceiling set by the rubric | A low ceiling means fix the rubric first |
| Rank correlation across models | Does the judge order models as humans do | The property a model switch depends on |

The last row matters more than it looks. A gate compares two models, so a judge with a constant lenient offset can still rank them correctly. A judge that is lenient only on one model's style cannot.

```python
import math
from collections import Counter

def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
    """95% Wilson interval for a proportion."""
    if n == 0:
        return (0.0, 1.0)
    p = k / n
    d = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / d
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return (centre - half, centre + half)

def report(human: list[int], judge: list[int]) -> dict:
    """human, judge: 1 = pass, 0 = fail, same item order."""
    n = len(human)
    tp = sum(h == 1 and j == 1 for h, j in zip(human, judge))
    tn = sum(h == 0 and j == 0 for h, j in zip(human, judge))
    fp = sum(h == 0 and j == 1 for h, j in zip(human, judge))  # false pass
    fn = sum(h == 1 and j == 0 for h, j in zip(human, judge))  # false fail
    po = (tp + tn) / n
    ph, pj = Counter(human), Counter(judge)
    pe = sum((ph[c] / n) * (pj[c] / n) for c in (0, 1))
    kappa = (po - pe) / (1 - pe) if pe < 1 else float("nan")
    return {
        "agreement": (po, wilson(tp + tn, n)),
        "kappa": kappa,
        "sensitivity": tp / (tp + fn) if tp + fn else None,
        "specificity": tn / (tn + fp) if tn + fp else None,
        "false_pass_rate": (fp / (fp + tn) if fp + tn else None),
        "n": n,
    }
```

Run this on the held-out test set only, per stratum, and keep the confusion matrices. Read the disagreements by hand. Ten minutes with twenty mismatched items usually shows whether the judge prompt is wrong, the rubric is vague, or the human label is.

## Probe for bias with controlled edits

Agreement on a natural sample does not reveal systematic biases, because natural data rarely varies one factor at a time. Build small perturbation sets from items you already have and check that the verdict does not move when it should not. These are known failure modes of LLM judges, tested here as hypotheses about your judge, not assumed.

- **Position bias (pairwise).** Score every pair twice with A and B swapped. The share of pairs whose winner flips with the order is your position-bias rate. Count only consistent verdicts, or average the two orders.
- **Length bias.** Pad a correct answer with harmless filler, or trim a correct answer to its core. A pass that turns into a fail, or a score that climbs with length, shows the judge rewards verbosity or brevity.
- **Self-preference.** Score the same answers with judges from two different providers. If the order of candidate models changes with the judge's family, the judge favours its own style.
- **Format and style.** Reformat the same content (bullets, headers, a confident tone). A change in verdict means the judge is grading presentation.
- **Leniency and severity.** Compare the judge's pass rate with the humans' pass rate on the same items. A gap of several points that is stable in direction is an offset you can correct for.
- **Sensitivity to the rubric wording.** Paraphrase the rubric without changing its meaning. Large swings mean the judge prompt is brittle.

Also run the judge on a few items twice with the same input. If it flips on repeats, set temperature low where the provider lets you, and either take the majority of several calls or treat items near the boundary as uncertain.

Fix what you find in the judge prompt, using the tuning set, then re-measure on the held-out set. If a bias cannot be fixed, record it as a known offset and carry it into the gate below.

## Set the pass threshold from measured error

A threshold such as "pass rate must be at least 90%" is not meaningful until you know what the judge's pass rate means. Two corrections turn it into one.

**Correct for judge error.** If the judge has sensitivity *se* and specificity *sp* against humans, the observed pass rate *q* relates to the true pass rate *p* as q = p·se + (1−p)(1−sp). Solving gives p = (q + sp − 1) / (se + sp − 1). This holds only when the judge's error rates are similar on the two models being compared, which is what you check next.

**Gate on a bound, not a mean.** Run baseline and candidate on the same prompts and compare per item. Use a paired bootstrap on the difference in pass rate, and require the lower end of the interval to clear a margin the product owner has agreed in advance.

```python
import random

def paired_diff_lower_bound(base: list[int], cand: list[int],
                            iters: int = 5000, alpha: float = 0.05) -> float:
    n = len(base)
    diffs = []
    for _ in range(iters):
        idx = [random.randrange(n) for _ in range(n)]
        diffs.append(sum(cand[i] - base[i] for i in idx) / n)
    diffs.sort()
    return diffs[int(alpha * iters)]

def may_switch(base, cand, margin=0.02, slice_results=None) -> bool:
    if paired_diff_lower_bound(base, cand) < -margin:
        return False
    # Every critical slice must clear its own bound.
    for name, (b, c, slice_margin) in (slice_results or {}).items():
        if paired_diff_lower_bound(b, c) < -slice_margin:
            return False
    return True
```

> **Example: Gate definition, illustrative**
> These numbers are invented to show the shape of a gate, not recommendations for your product.
> 
> - Judge floor: agreement with the human majority at least 0.85 with a lower interval bound above 0.80, and false-pass rate under 0.05, on the held-out set. Below that, the judge cannot gate anything.
> - Switch rule: lower 95% bound of the paired pass-rate difference no worse than −2 points overall, and no worse than −5 points in any critical slice (for example, refunds or medical wording).
> - Hard stops, independent of the judge: schema validity, refusal rate and tool-call correctness checked by code.

Use code-graded checks for anything code can decide. Anthropic's guide ranks exact match and code grading beside LLM grading, and the judge should carry only what needs judgement.[2]

## Re-check the judge on the candidate's outputs

A judge calibrated on your current model's answers has not been calibrated on the candidate's. A new model may be more verbose, format differently, or refuse differently, and each of those shifts the judge's error rates. Before trusting a gate result, draw a fresh stratified sample of the candidate's outputs, have humans label it, and confirm the judge floor still holds there. If it does not, tune on the new sample and rerun the whole comparison.

This step is what separates a model switch from a routine regression run. It is also cheap relative to the cost of a bad switch, since the sample is a hundred or so items per decision, not the whole suite.

## Pin the judge and watch for drift

The judge is also a model, and it can change under you. Treat that as a first-class risk.

- **Pin a dated snapshot** as the judge, not an alias, so a provider's update cannot move your scores. The same reasoning applies to the models under test; see [model aliases vs pinned snapshots](https://infro.io/blog/model-aliases-vs-pinned-snapshots).
- **Keep a canary set.** Fifty to a hundred human-labelled items, never used for tuning, scored by the judge on a schedule and on every judge or prompt change. Alert when agreement on the canary falls below the floor or when the pass rate on it moves by more than your margin.
- **Version the judge as a unit:** model id, prompt text, rubric, temperature and parsing code. A change to any of them is a new judge and needs a new held-out measurement.
- **Re-label on a cadence.** Product and traffic change. Draw a small fresh human sample each quarter, or whenever the input mix shifts, and compare it with the judge.
- **Plan for retirement.** When the judge model is deprecated, you must recalibrate its replacement; build that into the retirement calendar like any other dependency.

## Where a gateway helps, and where it stops

Three parts of this method touch infrastructure. INFRO is a control plane that sits between your application and providers you already pay; the shipped pieces below run on your own provider accounts.

- **Eval monitors** score a share of served chat answers against your rubric in the background, with Jev or another decision model you connect, recorded per request and summarised per served model.[3] That gives you a live pass rate to compare with the canary set. It is a judge like any other: calibrate it against humans first.
- **Decision models** return typed answers with probabilities, and a decision carries a confidence band computed from the policy's thresholds (defaults of 0.9 and 0.5, with human review below the lower one).[3] The documentation lists the decision models you can connect, from several providers, on your own accounts.[4] Using one as a judge is a choice to test against humans, not a given.
- **Traffic changes** flag a model answered by a new snapshot, longer answers and more fallbacks, using request metadata. INFRO says plainly it does not read answers and cannot tell that quality dropped.[5] Use those findings as the cue to run your canary, not as a quality measure.

Governance teams may also want this written down as practice. NIST's AI Risk Management Framework is a voluntary framework meant to help build trustworthiness considerations into the design, development, use and evaluation of AI systems, and NIST has published a Generative AI Profile alongside it.[6] A documented judge validation, with sample design, agreement figures and a version history, is the kind of record that supports an evaluation claim.

## Checklist before a judge gates a switch

- [ ] Each rubric question is written so two people could apply it, with one dimension per call.
- [ ] Items are sampled in strata (input type, judge verdict, producing model), with hard cases tagged.
- [ ] Labels are split into a tuning set and a held-out test set, and a share is double-labelled for a human-human ceiling.
- [ ] Agreement, kappa, false-pass and false-fail rates are reported with intervals, per slice.
- [ ] Position, length, self-preference and format biases have been probed with controlled edits.
- [ ] The judge is a pinned snapshot, versioned with its prompt, rubric and parser.
- [ ] A canary set runs on a schedule and on every judge change, with an alert threshold.
- [ ] The gate uses a lower confidence bound on the paired difference, with margins agreed in advance, and code-graded checks as hard stops.
- [ ] Judge agreement has been re-confirmed on a fresh sample of the candidate's outputs before the decision.

## Questions

### How many human labels do I need to validate a judge?

Enough that the interval on agreement is narrower than the decision you want to make. With the normal approximation, a judge that agrees about 90% of the time needs roughly 140 labelled items for a ±5-point 95% interval, and roughly 35 for ±10 points. Add items per stratum if you want per-slice numbers, and treat any slice under about 30 items as a spot check.

### Can the judge be the same model as the one being graded?

Avoid it. A judge can favour text in its own style, so a candidate model from the same family may be scored higher for reasons unrelated to quality. Test for it by scoring the same answers with a judge from another provider and comparing; if the ranking of models changes, you have self-preference to account for.

### Should I use a decision model instead of a chat model as the judge?

It is a choice to test, not an assumption. A decision model returns a typed answer with a probability, which makes thresholds and confidence bands straightforward, but you still calibrate it against human labels the same way as any judge.

## Sources

1. OpenAI Evals shuts down: export your evals and rebuild them in CI — INFRO. https://infro.io/blog/openai-evals-shutdown-migration (accessed 2026-10-31)
2. Define success criteria and build evaluations — Anthropic. https://docs.anthropic.com/en/docs/test-and-evaluate/develop-tests (accessed 2026-10-31)
3. Decisions with Jev — INFRO Docs. https://infro.io/docs/features/decisions (accessed 2026-10-31)
4. Decision models — INFRO Docs. https://infro.io/docs/features/decision-models (accessed 2026-10-31)
5. Traffic changes — INFRO Docs. https://infro.io/docs/observability/traffic-changes (accessed 2026-10-31)
6. AI Risk Management Framework — NIST. https://www.nist.gov/itl/ai-risk-management-framework (accessed 2026-10-31)