Can an LLM judge be trusted? Calibrating graders against humans

A judge is an instrument, so calibrate it

OpenAI's hosted Evals platform is closing: existing evals become read-only on 31 October 2026 and the dashboard and API are scheduled to shut down on 30 November 2026.¹ Teams that used model-graded checks as a regression gate are now writing their own LLM-as-judge graders. The danger in that migration is quiet. A rebuilt grader will produce numbers, the numbers will look plausible, and nobody will know whether they track what a person would decide.

The migration advice OpenAI gives is itself a warning: recreated LLM-as-a-judge graders should be validated before you rely on them, because scores will not be identical across systems.¹ Anthropic's evaluation guide lists LLM-based grading beside exact match and code grading as one way to automate, and treats a Likert-scale judge as suited to subjective qualities such as tone.² Neither tells you how far to trust the score. This method does.

The claim you are trying to earn is narrow: for this rubric, on outputs like these, the judge agrees with trained humans at least this often, and its errors lean in a known direction. Anything wider than that is a guess. The steps are:

1. Fix the rubric and the unit of judgement.
2. Sample items and collect human labels, with a share labelled twice.
3. Measure agreement and error direction, with intervals.
4. Probe for bias with controlled edits.
5. Set a gate that uses the measured error.
6. Pin the judge and watch for drift.

Fix the rubric before you sample anything

Most disagreement between a judge and a human is really disagreement about the rubric. Before collecting labels, write the criterion so that two people could apply it without talking to each other. Anthropic's guidance on success criteria asks for criteria that are specific and measurable, and notes that even hazy qualities can be quantified when defined precisely.² Apply the same standard to each judge question.

Prefer a binary or small-ordinal verdict per criterion over a 1–10 score. Fine-grained scales produce low agreement among humans, which caps how well any judge can do. Split compound rubrics ("accurate, concise and polite") into one question per judge call, so you can see which dimension fails.

Include the hard cases on purpose. Anthropic's eval design notes name ambiguous test cases, where even humans would struggle to reach consensus, as a category to include.² Keep them, but tag them. If humans disagree with each other on an item, a judge disagreeing with one of them is not evidence of a bad judge.

Decide the unit of judgement as well. Pointwise grading (is this answer acceptable?) is simpler to calibrate than pairwise grading (which of A or B is better?), but pairwise is what a model switch usually needs. If you use pairwise, you must also test for position bias, covered below.

Sample human labels so they represent the decision

A random sample of last week's traffic over-represents easy, common cases and under-represents the failures the gate exists to catch. Sample in strata instead.

• By input type. Intent, language, length bucket, tool use, customer tier; whatever your product splits on.
• By judge verdict. Draw items the judge passed and items it failed in comparable numbers. If 95% of items pass, a random sample holds few failures, and you cannot estimate the false-pass rate.
• By producing model. Include outputs from the current model, the candidate and, if you have one, a deliberately weaker model. The judge must discriminate across that range.
• Near-threshold items. If the judge returns a score or probability, oversample the middle of its range, where mistakes concentrate.

Then split the labelled set. Keep a tuning set for editing the judge prompt and a held-out test set that nobody looks at until the judge is frozen. Reusing the same items for both inflates agreement, just as it does for any model.

Label a share of items twice, by different people, to measure human-human agreement. That number is your ceiling. A judge that agrees with a single annotator 85% of the time when two annotators agree 86% of the time is performing at the limit of the rubric.

Measure agreement and the direction of error

Raw accuracy misleads on skewed data. If 90% of answers are acceptable, a judge that passes everything is 90% "accurate". Report these together:

The last row matters more than it looks. A gate compares two models, so a judge with a constant lenient offset can still rank them correctly. A judge that is lenient only on one model's style cannot.

import math
from collections import Counter

def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
    """95% Wilson interval for a proportion."""
    if n == 0:
        return (0.0, 1.0)
    p = k / n
    d = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / d
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return (centre - half, centre + half)

def report(human: list[int], judge: list[int]) -> dict:
    """human, judge: 1 = pass, 0 = fail, same item order."""
    n = len(human)
    tp = sum(h == 1 and j == 1 for h, j in zip(human, judge))
    tn = sum(h == 0 and j == 0 for h, j in zip(human, judge))
    fp = sum(h == 0 and j == 1 for h, j in zip(human, judge))  # false pass
    fn = sum(h == 1 and j == 0 for h, j in zip(human, judge))  # false fail
    po = (tp + tn) / n
    ph, pj = Counter(human), Counter(judge)
    pe = sum((ph[c] / n) * (pj[c] / n) for c in (0, 1))
    kappa = (po - pe) / (1 - pe) if pe < 1 else float("nan")
    return {
        "agreement": (po, wilson(tp + tn, n)),
        "kappa": kappa,
        "sensitivity": tp / (tp + fn) if tp + fn else None,
        "specificity": tn / (tn + fp) if tn + fp else None,
        "false_pass_rate": (fp / (fp + tn) if fp + tn else None),
        "n": n,
    }

Run this on the held-out test set only, per stratum, and keep the confusion matrices. Read the disagreements by hand. Ten minutes with twenty mismatched items usually shows whether the judge prompt is wrong, the rubric is vague, or the human label is.

Probe for bias with controlled edits

Agreement on a natural sample does not reveal systematic biases, because natural data rarely varies one factor at a time. Build small perturbation sets from items you already have and check that the verdict does not move when it should not. These are known failure modes of LLM judges, tested here as hypotheses about your judge, not assumed.

• Position bias (pairwise). Score every pair twice with A and B swapped. The share of pairs whose winner flips with the order is your position-bias rate. Count only consistent verdicts, or average the two orders.
• Length bias. Pad a correct answer with harmless filler, or trim a correct answer to its core. A pass that turns into a fail, or a score that climbs with length, shows the judge rewards verbosity or brevity.
• Self-preference. Score the same answers with judges from two different providers. If the order of candidate models changes with the judge's family, the judge favours its own style.
• Format and style. Reformat the same content (bullets, headers, a confident tone). A change in verdict means the judge is grading presentation.
• Leniency and severity. Compare the judge's pass rate with the humans' pass rate on the same items. A gap of several points that is stable in direction is an offset you can correct for.
• Sensitivity to the rubric wording. Paraphrase the rubric without changing its meaning. Large swings mean the judge prompt is brittle.

Also run the judge on a few items twice with the same input. If it flips on repeats, set temperature low where the provider lets you, and either take the majority of several calls or treat items near the boundary as uncertain.

Fix what you find in the judge prompt, using the tuning set, then re-measure on the held-out set. If a bias cannot be fixed, record it as a known offset and carry it into the gate below.

Set the pass threshold from measured error

A threshold such as "pass rate must be at least 90%" is not meaningful until you know what the judge's pass rate means. Two corrections turn it into one.

Correct for judge error. If the judge has sensitivity se and specificity sp against humans, the observed pass rate q relates to the true pass rate p as q = p·se + (1−p)(1−sp). Solving gives p = (q + sp − 1) / (se + sp − 1). This holds only when the judge's error rates are similar on the two models being compared, which is what you check next.

Gate on a bound, not a mean. Run baseline and candidate on the same prompts and compare per item. Use a paired bootstrap on the difference in pass rate, and require the lower end of the interval to clear a margin the product owner has agreed in advance.

import random

def paired_diff_lower_bound(base: list[int], cand: list[int],
                            iters: int = 5000, alpha: float = 0.05) -> float:
    n = len(base)
    diffs = []
    for _ in range(iters):
        idx = [random.randrange(n) for _ in range(n)]
        diffs.append(sum(cand[i] - base[i] for i in idx) / n)
    diffs.sort()
    return diffs[int(alpha * iters)]

def may_switch(base, cand, margin=0.02, slice_results=None) -> bool:
    if paired_diff_lower_bound(base, cand) < -margin:
        return False
    # Every critical slice must clear its own bound.
    for name, (b, c, slice_margin) in (slice_results or {}).items():
        if paired_diff_lower_bound(b, c) < -slice_margin:
            return False
    return True

Use code-graded checks for anything code can decide. Anthropic's guide ranks exact match and code grading beside LLM grading, and the judge should carry only what needs judgement.²

Re-check the judge on the candidate's outputs

A judge calibrated on your current model's answers has not been calibrated on the candidate's. A new model may be more verbose, format differently, or refuse differently, and each of those shifts the judge's error rates. Before trusting a gate result, draw a fresh stratified sample of the candidate's outputs, have humans label it, and confirm the judge floor still holds there. If it does not, tune on the new sample and rerun the whole comparison.

This step is what separates a model switch from a routine regression run. It is also cheap relative to the cost of a bad switch, since the sample is a hundred or so items per decision, not the whole suite.

Pin the judge and watch for drift

The judge is also a model, and it can change under you. Treat that as a first-class risk.

• Pin a dated snapshot as the judge, not an alias, so a provider's update cannot move your scores. The same reasoning applies to the models under test; see model aliases vs pinned snapshots.
• Keep a canary set. Fifty to a hundred human-labelled items, never used for tuning, scored by the judge on a schedule and on every judge or prompt change. Alert when agreement on the canary falls below the floor or when the pass rate on it moves by more than your margin.
• Version the judge as a unit: model id, prompt text, rubric, temperature and parsing code. A change to any of them is a new judge and needs a new held-out measurement.
• Re-label on a cadence. Product and traffic change. Draw a small fresh human sample each quarter, or whenever the input mix shifts, and compare it with the judge.
• Plan for retirement. When the judge model is deprecated, you must recalibrate its replacement; build that into the retirement calendar like any other dependency.

Where a gateway helps, and where it stops

Three parts of this method touch infrastructure. INFRO is a control plane that sits between your application and providers you already pay; the shipped pieces below run on your own provider accounts.

• Eval monitors score a share of served chat answers against your rubric in the background, with Jev or another decision model you connect, recorded per request and summarised per served model.³ That gives you a live pass rate to compare with the canary set. It is a judge like any other: calibrate it against humans first.
• Decision models return typed answers with probabilities, and a decision carries a confidence band computed from the policy's thresholds (defaults of 0.9 and 0.5, with human review below the lower one).³ The documentation lists the decision models you can connect, from several providers, on your own accounts.⁴ Using one as a judge is a choice to test against humans, not a given.
• Traffic changes flag a model answered by a new snapshot, longer answers and more fallbacks, using request metadata. INFRO says plainly it does not read answers and cannot tell that quality dropped.⁵ Use those findings as the cue to run your canary, not as a quality measure.

Governance teams may also want this written down as practice. NIST's AI Risk Management Framework is a voluntary framework meant to help build trustworthiness considerations into the design, development, use and evaluation of AI systems, and NIST has published a Generative AI Profile alongside it.⁶ A documented judge validation, with sample design, agreement figures and a version history, is the kind of record that supports an evaluation claim.

Checklist before a judge gates a switch

• Each rubric question is written so two people could apply it, with one dimension per call.
• Items are sampled in strata (input type, judge verdict, producing model), with hard cases tagged.
• Labels are split into a tuning set and a held-out test set, and a share is double-labelled for a human-human ceiling.
• Agreement, kappa, false-pass and false-fail rates are reported with intervals, per slice.
• Position, length, self-preference and format biases have been probed with controlled edits.
• The judge is a pinned snapshot, versioned with its prompt, rubric and parser.
• A canary set runs on a schedule and on every judge change, with an alert threshold.
• The gate uses a lower confidence bound on the paired difference, with margins agreed in advance, and code-graded checks as hard stops.
• Judge agreement has been re-confirmed on a fresh sample of the candidate's outputs before the decision.