Using a large language model to grade other model outputs has become the default move in evaluation. You write a rubric, ask GPT-4o or Claude to score a thousand responses, and read off an accuracy number. It is fast, it is cheap, and it scales past anything a human team can label by hand.

It also quietly assumes the thing you most need to check: that the judge agrees with people. An LLM-as-a-judge that is confidently wrong produces a clean-looking leaderboard built on sand. Before you trust a judge's verdicts, you have to measure how well they track human judgment. That measurement step is calibration, and Potato 2.6 adds a workflow for it.

This post covers Judge Calibration: how it samples models, how it keeps the human pass honest, and what the report actually tells you. The reference docs have the complete option list.

An LLM judge verdict shown beside a human annotation, with a running kappa Inline judge calibration in Potato

The shape of the problem

A judge can fail in two different ways, and you want to catch both.

The first is disagreement: the judge calls things "correct" that a careful human would call "wrong." That is what accuracy and agreement metrics measure.

The second is bad confidence: the judge says it is 95% sure and is right 60% of the time. A judge can have decent accuracy and still be badly miscalibrated, which matters the moment you start using its confidence to route work or set thresholds. That is what calibration error measures.

Potato's calibration pass is built to surface both at once.

A reliability diagram: the perfect-calibration diagonal versus an overconfident judge whose accuracy sits below its stated confidence Accurate is not the same as calibrated

How it works

The workflow runs as a short state machine, from configuring the judge to a finished report:

The calibration run moves through five states: setup, generating, human calibration, report, and completed The five states of a calibration run

Generating. Each model is queried k times per item. The modal label across those k samples is the model's prediction, and the fraction of samples that agree with it is the model's confidence. Sampling k times instead of once is what gives you an empirical confidence signal rather than a number the model made up about itself. These results go to a dedicated store and are never written into the annotation data.

Human calibration. Potato draws a random or stratified sample of the items and routes them to one or more humans, who label them through the normal annotation interface, without ever seeing the model's answers.

Report. Metrics are computed over the overlap between what the models labeled and what the humans labeled, then written to disk.

The blindness here is the important part. Because the model labels live in a separate store and are never injected into the UI, the human cannot be anchored by them even accidentally. Blindness is structural, not a matter of asking annotators to look away.

The calibration pipeline: k-sample model voting, a blind human pass, and a comparison report How Potato calibrates a judge against blind human labels

Configuration

A judge calibration is one config block. You write the judge prompt, list the models, and set how many times to sample each:

yaml

judge_calibration:
  enabled: true
  prompt: |                       # supports {text}, {labels}, {description}
    You are an impartial expert annotator. Classify the sentiment as exactly
    one of: positive, negative, neutral.
  models:
    - endpoint_type: openai        # openai | anthropic | ollama | vllm | gemini | ...
      model: gpt-4o-mini
      api_key: ${OPENAI_API_KEY}
      temperature: 0.7             # must be > 0 so the k samples vary
    - endpoint_type: ollama
      model: llama3.1:8b
      base_url: http://localhost:11434
      temperature: 0.7
  k_samples: 5                     # samples per model per item
  max_items: 1000                  # cap on items the LLMs label (null = all)
  sampling:
    strategy: stratified           # random | stratified | all
    sample_size: 200               # how many items humans blind-label
    seed: 42
  human:
    num_raters: 1                  # 1 = solo researcher; N adds human-human IAA
    gold: single                   # single | majority
  schemas: [sentiment]
  output:
    dir: judge_calibration_output

Warning: Set temperature > 0. With k_samples > 1 and temperature 0, the samples are identical, confidence is pinned at 1.0, and the calibration report is meaningless. Potato emits a startup warning when it sees that combination.

You can list more than one model and calibrate them side by side, which is the natural way to choose between a cheap local judge and an expensive hosted one.

Trying it without an API key

The bundled example uses a local Ollama model, so you can run the whole loop offline. Start Ollama, pull the model, and launch:

bash

ollama pull llama3.2:3b
python potato/flask_server.py start examples/ai-assisted/judge-calibration/config.yaml -p 8000 --debug

Open http://localhost:8000/judge_calibration/admin to configure and run, blind-label the sample at /annotate, then build the report and read it at /judge_calibration/report.

What the report tells you

The report is built to answer "should I trust this judge?" with numbers you can put in a methods section:

Accuracy, precision, recall, F1 for each model against the human gold label.
Cohen's κ broken out into human↔model, model↔model, and human↔human pairs, so you can see whether the judge agrees with people as well as people agree with each other.
Fleiss' κ and Krippendorff's α across all raters.
Expected Calibration Error (ECE), reliability bins, and a Brier score: the answer to the bad-confidence failure mode.
A confusion matrix per model, which usually tells the real story: a judge that is fine on the easy classes and falls apart on one hard distinction.

Everything is computed over the overlap: items both the models and the humans labeled, restricted to the calibration sample. Output lands under output.dir as llm_labels.jsonl, report.json, and a readable report.html.

What it handles

Calibration is fully supported on the categorical schemes most judges use, and reaches into harder types:

Type	Status	Metrics
`radio` / `select`	Supported	accuracy, P/R/F1, Cohen/Fleiss κ, Krippendorff α, ECE, confusion
`likert`	Supported	the above plus MAE and ordinal Krippendorff α
`multiselect`	Supported	per-label P/R/F1, mean Jaccard, exact-match accuracy, calibration
`span`	Experimental	IoU-matched P/R/F1, mean IoU, span-F1, span-level calibration

Span calibration clusters the judge's character-offset spans across the k samples and matches them to gold by intersection-over-union; treat its numbers as directional rather than exact.

Calibration versus alignment

Potato ships a second, related workflow that is easy to confuse with this one. Judge Alignment calibrates a single judge against an existing human gold set, shows its verdict inline during annotation, and is built around iterating on a rubric until agreement climbs.

The rule of thumb: reach for calibration when you are vetting candidate judges and want a blind, empirical confidence number; reach for alignment when you have settled on one judge and are tuning its rubric against a fixed gold set. The two are covered together in Closing the Loop.

LLM judges are not going away; there is too much to evaluate and too few people to do it by hand. The point of calibration is not to replace the judge with humans, but to know, with a number, exactly how far the judge can be trusted before a human has to look.

The Judge Calibration docs cover every option, and the inter-annotator agreement guide explains the kappa and alpha metrics in depth.