# How to Measure Agreement Between an LLM Judge and Human Annotators

Source: https://www.potatoannotator.com/docs/guides/llm-judge-human-agreement

**To decide whether to trust an LLM judge, have at least two people label a sample of items without seeing the judge's answers, then compare the judge's agreement with those people to the people's agreement with each other. A judge whose Cohen's kappa with the humans is close to the humans' kappa with each other is doing about as well as another annotator. One well below it is not.**

An [LLM-as-a-judge](https://huggingface.co/learn/cookbook/en/llm_judge) is a language model prompted to grade outputs, usually against a rubric. [Cohen's kappa](https://en.wikipedia.org/wiki/Cohen%27s_kappa) measures agreement between two raters, corrected for the agreement expected by chance. A judge is [calibrated](https://en.wikipedia.org/wiki/Calibration_(statistics)) when its confidence matches how often it is right: of the verdicts it gives with 80% confidence, about 80% should be correct.

## Why accuracy against one person is not enough

The usual first check is accuracy. You label a few hundred items yourself and count how often the judge matches. That number has two problems. It treats one person's labels as the truth, mistakes included, and it is not corrected for chance, so a judge that calls every response "helpful" scores well on a dataset where most responses are helpful.

Kappa fixes the second problem. A second human annotator fixes the first, because the disagreement between two careful people shows how much the task itself allows, and a judge cannot be held to more than that.

## Compare the judge with the humans' agreement with each other

Report three kinds of pair: human with human, judge with human, and judge with judge when you are comparing several judges. If two people agree at κ = 0.62, a judge at 0.60 with each of them is behaving like a third annotator, and asking it for 0.9 asks more than the task allows. A judge at 0.35 is not agreeing with people, however good its raw accuracy looks.

[Zheng et al. (2023)](https://arxiv.org/abs/2306.05685) made the same comparison on MT-Bench and found GPT-4's agreement with human preferences comparable to the agreement between humans.

With more than two humans, or with items missing some labels, report Fleiss' kappa or Krippendorff's alpha across all raters as well as the pairwise kappas. [Inter-Annotator Agreement Explained](/docs/guides/inter-annotator-agreement) covers the choice.

## Keep the humans blind

If annotators can see the judge's answer, they anchor on it, and part of the agreement you measure is the judge agreeing with itself. Collect the human labels without the judge's answers on screen. The same effect raises agreement when annotators review model pre-labels. See [Detecting Rubber-Stamped Pre-Labels](/docs/guides/detecting-rubber-stamped-prelabels).

## Check confidence as well as verdicts

A judge can agree with people and still be overconfident. If its confidence decides which items go to a human, overconfidence lets the wrong items skip review.

A confidence the model states about itself is generated text like the rest of its answer. Sampling gives an empirical one: query the judge k times per item at a temperature above zero, take the most common answer as its verdict, and use the share of samples that gave that answer as its confidence. At temperature 0 the k samples are identical and every confidence is 1.0.

Expected calibration error (ECE) groups verdicts by confidence and compares each group's confidence with its accuracy. The [Brier score](https://en.wikipedia.org/wiki/Brier_score) is the mean squared difference between confidence and outcome. A reliability diagram plots the groups against the diagonal a perfectly calibrated judge would follow.

## Test pairwise judges for position bias

A judge comparing two responses may favour whichever one comes first, or second, regardless of content. Zheng et al. documented this position bias in LLM judges. Run each pair twice with the order swapped. A verdict that flips when the order flips was decided by position, and the share of pairs that stay consistent belongs in the report next to agreement. See [Pairwise Model Comparison](/docs/guides/pairwise-model-comparison).

## Ordered scales

On a 1–5 rating, a judge that says 4 where a human says 5 has nearly agreed. Plain kappa counts that the same as 1 against 5. Use weighted kappa or Krippendorff's alpha with an ordinal or interval metric, and report the mean absolute error alongside. See [Rating Scales](/docs/guides/rating-scales).

## How many items to label

A kappa computed on 50 items has a wide confidence interval, wide enough that two judges can look different when the difference is noise. Report the interval with the estimate, and size the human sample before you start; [statistical power for annotation studies](/docs/guides/statistical-power-annotation) shows how. Stratifying the sample by the judge's label keeps rare classes in it.

## Doing it in Potato

Potato's judge calibration runs each judge k times per item and keeps its labels in a separate store, so annotators never see them. It then draws a random or stratified sample for blind human labelling and writes a report.

```yaml
annotation_schemes:
  - annotation_type: radio
    name: helpfulness
    description: "Is this response helpful?"
    labels: [helpful, unhelpful]

judge_calibration:
  enabled: true
  prompt: |
    You are an impartial expert annotator. Classify the response as exactly
    one of: helpful, unhelpful.
  models:
    - endpoint_type: ollama
      model: llama3.1:8b
      base_url: http://localhost:11434
      temperature: 0.7
  k_samples: 5
  sampling:
    strategy: stratified
    sample_size: 200
    seed: 42
  human:
    num_raters: 2
    gold: majority
  schemas: [helpfulness]
```

The report gives accuracy, precision, recall and F1 for each model against the human gold label, and Cohen's κ split into human–model, model–model and human–human pairs. It adds Fleiss' κ and Krippendorff's α across all raters, ECE with reliability bins and the Brier score, and a confusion matrix per model. For Likert schemes it adds mean absolute error and ordinal α. Since version 2.9 the judge eval card also shows position-swap consistency, and the generated labels can be read at `/judge_calibration/results` before the human phase finishes.

[LLM-as-Judge Calibration](/docs/agent-evaluation/judge-calibration) lists every option. [Judge Alignment](/docs/agent-evaluation/judge-alignment) covers tuning one judge's rubric against an existing gold set.

## Other tools

LangSmith, Langfuse and Galileo each let you align a judge with human corrections. Langfuse computes Cohen's κ between a human score and a judge score, two series at a time. Label Studio's paid plans show where annotators and LLMs agree. See [AI Agent Evaluation Tools Compared](/docs/guides/agent-evaluation-tools-compared).

## Frequently asked questions

### What is a good kappa between an LLM judge and humans?

There is no fixed threshold. Compare the judge-human kappa with the kappa between two humans on the same items. A judge close to the human-human figure agrees with people about as well as another annotator would. The usual rough guide for κ and α, 0.8 and above to rely on and 0.67 to 0.8 for tentative conclusions, describes the task as a whole, and a hard task caps what any judge can reach.

### Should annotators see the LLM judge's answer?

Not while you are measuring agreement. Annotators who can see the judge's verdict tend to anchor on it, which inflates the judge's apparent agreement. Show the verdict only after the calibration sample is labelled, if at all.

### How do I check an LLM judge for position bias?

Present every pair twice, once in each order, and count how often the verdict changes with the order. Report the share of consistent pairs next to agreement, and discard or re-run the pairs where the verdict flips.

### What is expected calibration error for an LLM judge?

ECE measures the gap between a judge's confidence and its accuracy. Verdicts are grouped by confidence, and ECE is the average difference between each group's confidence and the share of its verdicts that matched the human label, weighted by group size. A well-calibrated judge has an ECE near zero.

## Further reading

- [LLM-as-Judge Calibration](/docs/agent-evaluation/judge-calibration)
- [Can You Trust Your LLM Judge?](/blog/trust-your-llm-judge-calibration)
- [Rubric-Based LLM Evaluation](/docs/guides/rubric-based-llm-evaluation)
- [Inter-Annotator Agreement Explained](/docs/guides/inter-annotator-agreement)
- [AI Agent Evaluation Tools Compared](/docs/guides/agent-evaluation-tools-compared)