Judging the Judge: Human and Model Reasoning, Side by Side

Everyone evaluating LLMs at scale eventually uses an LLM as the judge, and everyone eventually asks whether the judge can be trusted. The usual answer is to check it against human labels. But that check is only as good as the human labels, and "the judge agrees with humans" is a weak claim if the humans disagreed with each other and nobody wrote down by how much. This is about closing that gap: measuring how a model reasons, measuring how a person reasons, and comparing the two honestly.

Potato 2.7 is organized around one idea: an annotation tool should measure how a judgment was made, not just record it. The interesting part is that it applies to both kinds of judge.

The problem with validating a judge

The standard workflow is reasonable. You have a task too big to label by hand, so you prompt a model to score it. Before trusting the scores, you take a sample, have humans label it, and measure agreement between the judge and the humans. If Cohen's κ is high enough, you ship.

The soft spot is the last step. That κ compares your judge against a human label, usually a majority vote over two or three annotators. And a majority vote is a point estimate with no error bar. It says "sarcastic" with exactly the same confidence whether all three annotators agreed instantly or two overruled a third after a long argument.

So when you report that your judge agrees with humans at κ = 0.71, you are reporting agreement with a number whose own uncertainty you never measured. If the human labels on the hard items are close to coin flips, then κ on those items is mostly measuring noise, and no amount of prompt engineering will move it.

We wrote about the mechanics of judge calibration and judge–human alignment before. What 2.7 adds is the other half: making the human side of that comparison carry its own uncertainty.

Give the human labels error bars first

The psychometrics engine fits an item response theory model to your human annotations and reports every label as a posterior with an interval (sarcastic, p = 0.94 [0.88 – 0.97]), weighing who voted rather than only how many.

Truth Serum adds peer-prediction scoring, which flags the items where the crowd is probably wrong even though it was confident.

Now the judge comparison changes shape. Instead of

the judge agrees with the majority label on 71% of items

you can say

the judge agrees with the human posterior on 94% of items where the human label is confident (p > 0.9), and 52% of items where it is not. Those low-confidence items are 18% of the set.

That second statement is actionable in a way the first is not. It tells you your judge is fine on the easy cases and unreliable on the hard ones, and it tells you how much of your data is hard. It also tells you something uncomfortable and useful: on the genuinely ambiguous items, there may be no ground truth for the judge to agree with, and chasing a higher κ there is chasing noise.

This is what the judge–human alignment dashboard is for, and why the human-process features and the agent-evaluation features are the same release rather than two.

Two chains of thought on the same item

The other half of the comparison is the reasoning itself.

Think-Aloud Mode records how a person reasons to a label. The annotator talks while they work; speech-to-text runs locally; the transcript is stored verbatim, un-summarized, because paraphrasing a think-aloud protocol destroys the thing you were collecting.

Human chain of thought, captured verbatim

Process-reward annotation records how a model reasons. The model's chain of thought is segmented into steps, and each step gets a good/bad/neutral reward signal, which is the training signal for a process reward model.

annotation_schemes:
  - annotation_type: process_reward
    name: cot_review
    description: "Rate each step of the model's reasoning."
    steps_key: reasoning_steps

Same item. Two chains of thought. Put them next to each other.

What you actually learn from the comparison

The obvious use is finding where the model reasons differently from the person, and it is worth being concrete about why that matters.

Final-answer scoring cannot see the difference between a model that got the right answer for the right reason and one that got the right answer for the wrong reason. On a sarcasm task, a human might say "the 'thanks so much' is doing the work here. Nobody thanks you that hard for a two-day delay." A model might land on the same label because the text contains an exclamation mark. Both are marked correct. Only one of them will generalize.

You cannot detect that from labels. You can detect it from the reasoning, which is why having both is worth the trouble.

The second use is training data. Steps where the human's reasoning and the model's reasoning diverge are, almost by definition, the highest-signal steps in a process-reward dataset. They are the places the model's process is wrong in a way its output does not reveal.

One honest limitation. Potato gives you both capture surfaces and puts them side by side. It does not compute an automated similarity metric between a human's chain of thought and a model's, and we have deliberately not shipped one. What "these two reasoning traces agree" means is an open research question, and a number we invented would be worse than no number, because it would look authoritative and mean very little. The artifact is the thing; the metric is yours to define.

Putting it together

A defensible evaluation pipeline in 2.7 looks roughly like this.

1. Label a sample with humans, with psychometrics on. Every label gets a posterior and an interval. Ambiguous items are identified as ambiguous rather than silently mislabeled.
2. Run Truth Serum to catch the items where a confident crowd is probably wrong.
3. Fix the guideline where the codebook-bug detector flags negative discrimination, ideally in a norming room where you can watch agreement move as you converge.
4. Calibrate the judge against those labels, and report alignment conditioned on human confidence rather than as a single number.
5. Check the reasoning, not only the verdict, on the items where the judge and the humans disagree.

Every step is YAML, self-hosted, and free. Steps 1 through 3 need no LLM at all.

None of this makes an LLM judge trustworthy by itself. What it does is make the claim of trustworthiness a real one: a statement with an interval on it, against human labels that also have intervals, on items you can say something honest about the difficulty of.

That is a lower bar than "our judge is aligned with human values" and a much more useful one.

Further reading

• Judge–human alignment and judge calibration
• Labels with error bars — the IRT layer under all of this
• Quality control without gold labels
• Process reward models
• Think-Aloud Mode
• Potato 2.7 — the full release