Instrumenting the Calibration Meeting

Two things every annotation team does badly, because their tools do not help: the calibration meeting, which happens over screen-share and leaves no trace, and understanding why an annotator chose a label, which usually reduces to a free-text box nobody fills in. Potato 2.7 instruments both: a live norming room with a real-time agreement meter, and a think-aloud recorder that runs speech-to-text locally. Neither needs an LLM.

The meeting that leaves no evidence

If you have run an annotation project with more than two people, you have run a norming session. Everyone labels the same handful of items, you get on a call, you argue about the disagreements, and eventually you converge on what the guideline actually means.

It works. It is also completely uninstrumented. The votes people held before the discussion are gone. Who changed their mind, and whether they changed it because they were persuaded or because a senior colleague spoke first, is gone. And whether the meeting improved agreement at all is a question nobody can answer, because nobody measured it.

Multiplayer Rooms move that session into the tool.

A norming room mid-session

rooms:
  enabled: true
  who_can_create: any
  persist_votes: true
  schema: sarcasm

Open /rooms, create one, and it gets a six-letter code you can read aloud on the call. The flow is deliberate:

1. Everyone votes blind. Members can see who has voted, never what. This is enforced on the server, so first impressions are genuinely independent.
2. The host reveals. At this moment the blind votes become immutable.
3. The group discusses, in a side chat or on your call.
4. Anyone may change their vote. Every post-reveal change is logged with what they switched from, what they switched to, and what the majority was at that moment.

That fourth step is the interesting one. A vote change after the reveal is not the same event as a vote; it is a change in the presence of a known majority. Logging it gives you a per-member conformity record, which is how you notice that one annotator flips to the majority every single time. That is worth knowing before you treat their agreement as independent evidence.

The agreement meter answers "was that worth it?"

Krippendorff's α is computed twice, live, over the revealed items: once on the blind votes, once on the current votes. The gap between them is the session's norming lift, and it is on screen while the session is happening.

This is the number that makes the meeting accountable. If an hour of discussion moved α from 0.61 to 0.78, you know the hour bought something. If it moved it from 0.61 to 0.62, you have learned that the disagreement is not about interpretation. It is about the guideline, and no amount of talking will fix it. Go rewrite the guideline instead.

Two variants are worth knowing about. Huddles seed a room from exactly the items your team currently disagrees on, computed live from annotation state, which makes them a synchronous version of adjudication. Shadow rooms let trainees watch a senior annotator work in real time, including which span of text they are highlighting. It is the closest thing to sitting next to someone while they annotate.

Rooms are event-sourced to a JSONL log, so a live room survives a server restart, and the log doubles as the session's audit trail. There are no WebSockets involved; clients poll with a cursor, which means rooms work under any WSGI deployment.

The reasoning you never captured

The other quiet failure is rationales. Most tools offer a free-text box, and most annotators type "obvious" into it, or nothing.

This is not laziness. Writing out your reasoning is genuinely slower than having it, and if the task is 400 items long, the rationale box is the first thing to go.

Think-Aloud Mode removes the friction by changing the modality: annotators just talk while they work.

Think-Aloud, with a detected label

thinkaloud:
  enabled: true
  schema: politeness
  model: tiny.en

Speech-to-text runs locally through faster-whisper, which is CPU real-time on the 39 MB tiny.en model. No cloud API, no per-token cost, and nothing leaves the machine, which is the difference between "we can use this" and "legal said no" for a lot of sensitive data.

The think-aloud protocol was the gold standard for studying how people make decisions long before any of this was computational. It never made it into annotation tooling, because asking someone to narrate into a recorder and then transcribe it by hand is not a workflow anyone sustains.

Verbatim, on purpose

The transcript is stored exactly as spoken and deliberately not summarized. This is a real design decision, not laziness: paraphrasing a think-aloud protocol contaminates the artifact. The hesitations, the self-corrections, the "well, it could be sarcastic, but…": that is the data. A tidy summary of it is a different and much less useful object.

Annotators can also commit a label by voice, using set phrasings that a rule-based parser detects:

• "I label this Polite"
• "My answer is impolite"
• "Final answer: neutral"

Only those phrasings commit. Everything said while thinking is ignored, which is what makes a rule-based parser sufficient. There is no LLM anywhere in the pipeline, and no model deciding what you meant. Say a new phrase later and the last commitment wins.

You also get deterministic hesitation signals (silent-chunk counts, filler-word counts) computed with arithmetic rather than a model. Long pauses on an item are a decent proxy for difficulty, and they cost nothing to collect.

Why this matters beyond rationales

Here is the part that connects to the rest of Potato 2.7.

Think-Aloud is a human chain-of-thought recorder. It captures how a person actually reasons their way to a label: not a cleaned-up justification written afterwards, but the reasoning as it happens, false starts included.

Potato's process-reward annotation does the same job for a model: it segments a model's chain of thought and scores it step by step.

Same item, two chains of thought, one human and one model. You can put them side by side and look at where the reasoning diverges. That is a genuinely new thing to have, and it is the raw material for better process-reward data, because "the model got the right answer for the wrong reason" is exactly the failure that final-answer scoring cannot see.

Potato gives you both capture surfaces. It does not compute a diff between them for you. That is a research question rather than a feature, and we would rather hand you the artifact than a metric we made up.

Further reading

• Multiplayer Rooms and Think-Aloud Mode — full configuration
• Inter-annotator agreement — what α is actually measuring
• Adjudication and disagreement
• Labels with error bars — psychometrics flags the codebook bugs; rooms are where you fix them
• Process reward models — the model-side chain of thought