Measurement and Integrity
Potato reports chance-corrected agreement over geometry, time, 3D cuboids, captions and world-model break-points, and records how the annotations were made. Overview of the measurement surface.
Every annotation type in Potato ships with a way to measure whether annotators agree, chance-corrected, with its failure modes stated. That includes the spatial and temporal types, where the industry standard is raw IoU or percent match, and neither is chance-corrected.
This section covers both halves of the question: whether the labels agree, and how they were produced.
The through-line
| Question | Measure |
|---|---|
| Did annotators find the same objects? | detection α |
| Did they call them the same thing? | classification α |
| Did they put them in the same place? | σ and KS against an empirical chance baseline |
| Whose mask do we trust? | STAPLE |
| Do they agree when an event starts? | temporal IoU and boundary α |
| Do they agree on a 3D box? | exact rotated 3D IoU |
| Do they describe a region the same way? | α with a semantic distance over text |
| Do they agree when the world broke? | break-point agreement with a tolerance sweep |
| Is an annotator competent? | MACE, for the genuinely categorical parts |
Why chance correction
Raw agreement is inflated whenever one answer dominates, and one answer nearly always dominates.
The clearest case is spatial. A corpus where every image holds one large centred object will show a mean IoU around 0.95 no matter who annotates it — including annotators who never looked at the image and drew a box in the middle. Chance correction is what separates "these annotators agree" from "this task is too easy to disagree on".
Most annotation platforms report agreement as raw IoU, exact match, or percent agreement against a ground-truth job. Those are useful numbers that answer a different question. We have found no other annotation platform that reports a chance-corrected coefficient for spatial labels at all.
Two design properties
Undefined values explain themselves. α is genuinely undefined on a unanimous corpus. A bare NaN reads as a broken computation and a 1.0 would be a lie, so the report says which it is, in words.
Nothing is truncated silently. A report that hits its pairwise budget says so and states how many items were included. A number computed over a sample that presents itself as a number computed over everything is worse than no number.
The bug this layer exists because of
Adjudication used to compare the keys of an annotation, and an image schema stores everything under one key named _data. Every image pair therefore scored 1.0 agreement: two annotators who agreed on nothing looked unanimous, and no image was ever routed for review.
That is fixed, and it is the reason the agreement layer is built into each annotation type rather than bolted on top of them.
How the data was made
Agreement cannot catch two annotators confidently agreeing on the wrong thing, and it cannot catch a pre-label accepted without being read. Two surfaces address that:
- Drawing telemetry records time per shape, stroke dynamics, revision counts, and AI-suggestion accept latency.
- Keystroke logging is the text-side twin, for how a free-text answer was produced.
Both are opt-in, and annotators see a recording notice.
Trust and deployment
- Air-gapped deployment — every asset serves from your install, verified at zero external requests.
- Machine-readable specs — a JSON Schema and an OpenAPI document, both generated from the code and checked in CI.