Raw IoU Is Not Agreement
Mean IoU between annotators reads about 0.95 on a typical detection corpus, including for annotators who never looked at the image. What a chance-corrected alternative looks like, and why Krippendorff's alpha over IoU distance does not work.
Every computer-vision annotation platform that reports inter-annotator agreement reports it as raw IoU against a per-class threshold. CVAT's consensus engine does. V7's consensus stage does. Label Studio Enterprise's metric list is exact match, numeric difference, IoU and span overlap.
None of them applies a chance correction, and without one the number does not mean what it appears to mean.
The problem, concretely
Take a detection corpus where each image holds one large, roughly centred object. Two annotators both draw a box around it. Their boxes overlap heavily, mean IoU comes out around 0.95, and the dashboard says agreement is excellent.
Now replace one annotator with a process that draws a box in the middle of every image without looking. Mean IoU stays around 0.9.
The measure is reporting a property of the corpus — how constrained the task is — rather than a property of the annotators. This is the same failure that percent agreement has on categorical labels when one class dominates, and it has been understood there since Cohen introduced kappa in 1960.
Spatial annotation never got the equivalent fix.
The obvious repair does not work
Krippendorff's α accepts any distance function. So the obvious move is α over 1 − IoU, and that was our original plan.
It fails for a structural reason. α computes 1 − D_o/D_e, where D_e is expected disagreement under random pairing. IoU distance is bounded in [0, 1], and two randomly paired boxes almost always have IoU 0. So D_e collapses to approximately 1, and the whole coefficient degenerates into 1 − mean observed distance with no working chance correction inside it.
There is a second problem. IoU is flat exactly where annotators disagree: two boxes that do not overlap score 0 whether they are touching or at opposite corners of the image. The measure has no gradient in the region you care about.
This is not only a theoretical objection. Braylan, Alonso and Lease (WWW 2022) evaluated candidate distance functions by whether the resulting agreement score ranks annotator quality correctly — arguably the only property that matters — and found that for bounding boxes, α ranks plain L2 (0.687) above IoU (0.505) and GIoU (0.507). That inverts the ordering their own distribution-based measures give, and the ordering practitioners give.
What we report instead
σ, which keeps α's form but estimates the chance baseline empirically:
σ = 1 − mean(within-item distance) / mean(between-item distance)
The denominator is the key change. Instead of assuming a distribution of random pairings, compare annotations of different items. That gives an empirical answer to "how far apart would these annotators be if they were not looking at the same thing", which is what a chance baseline is supposed to be.
| σ | Reading |
|---|---|
| 1.0 | Perfect agreement |
| 0.0 | No better than annotating unrelated images |
| below 0 | Systematically further apart on the same image than on different ones |
We do not clamp negative values. "Further apart on the same image than on different images" is a real and diagnosable state, and it almost always means the guidelines are ambiguous rather than that the annotators are careless. Hiding it helps nobody.
Alongside σ we report a two-sample Kolmogorov–Smirnov statistic between the within-item and between-item distance distributions. σ compares two means and can be dragged by outliers; KS compares whole distributions. When the two disagree, a minority of items is carrying most of the disagreement, which tells you which items to open.
One number was never enough anyway
Even correctly chance-corrected, a single spatial agreement figure cannot say which of three things went wrong:
- Detection. Did they find the same objects at all?
- Classification. Did they give matched objects the same label?
- Localization. Are matched objects in the same place?
An annotator who finds every object and mislabels them all is a completely different problem from one who labels correctly but boxes loosely. The fixes are different. So the three are reported separately.
Masks and 3D need their own tools
For pixel masks, "do they agree" and "whose boundary should the dataset record" are different questions. The second is answered by STAPLE, which estimates a latent consensus boundary along with a sensitivity and specificity per annotator. Those two are reported separately, because under-segmenting and over-segmenting have opposite fixes.
STAPLE also weights by demonstrated reliability rather than counting votes, which matters when careful annotators are outnumbered. Two careful annotators against three noisy ones: majority vote scores Dice 0.846 against truth, STAPLE scores 1.000.
For 3D cuboids, use exact rotated 3D IoU. An axis-aligned approximation is adequate for level automotive data and wrong for drone, handheld and indoor scans, where it manufactures disagreement that is an artefact of the measure.
Why this took until now
The honest answer is that the bug came first. Potato's adjudication compared the keys of an annotation, and an image schema stores everything under one key called _data. Every image pair scored 1.0 agreement. Two annotators who agreed on nothing looked unanimous, and no image was ever routed for review.
Fixing that meant asking what the right number would have been, and the literature does not have a settled answer for geometry. This is ours, with its limits stated: temporal segments currently get the uncorrected measures only, because "a segment from a different clip" is not a meaningful chance comparison when clips differ in length.