Agreement over Geometry

Potato decomposes spatial agreement into three questions — did annotators find the same objects, call them the same thing, and put them in the same place — and answers the third against an empirically estimated chance baseline rather than with raw IoU.

The problem with raw IoU

CVAT's consensus engine and V7's consensus stage both compare annotators with raw IoU against a per-class threshold. Neither applies any chance correction, which makes the headline number uninterpretable on its own:

A corpus where every image holds one large, centred object will show a mean IoU around 0.95 no matter who annotates it. That includes annotators who never looked at the image and simply drew a box in the middle.

Chance correction is what separates "these annotators agree" from "this task is too easy to disagree on".

Krippendorff's α over 1 − IoU does not work here

That was the original plan here, and it is empirically the wrong default.

Braylan, Alonso and Lease (WWW 2022) evaluate candidate distance functions by whether the resulting agreement score ranks annotator quality correctly — the property that actually matters — and report that for bounding boxes, α ranks plain L2 (0.687) above IoU (0.505) and GIoU (0.507), inverting the ordering their own distribution-based measures and practitioners both give.

There is a structural reason. IoU distance is bounded in [0, 1] and saturates: two randomly paired shapes almost always have IoU 0, so expected disagreement collapses to about 1 and α degenerates to 1 − mean distance with no working chance correction left in it.

Plain IoU is also flat where it matters most. Two boxes that do not overlap score 0 whether they are touching or at opposite corners of the image, so the measure has no gradient exactly where annotators disagree.

What Potato reports instead

σ, the primary measure

σ = 1 − mean(within-item distance) / mean(between-item distance)

This is α's own 1 − D_o/D_e form generalized to an arbitrary distance, with the chance baseline estimated empirically by comparing annotations of different items.

Negative values are not clamped. "Further apart on the same image than on different images" is a real, diagnosable state, and hiding it helps nobody.

KS, the companion measure

The two-sample Kolmogorov–Smirnov statistic between the within-item and between-item distance distributions. σ compares two means and can be dragged by a few outliers; KS compares whole distributions and holds up better when item difficulty varies.

Report both. Their disagreeing is itself informative: it means a minority of items carry most of the disagreement.

The decomposition

A single spatial agreement number cannot say which of three different failures occurred:

An annotator who finds every object and mislabels them all is a different problem from one who labels correctly but boxes loosely, and a combined score cannot distinguish them.

Configuration

agreement_metrics:
  enabled: true

Reports appear at /admin/iaa.

Related

• Mask consensus with STAPLE
• Temporal agreement
• Guide: measuring agreement on bounding boxes
• Source documentation