How to Measure Inter-Annotator Agreement on Bounding Boxes

Mean IoU between annotators measures how easy the task is, not how much they agree. On a typical detection corpus it reads around 0.95 even for annotators who never looked at the image. Measuring agreement on spatial labels means correcting for chance, and reporting detection, classification and localization as separate numbers.

Why raw IoU misleads

Consider a corpus where each image holds one large, centred object. Two annotators who both draw a box in the middle will overlap heavily, so mean IoU comes out around 0.95. So will an annotator who drew a box in the middle without looking.

The number is measuring the task, not the annotators. That is the same problem percent agreement has on categorical labels, where one dominant class inflates the score, and it has the same fix: correct for how much agreement the task produces by accident.

CVAT's consensus engine and V7's consensus stage both compare annotators with raw IoU against a per-class threshold. Neither applies chance correction, and both are widely used, so a reported "0.9 agreement" from either usually cannot be compared with anything.

Why not Krippendorff's α over 1 − IoU

The obvious fix is to feed IoU distance into Krippendorff's α, which accepts any distance function. It does not work well here, and the reason is worth knowing before you build it.

Braylan, Alonso and Lease (WWW 2022) evaluate distance functions by whether the resulting score ranks annotator quality correctly. For bounding boxes, α ranks plain L2 (0.687) above IoU (0.505) and GIoU (0.507), which inverts the ordering that both their distribution-based measures and practitioners give.

Structurally: IoU distance is bounded in [0, 1] and saturates. Two randomly paired boxes almost always have IoU 0, so expected disagreement collapses to about 1, and α degenerates into 1 − mean distance with no chance correction left in it. IoU is also flat where it matters: two boxes that do not overlap score 0 whether they are touching or at opposite corners, so there is no gradient exactly where annotators disagree.

What to report instead

Split the question three ways

An annotator who finds every object and mislabels them all is a different problem from one who labels correctly but boxes loosely. A combined score cannot tell you which you have, and the two have different fixes.

σ, with an empirical chance baseline

σ = 1 − mean(within-item distance) / mean(between-item distance)

This is α's 1 − D_o/D_e form generalized to any distance, with the chance baseline estimated by comparing annotations of different items. σ = 0 means annotators agree no more than they would on unrelated images.

Do not clamp negative values. "Further apart on the same image than on different images" is a real state, and it almost always means the guidelines are ambiguous rather than that the annotators are careless.

KS alongside it

The two-sample Kolmogorov–Smirnov statistic between the within-item and between-item distance distributions. σ compares two means and can be dragged by a few outliers; KS compares whole distributions.

Report both. When they disagree, a minority of items is carrying most of the disagreement, which tells you to go and look at those items.

Masks need a different tool

For pixel masks, the question "whose boundary do we record" is not the same as "do they agree". Use STAPLE (Warfield, Zou and Wells, 2004), which runs expectation-maximization over per-pixel labels and estimates a sensitivity and specificity per annotator.

Sensitivity and specificity are reported separately because they have opposite fixes: low sensitivity means under-segmenting, low specificity means over-segmenting, and a single accuracy number scores both identically.

STAPLE also beats a majority vote when careful annotators are outnumbered. With two careful annotators against three noisy ones, majority vote scores Dice 0.846 against truth and STAPLE scores 1.000.

3D boxes need exact rotated IoU

For cuboids, use exact rotated 3D IoU. An axis-aligned approximation reports disagreement that is an artefact of the measure as soon as the data has real pitch and roll, which drone, handheld and indoor scans all do.

Doing it in Potato

agreement_metrics:
  enabled: true
 
annotation_schemes:
  - annotation_type: image_annotation
    name: objects
    description: "Draw a tight box around every vehicle."
    source_field: image
    tools: [bbox]
    labels:
      - {name: car, color: "#FF6B6B"}
      - {name: truck, color: "#4ECDC4"}
 
num_annotators_per_item:
  default: 2

Reports appear at /admin/iaa. Two properties are worth knowing about:

• Undefined values explain themselves. α is genuinely undefined on a unanimous corpus. A bare NaN reads as a broken computation and a 1.0 would be a lie.
• Nothing is truncated silently. Pairwise comparison is quadratic in annotators and in instances per item, so there is a budget. When it is hit, the item is skipped whole rather than half-measured, and the report says how many items were included.

Checklist

1. Get at least two independent annotations per item. Agreement over a single annotator is undefined rather than perfect.
2. Report detection, classification and localization separately.
3. Correct for chance against an empirically estimated baseline rather than a fixed threshold.
4. Report σ and KS together.
5. Use STAPLE for masks and rotated 3D IoU for cuboids.
6. State coverage. A number computed over a sample that presents itself as complete will be quoted as complete.

Further reading

• Agreement over geometry
• Mask consensus with STAPLE
• Inter-annotator agreement explained — the categorical case
• Agreement for spans
• Interactive IAA calculator