# How to Measure Inter-Annotator Agreement on Bounding Boxes

Source: https://www.potatoannotator.com/docs/guides/measuring-agreement-on-bounding-boxes

**Mean [IoU](https://en.wikipedia.org/wiki/Jaccard_index) between annotators measures how easy the task is, not how much they agree. On a typical detection corpus it reads around 0.95 even for annotators who never looked at the image.** Measuring agreement on spatial labels means correcting for chance, and reporting detection, classification and localization as separate numbers.

## Why raw IoU misleads

Consider a corpus where each image holds one large, centred object. Two annotators who both draw a box in the middle will overlap heavily, so mean IoU comes out around 0.95. So will an annotator who drew a box in the middle without looking.

The number is measuring the task, not the annotators. That is the same problem [percent agreement](/docs/guides/inter-annotator-agreement) has on categorical labels, where one dominant class inflates the score, and it has the same fix: correct for how much agreement the task produces by accident.

CVAT's consensus engine and V7's consensus stage both compare annotators with raw IoU against a per-class threshold. Neither applies chance correction, and both are widely used, so a reported "0.9 agreement" from either usually cannot be compared with anything.

## Why not Krippendorff's α over 1 − IoU

The obvious fix is to feed IoU distance into [Krippendorff's α](https://en.wikipedia.org/wiki/Krippendorff%27s_alpha), which accepts any distance function. It does not work well here, and the reason is worth knowing before you build it.

Braylan, Alonso and Lease ([WWW 2022](https://dl.acm.org/doi/10.1145/3485447.3512242)) evaluate distance functions by whether the resulting score **ranks annotator quality correctly**. For bounding boxes, α ranks plain L2 (0.687) above IoU (0.505) and GIoU (0.507), which inverts the ordering that both their distribution-based measures and practitioners give.

Structurally: IoU distance is bounded in [0, 1] and saturates. Two randomly paired boxes almost always have IoU 0, so expected disagreement collapses to about 1, and α degenerates into `1 − mean distance` with no chance correction left in it. IoU is also flat where it matters: two boxes that do not overlap score 0 whether they are touching or at opposite corners, so there is no gradient exactly where annotators disagree.

## What to report instead

### Split the question three ways

| Component | Question | Measure |
|---|---|---|
| Detection | Did they find the same objects at all? | Nominal α over presence |
| Classification | Did they give matched objects the same label? | Nominal α over labels |
| Localization | Are matched objects in the same place? | σ and KS |

An annotator who finds every object and mislabels them all is a different problem from one who labels correctly but boxes loosely. A combined score cannot tell you which you have, and the two have different fixes.

### σ, with an empirical chance baseline

```
σ = 1 − mean(within-item distance) / mean(between-item distance)
```

This is α's `1 − D_o/D_e` form generalized to any distance, with the chance baseline estimated by comparing annotations of *different* items. σ = 0 means annotators agree no more than they would on unrelated images.

Do not clamp negative values. "Further apart on the same image than on different images" is a real state, and it almost always means the guidelines are ambiguous rather than that the annotators are careless.

### KS alongside it

The two-sample [Kolmogorov–Smirnov](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test) statistic between the within-item and between-item distance distributions. σ compares two means and can be dragged by a few outliers; KS compares whole distributions.

Report both. When they disagree, a minority of items is carrying most of the disagreement, which tells you to go and look at those items.

## Masks need a different tool

For pixel masks, the question "whose boundary do we record" is not the same as "do they agree". Use [STAPLE](https://pubmed.ncbi.nlm.nih.gov/15250643/) (Warfield, Zou and Wells, 2004), which runs expectation-maximization over per-pixel labels and estimates a sensitivity and specificity per annotator.

Sensitivity and specificity are reported separately because they have **opposite fixes**: low sensitivity means under-segmenting, low specificity means over-segmenting, and a single accuracy number scores both identically.

STAPLE also beats a majority vote when careful annotators are outnumbered. With two careful annotators against three noisy ones, majority vote scores Dice 0.846 against truth and STAPLE scores 1.000.

## 3D boxes need exact rotated IoU

For [cuboids](/docs/vision-spatial/point-clouds), use exact rotated 3D IoU. An axis-aligned approximation reports disagreement that is an artefact of the measure as soon as the data has real pitch and roll, which drone, handheld and indoor scans all do.

## Doing it in Potato

```yaml
agreement_metrics:
  enabled: true

annotation_schemes:
  - annotation_type: image_annotation
    name: objects
    description: "Draw a tight box around every vehicle."
    source_field: image
    tools: [bbox]
    labels:
      - {name: car, color: "#FF6B6B"}
      - {name: truck, color: "#4ECDC4"}

num_annotators_per_item:
  default: 2
```

Reports appear at `/admin/iaa`. Two properties are worth knowing about:

- **Undefined values explain themselves.** α is genuinely undefined on a unanimous corpus. A bare `NaN` reads as a broken computation and a `1.0` would be a lie.
- **Nothing is truncated silently.** Pairwise comparison is quadratic in annotators and in instances per item, so there is a budget. When it is hit, the item is skipped whole rather than half-measured, and the report says how many items were included.

## Checklist

1. Get at least two independent annotations per item. Agreement over a single annotator is undefined rather than perfect.
2. Report detection, classification and localization separately.
3. Correct for chance against an empirically estimated baseline rather than a fixed threshold.
4. Report σ and KS together.
5. Use STAPLE for masks and rotated 3D IoU for cuboids.
6. State coverage. A number computed over a sample that presents itself as complete will be quoted as complete.

## Further reading

- [Agreement over geometry](/docs/measurement/geometry-agreement)
- [Mask consensus with STAPLE](/docs/measurement/mask-consensus-staple)
- [Inter-annotator agreement explained](/docs/guides/inter-annotator-agreement) — the categorical case
- [Agreement for spans](/docs/guides/agreement-for-spans)
- [Interactive IAA calculator](/tools/inter-annotator-agreement)