# Agreement over Geometry

Source: https://www.potatoannotator.com/docs/measurement/geometry-agreement

**Potato decomposes spatial agreement into three questions — did annotators find the same objects, call them the same thing, and put them in the same place — and answers the third against an empirically estimated chance baseline rather than with raw IoU.**

## The problem with raw IoU

CVAT's consensus engine and V7's consensus stage both compare annotators with **raw [IoU](https://en.wikipedia.org/wiki/Jaccard_index) against a per-class threshold**. Neither applies any chance correction, which makes the headline number uninterpretable on its own:

> A corpus where every image holds one large, centred object will show a mean IoU around 0.95 no matter who annotates it. That includes annotators who never looked at the image and simply drew a box in the middle.

Chance correction is what separates "these annotators agree" from "this task is too easy to disagree on".

## Krippendorff's α over 1 − IoU does not work here

That was the original plan here, and it is empirically the wrong default.

Braylan, Alonso and Lease ([WWW 2022](https://dl.acm.org/doi/10.1145/3485447.3512242)) evaluate candidate distance functions by whether the resulting agreement score **ranks annotator quality correctly** — the property that actually matters — and report that for bounding boxes, α ranks plain L2 (0.687) *above* IoU (0.505) and GIoU (0.507), inverting the ordering their own distribution-based measures and practitioners both give.

There is a structural reason. IoU distance is bounded in [0, 1] and saturates: two randomly paired shapes almost always have IoU 0, so expected disagreement collapses to about 1 and α degenerates to `1 − mean distance` with no working chance correction left in it.

Plain IoU is also **flat where it matters most**. Two boxes that do not overlap score 0 whether they are touching or at opposite corners of the image, so the measure has no gradient exactly where annotators disagree.

## What Potato reports instead

### σ, the primary measure

```
σ = 1 − mean(within-item distance) / mean(between-item distance)
```

This is α's own `1 − D_o/D_e` form generalized to an arbitrary distance, with the chance baseline estimated **empirically** by comparing annotations of *different* items.

| σ | Meaning |
|---|---|
| 1.0 | Perfect agreement |
| 0.0 | Annotators agree no more than they would on unrelated items |
| below 0 | Systematic disagreement, usually a *definition* problem rather than carelessness |

Negative values are **not clamped**. "Further apart on the same image than on different images" is a real, diagnosable state, and hiding it helps nobody.

### KS, the companion measure

The two-sample [Kolmogorov–Smirnov](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test) statistic between the within-item and between-item distance *distributions*. σ compares two means and can be dragged by a few outliers; KS compares whole distributions and holds up better when item difficulty varies.

Report both. Their disagreeing is itself informative: it means a minority of items carry most of the disagreement.

## The decomposition

A single spatial agreement number cannot say which of three different failures occurred:

| Component | Question | Measure |
|---|---|---|
| **Detection** | Did they find the same objects at all? | nominal α over presence |
| **Classification** | Did they give matched objects the same label? | nominal α over labels |
| **Localization** | Are matched objects in the same place? | σ and KS, plus GIoU distance |

An annotator who finds every object and mislabels them all is a different problem from one who labels correctly but boxes loosely, and a combined score cannot distinguish them.

## Configuration

```yaml
agreement_metrics:
  enabled: true
```

Reports appear at `/admin/iaa`.

## Related

- [Mask consensus with STAPLE](/docs/measurement/mask-consensus-staple)
- [Temporal agreement](/docs/measurement/temporal-agreement)
- [Guide: measuring agreement on bounding boxes](/docs/guides/measuring-agreement-on-bounding-boxes)
- [Source documentation](https://github.com/davidjurgens/potato/blob/main/docs/advanced/geometry_agreement.md)