# Mask Consensus with STAPLE

Source: https://www.potatoannotator.com/docs/measurement/mask-consensus-staple

**Agreement answers "do they agree". It does not answer "whose mask should the dataset record, and who drew it well".** That is a different question and needs a different model.

## Why not MACE

[MACE](/docs/features/mace) models each annotator as knowing-or-guessing over a *finite shared label set*. A mask lives in an unbounded space with no categorical variable to estimate. Forcing masks through it would mean inventing a label set, and whatever was invented would drive the answer.

## What STAPLE does

[STAPLE](https://pubmed.ncbi.nlm.nih.gov/15250643/) (Warfield, Zou and Wells, 2004) is the established tool for this in medical imaging. It runs [expectation-maximization](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm) over per-pixel labels and estimates, per annotator:

| | Meaning | A low value means |
|---|---|---|
| **Sensitivity** | Of the true foreground, how much did you include? | Under-segmenting — drawing inside the boundary |
| **Specificity** | Of the true background, how much did you correctly exclude? | Over-segmenting — drawing generously around it |

Both are reported because they have **opposite fixes**. A single "accuracy" number would score the two failure modes identically and tell an annotator nothing about what to change.

## Weighted consensus, not a majority vote

A careful annotator's disagreement moves the consensus more than a careless one's, and the algorithm works out which is which from the data itself.

Measured on two careful annotators outnumbered three-to-two by noisy ones:

| Method | Dice against truth |
|---|---|
| Majority vote | 0.846 |
| **STAPLE** | **1.000** |

A vote lets the three noisy annotators decide. STAPLE notices that the careful pair agree with each other and the noisy three do not.

## Caveats worth knowing

- An annotator with three instances of a class contributes their **union**. STAPLE compares pixel by pixel, and keeping instances apart would require matching them across annotators first — which is the detection question, answered separately in [geometry agreement](/docs/measurement/geometry-agreement).
- Items with fewer than two mask-bearing annotators are **counted and skipped**, not silently dropped. Otherwise coverage is overstated.

## Cost control

Pairwise comparison is quadratic in annotators *and* in instances per item. A five-annotator project with 50 instances per image costs 25,000 mask decodes per image, which turns an admin page load into a hang.

There is a hard budget, `max_pairs`, defaulting to 200,000. When it is hit:

- the item is skipped **whole**, never half-measured, because a partially processed item biases the mean toward whichever annotator pair happened to run;
- the report sets `truncated: true` and states how many items did fit.

A truncated agreement number that reads as complete is worse than no number, because it will be quoted.

## Related

- [Geometry agreement](/docs/measurement/geometry-agreement)
- [Interactive segmentation](/docs/vision-spatial/segmentation)
- [MACE](/docs/features/mace) — for the genuinely categorical parts
- [Source documentation](https://github.com/davidjurgens/potato/blob/main/docs/advanced/geometry_agreement.md)