Skip to content
Cette page n'est pas encore disponible dans votre langue. La version anglaise est affichée.

Mask Consensus with STAPLE

Which mask should the dataset record, and who drew it well? STAPLE estimates a latent boundary with per-rater sensitivity and specificity, and beats majority vote when careful annotators are outnumbered.

Agreement answers "do they agree". It does not answer "whose mask should the dataset record, and who drew it well". That is a different question and needs a different model.

Why not MACE

MACE models each annotator as knowing-or-guessing over a finite shared label set. A mask lives in an unbounded space with no categorical variable to estimate. Forcing masks through it would mean inventing a label set, and whatever was invented would drive the answer.

What STAPLE does

STAPLE (Warfield, Zou and Wells, 2004) is the established tool for this in medical imaging. It runs expectation-maximization over per-pixel labels and estimates, per annotator:

MeaningA low value means
SensitivityOf the true foreground, how much did you include?Under-segmenting — drawing inside the boundary
SpecificityOf the true background, how much did you correctly exclude?Over-segmenting — drawing generously around it

Both are reported because they have opposite fixes. A single "accuracy" number would score the two failure modes identically and tell an annotator nothing about what to change.

Weighted consensus, not a majority vote

A careful annotator's disagreement moves the consensus more than a careless one's, and the algorithm works out which is which from the data itself.

Measured on two careful annotators outnumbered three-to-two by noisy ones:

MethodDice against truth
Majority vote0.846
STAPLE1.000

A vote lets the three noisy annotators decide. STAPLE notices that the careful pair agree with each other and the noisy three do not.

Caveats worth knowing

  • An annotator with three instances of a class contributes their union. STAPLE compares pixel by pixel, and keeping instances apart would require matching them across annotators first — which is the detection question, answered separately in geometry agreement.
  • Items with fewer than two mask-bearing annotators are counted and skipped, not silently dropped. Otherwise coverage is overstated.

Cost control

Pairwise comparison is quadratic in annotators and in instances per item. A five-annotator project with 50 instances per image costs 25,000 mask decodes per image, which turns an admin page load into a hang.

There is a hard budget, max_pairs, defaulting to 200,000. When it is hit:

  • the item is skipped whole, never half-measured, because a partially processed item biases the mean toward whichever annotator pair happened to run;
  • the report sets truncated: true and states how many items did fit.

A truncated agreement number that reads as complete is worse than no number, because it will be quoted.