# How to Annotate Image Segmentation Masks

Source: https://www.potatoannotator.com/docs/guides/image-segmentation-annotation

**Segmentation labels which pixels belong to an object, and almost all of its difficulty is in the boundary rule rather than the drawing tool.** Two annotators given the same image and no rule about shadows will disagree systematically, and no amount of tooling fixes that.

## Polygon or mask

| | Polygon | Brush mask |
|---|---|---|
| Holes and disconnected parts | Awkward | Natural |
| Thin structures (wires, hair, legs) | Poor | Good |
| Format support | Wide: COCO, YOLO, CVAT, LabelMe | Narrower: COCO RLE, DAVIS, mask PNG |
| Editability | Move a vertex | Repaint a region |
| Storage | Small | Large unless sparse |

Polygons round-trip through more formats, which matters if the labels have to leave the tool. Masks are the honest answer when the object genuinely has holes or wisps.

If you are unsure, look at what the model consuming the labels needs. A [semantic segmentation](https://en.wikipedia.org/wiki/Image_segmentation) model consumes masks either way; the question is only what the annotators find least error-prone.

## Write the boundary rule first

This is the part that determines your agreement number. Decide, and give examples for:

- **Shadow and reflection.** In or out? Consistently.
- **Transparent and translucent parts.** A glass, a windscreen, a plastic bag.
- **Thin structures.** Where does a wire stop being worth tracing?
- **Occlusion.** Mask only the visible part, or infer the whole object?
- **Contact.** Two touching instances: where exactly is the split?
- **Motion blur.** Where is the boundary of a blurred edge?

A single worked example per rule is worth more than a paragraph of prose. If annotators disagree after this, you will see it as **low σ with high detection agreement** — they are finding the same objects and drawing them differently, which is a guidelines problem rather than a carelessness problem.

## Start from a model

Click-to-segment gives a first pass in roughly 130 ms, in the browser, with no GPU:

```yaml
annotation_schemes:
  - annotation_type: image_annotation
    name: objects
    description: "Outline every object."
    source_field: image
    tools: [sam, polygon, brush, eraser]
    labels:
      - {name: object, color: "#6e56cf"}
```

```bash
potato download-models onnxruntime
potato download-models mobile_sam
```

Positive clicks add to a mask, negative clicks cut away, a box constrains the search, and an existing mask can be refined.

**The correction is the annotation value.** If your annotators accept the first mask every time, you have collected a record of the model agreeing with itself — see [telling review from rubber-stamping](/docs/guides/detecting-rubber-stamped-prelabels), because agreement statistics will not tell you.

## Measuring agreement on masks

Do not use mean IoU between annotators. It answers "is this task easy", not "do these annotators agree" — see [measuring agreement on bounding boxes](/docs/guides/measuring-agreement-on-bounding-boxes) for why.

Use [STAPLE](https://pubmed.ncbi.nlm.nih.gov/15250643/), which estimates a latent consensus boundary along with a **sensitivity** and **specificity** per annotator:

- Low sensitivity means under-segmenting, drawing inside the true boundary.
- Low specificity means over-segmenting, drawing generously around it.

They have opposite fixes, which is why a single accuracy number is not enough to give an annotator feedback.

STAPLE also weights by demonstrated reliability rather than counting votes. Two careful annotators outnumbered three-to-two by noisy ones score Dice 0.846 under majority vote and 1.000 under STAPLE.

## Export

COCO carries polygons and RLE masks, including `iscrowd` crowd regions. DAVIS carries per-instance masks. YOLO carries polygons but not masks. Check the [format matrix](/docs/vision-spatial/cv-formats) before committing to a mask workflow if the labels have to end up somewhere specific.

## Further reading

- [Interactive segmentation](/docs/vision-spatial/segmentation)
- [Mask consensus with STAPLE](/docs/measurement/mask-consensus-staple)
- [Geometry primitives](/docs/vision-spatial/geometry-primitives)
- [Image annotation guide](/docs/guides/image-annotation)