# How Many Annotators Do You Actually Need?

Source: https://www.potatoannotator.com/blog/how-many-annotators-do-you-need

"How many annotators do I need?" is one of the first questions on any project, and the honest answer is that it depends on three things: how clear the task is, how much you can spend, and what you plan to do with the disagreements. There is no magic number, but there are good defaults.

## Coverage versus overlap

Every annotation budget gets split between two competing goals. Coverage means labeling more distinct items, each once. Overlap means labeling the same items several times so you can measure agreement and aggregate. You cannot maximize both at once.

A pattern that works well: fully overlap a small subset to measure agreement and confirm the task is well-defined, then single-annotate the rest once you trust it. You get a quality signal without paying to label everything three times.

## Rules of thumb

For tasks with clear categories and high agreement, one annotator handles most items, with two or three annotators overlapping on a 5 to 10 percent sample to monitor quality.

For moderately subjective tasks, use three annotators per item and resolve with majority vote or a competence-weighted model.

For genuinely subjective work, like judging offense, emotion, or preference, use five or more annotators per item, and consider keeping the full distribution of labels instead of collapsing it to one answer. The disagreement is often real signal, not noise.

More annotators reduce the variance of an item's aggregate label, but with diminishing returns. Going from one annotator to three helps far more than going from seven to nine.

## Setting overlap in Potato

Potato's automatic assignment controls how many annotators see each item and how items are distributed across people.

```yaml
automatic_assignment:
  on: true
  instance_per_annotator: 50     # items each person labels
  labels_per_instance: 3         # annotators per item (overlap)
```

## Headcount is not a substitute for quality control

Adding annotators does not help if some of them are unreliable. Pair overlap with gold-standard items and attention checks so you can weight or drop low-quality work before you aggregate. For the full reasoning, see the [How Many Annotators Do You Need?](/docs/guides/how-many-annotators) guide and [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement). For implementation details, see the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/advanced/task_assignment.md) and the [quality control docs](https://github.com/davidjurgens/potato/blob/master/docs/workflow/quality_control.md).