# Expert vs Crowd Annotation

Source: https://www.potatoannotator.com/docs/guides/expert-vs-crowd-annotation

**Use crowd annotators when a careful non-specialist following your guidelines would reach the same judgment as a specialist, and collect several judgments per item rather than one. [Snow et al. (2008)](https://aclanthology.org/D08-1027/) found that an average of four non-expert annotations matched the agreement of a single expert annotator across five natural language tasks.**

The choice between expert and crowd annotators is usually framed as a tradeoff between quality and cost, which is the wrong frame for most tasks. The question is whether the judgment depends on training the crowd does not have. Sentiment, textual entailment, and word similarity mostly do not. Clinical coding, legal classification, and phonetic transcription mostly do.

## What the evidence supports

The foundational result comes from [Snow et al. (2008)](https://aclanthology.org/D08-1027/), who compared non-expert annotations from Amazon Mechanical Turk against expert gold standards on affect recognition, word similarity, recognizing textual entailment, event temporal ordering, and word sense disambiguation. Pooling across tasks, they report that "it requires only 4 non-expert annotations per example to achieve the equivalent ITA as a single expert annotator". They also report a rate rather than a ratio for cost, collecting 7,000 affect annotations for two dollars and describing that as "at least 875 expert-equivalent labels per USD".

The per-task variation inside that average matters more than the average. On their affect task, anger, disgust, and sadness reached expert level with two non-experts, surprise needed nine, and fear never reached it within the ten annotators they collected. A task-level average hides the labels where the crowd cannot substitute for training.

[Callison-Burch (2009)](https://aclanthology.org/D09-1030/) found the same pattern for machine translation evaluation, recreating WMT08 judgments for ten dollars and reporting that combined non-expert judgments "correlate more strongly with expert judgments than Bleu does".

The counterweight is that experts are sometimes faster. [Alfter et al. (2022)](https://doi.org/10.3384/nejlt.2000-1533.2021.3128) ran a best-worst scaling task on multi-word expressions with second-language speakers, language professionals, and CEFR experts. The three groups' rankings correlated highly, so quality was comparable, but direct expert annotation took 15 to 90 minutes per project against roughly eight to nine hours for the crowd version, which they describe as "at least five times as fast". Where a small number of specialists is already available, routing the work to a crowd can cost more time than it saves.

## Redundancy is the mechanism, not the crowd

A single crowd judgment is worse than a single expert judgment in every study above. What closes the gap is collecting several judgments per item and combining them, so the design question is how to combine them.

Majority voting treats every annotator as equally reliable, which they are not. Two models estimate reliability from the data instead. [Dawid and Skene (1979)](https://doi.org/10.2307/2346806) introduced an expectation-maximization procedure that estimates a confusion matrix per observer along with the latent true labels, originally for clinical observer error. [MACE](https://aclanthology.org/N13-1132/), or Multi-Annotator Competence Estimation, is an item-response model built for the crowdsourcing case, learning without supervision which annotators are trustworthy and predicting the underlying labels. Its authors built it because "some annotators choose bad labels in order to maximize their pay", a failure mode majority voting absorbs silently when enough annotators share it.

Potato ships both. A `mace` block runs competence estimation over collected annotations, and the [Dawid-Skene calculator](/tools/dawid-skene) runs the estimator on labels you paste in.

```yaml
mace:
  enabled: true

num_annotators_per_item: 4

gold_standards:
  enabled: true

attention_checks:
  enabled: true
```

`num_annotators_per_item: 4` follows the Snow et al. finding rather than a house default, and it is the value to revisit per task rather than to carry across projects. Gold standards and attention checks catch the annotator who is not reading, which competence models handle better when they have some anchored items to calibrate against.

## Disagreement is not automatically error

Treating every disagreement as a quality problem leads to the wrong intervention. [Plank et al. (2014)](https://doi.org/10.3115/v1/P14-2083) analyzed annotator disagreements in part-of-speech annotation and found that "only 2% of annotator choices are linguistically unmotivated", concluding that the majority of disagreements come from linguistically debatable cases rather than from errors.

[Pavlick and Kwiatkowski (2019)](https://aclanthology.org/Q19-1043/) tested whether disagreement in natural language inference disappears with more data, and it does not. They report that disagreements "are not dismissible as annotation 'noise', but rather persist as we collect more ratings and as we vary the amount of context provided to raters". [Plank (2022)](https://doi.org/10.18653/v1/2022.emnlp-main.731) generalizes the point as human label variation.

The practical test is cheap. Collect more judgments on the items where annotators disagree. If agreement converges, the disagreement was noise and redundancy fixes it. If it stays flat, the item is genuinely ambiguous, and the right response is to record the distribution rather than to force a single label.

## Training changes agreement more than annotator type does

[Bayerl and Paul (2011)](https://doi.org/10.1162/COLI_a_00074) meta-analyzed 96 annotation studies across word-sense disambiguation, prosodic transcription, and phonetic transcription, covering 346 agreement indices. Seven factors explained variation in agreement, and two of them concern training, namely whether annotators received training at all and how intensive it was. The others are the annotation domain, the number of categories in the scheme, the number of annotators, the annotation purpose, and the method used to calculate percentage agreement.

That result reframes the expert-versus-crowd decision. A trained crowd annotator working from a tested scheme with few categories will often agree more than an untrained specialist working from a vague one. [Writing annotation guidelines](/docs/guides/writing-annotation-guidelines) and [choosing an annotation scheme](/docs/guides/choosing-an-annotation-scheme) do more for agreement than the recruitment channel does.

## Where crowd annotation fails

Three conditions make the crowd the wrong choice. The judgment requires credentialed knowledge, as in clinical or legal coding, where a wrong label is not merely noisy. The data cannot leave your infrastructure, which rules out a public marketplace regardless of quality. The label set is large or hierarchical enough that training cost per annotator exceeds the saving, which is common above roughly a few dozen categories.

When the first condition holds but the volume is high, a mixed design usually beats either extreme. Experts label a subset, that subset becomes the gold standard, and crowd annotators are screened against it before reaching the real task. [Gold standards and attention checks](/docs/guides/gold-standards-and-attention-checks) covers the screening mechanics.

## Further reading

- [How many annotators do you need?](/docs/guides/how-many-annotators) works through redundancy and statistical power.
- [Aggregating crowd labels](/docs/guides/aggregating-crowd-labels) covers combining judgments once collected.
- [Crowdsourcing with Prolific and MTurk](/docs/guides/crowdsourcing-prolific-mturk) covers recruitment and payment.
- [Inter-annotator agreement explained](/docs/guides/inter-annotator-agreement) covers the coefficients.

## References

Snow, R., O'Connor, B., Jurafsky, D., and Ng, A. Y. (2008). Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. *Proceedings of EMNLP 2008*, 254-263. https://aclanthology.org/D08-1027/

Callison-Burch, C. (2009). Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon's Mechanical Turk. *Proceedings of EMNLP 2009*, 286-295. https://aclanthology.org/D09-1030/

Dawid, A. P., and Skene, A. M. (1979). Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. *Journal of the Royal Statistical Society. Series C (Applied Statistics)*, 28(1), 20-28. https://doi.org/10.2307/2346806

Hovy, D., Berg-Kirkpatrick, T., Vaswani, A., and Hovy, E. (2013). Learning Whom to Trust with MACE. *Proceedings of NAACL-HLT 2013*, 1120-1130. https://aclanthology.org/N13-1132/

Plank, B., Hovy, D., and Søgaard, A. (2014). Linguistically debatable or just plain wrong? *Proceedings of ACL 2014 (Volume 2: Short Papers)*, 507-511. https://doi.org/10.3115/v1/P14-2083

Pavlick, E., and Kwiatkowski, T. (2019). Inherent Disagreements in Human Textual Inferences. *Transactions of the Association for Computational Linguistics*, 7, 677-694. https://aclanthology.org/Q19-1043/

Bayerl, P. S., and Paul, K. I. (2011). What Determines Inter-Coder Agreement in Manual Annotations? A Meta-Analytic Investigation. *Computational Linguistics*, 37(4), 699-725. https://doi.org/10.1162/COLI_a_00074

Alfter, D., Lindström Tiedemann, T., and Volodina, E. (2022). Crowdsourcing Relative Rankings of Multi-Word Expressions: Experts versus Non-Experts. *Northern European Journal of Language Technology*, 7(1). https://doi.org/10.3384/nejlt.2000-1533.2021.3128

Plank, B. (2022). The "Problem" of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation. *Proceedings of EMNLP 2022*, 10671-10682. https://doi.org/10.18653/v1/2022.emnlp-main.731