Dawid-Skene consensus label calculator
Aggregate noisy labels from multiple annotators into consensus labels with the Dawid-Skene EM model, and see which annotators the model trusts. Runs in your browser; nothing is uploaded.
Computed locally in your browser. Your data is not uploaded anywhere.
Why not just take the majority vote?
Majority voting weights every annotator equally. In practice some annotators are careful, some misunderstand one specific label, and a few click through at random. The Dawid-Skene model (Dawid & Skene, 1979) estimates a confusion matrix for each annotator and the true label of each item jointly, using expectation-maximization. An annotator who is usually right gets more say; one who systematically confuses two labels gets corrected rather than discarded. On items where the model and the majority disagree, the model is using reliability evidence the vote count cannot see.
The estimated accuracy column doubles as quality control: it flags unreliable annotators without requiring gold-standard questions. It only works when annotators overlap on shared items, so plan at least three ratings per item if you intend to use it.
Related reading
The guide on aggregating crowd labels compares majority vote, Dawid-Skene, and newer aggregation models, and adjudication and disagreement covers what to do with the items the model is unsure about. Potato ships Dawid-Skene consensus in its evaluation pipeline, so the numbers here match what the tool computes on a live study.