Annotation statistics tools

Krippendorff's alpha (nominal, ordinal, interval), Cohen's kappa (unweighted and weighted), Fleiss' kappa, and raw agreement from a CSV of labels, with bootstrap confidence intervals and a breakdown by label and annotator pair.

Agreement on bounding boxes, text spans and time segments, split into detection, label and location, with a chance-corrected localization score and a boundary tolerance sweep.

Compare an LLM judge with human labels using judge-human against human-human kappa, the alternative annotator test, and ECE and Brier score for the judge's confidence.

Turn noisy crowd labels into consensus labels with the Dawid-Skene model or MACE, and estimate each annotator's reliability from the data alone.

Fit strengths and win probabilities from pairwise preference judgments, the model behind LLM arena leaderboards. Elo included for comparison.

Score items from best-worst scaling judgments with the counting method and a Bradley-Terry fit, and check the ranking with split-half reliability.

Simulate how the confidence interval on Krippendorff's alpha narrows as you add items, and get a corpus-size estimate before you spend the budget.

Estimate what a study costs and how long it takes, with paid training, attention checks, adjudication and the Prolific or MTurk fee included.