Annotation power planner
How many items do you need before your agreement estimate is worth reporting? Describe the study, and this tool simulates hundreds of synthetic versions of it to show how the confidence interval on Krippendorff's alpha narrows as the corpus grows.
Study profile
Why plan corpus size at all?
An agreement coefficient computed on 30 items can swing by 0.2 or more between otherwise identical studies. Reviewers increasingly ask for confidence intervals on reliability numbers, and a wide interval can undermine a paper even when the point estimate looks fine. Deciding the corpus size after seeing the interval you want is cheaper than annotating twice.
The simulation here uses a deliberately simple generative model, stated under the results table, so you can judge whether it fits your task. It answers the planning question with Monte Carlo rather than a closed-form formula because no clean formula exists for Krippendorff's alpha with more than two annotators and missing data. For a two-rater kappa there are analytic approaches (for example Sim & Wright, 2005); the simulated and analytic answers agree closely in that special case.
One number this tool will not improve: expected agreement itself. If the simulated alpha under honest assumptions lands below 0.667, adding items only narrows the interval around a bad number. Fix the codebook, the training, or the task definition first, then plan the corpus.
Related reading
The guide on statistical power in annotation studies goes deeper on study design, and the agreement calculator computes the real interval once pilot data exists. In Potato you can watch agreement stabilize live as annotations arrive, which is the empirical version of this plan.