Data Annotation Glossary

This glossary defines 50 terms used in data annotation, grouped by the stage of a project where each term comes up. Each entry gives a definition in one to three sentences and links to the Potato guide that covers the term in depth.

The groups run from study design through agreement, quality control, model-assisted labeling, and LLM evaluation to qualitative coding and dataset documentation. Where a standard external definition exists, the entry links to Wikipedia or to the paper that introduced the term. The annotation concepts FAQ answers common questions about annotation, such as how many annotators a task needs. The annotation learning path puts the guides in reading order by goal.

Study design

Item

An item is one unit in a data file that an annotator labels, such as a sentence or an image. Potato reads each item's ID and content from the fields named under item_properties in the config. Data Formats covers the file layouts Potato reads.

Annotation scheme

An annotation scheme is one question or labeling component that annotators answer for each item, such as a single-choice category or a set of highlighted spans. A Potato config lists schemes under annotation_schemes, and each scheme names an annotation_type. Choosing an Annotation Scheme matches the kind of answer a study needs to a scheme.

Label set

A label set, or tagset, is the closed list of categories annotators choose from in one scheme. In a single-choice scheme the categories should not overlap and should cover every item, with an "Other" label where the list falls short. Named Entity Recognition shows how a team trims a standard entity label set to a domain.

Annotation guidelines

Annotation guidelines are the written rules that tell annotators which label applies to which item, with definitions and worked examples of hard cases. Guidelines change during a study, and a version number on each revision lets a reader trace a label to the rules in force when the label was given. Writing Effective Annotation Guidelines gives a structure for the document.

Pilot

A pilot is a small round of annotation run before the main study to find the rules and items that annotators read differently. The team reviews every disagreement from the pilot, fixes the guidelines where a rule was unclear, and runs another pilot until agreement settles. Pilot, then revise describes the loop.

Overlap

Overlap is the number of annotators who label each item independently. An overlap of two or more makes agreement measurable, and on a fixed budget each extra annotator per item means fewer distinct items labeled. Potato sets overlap with num_annotators_per_item, and How Many Annotators Do You Need? gives rules of thumb by task.

Span

A span is a labeled stretch inside an item, recorded as a start position, an end position, and a label. Named entities in text and hallucinated claims in model output are both annotated as spans. Span Annotation covers span tasks in text and audio.

IOB tagging

IOB tagging, also called BIO tagging, encodes spans as one tag per token. B- marks the first token of a span, I- marks a later token inside the span, and O marks a token outside every span. The format dates to Ramshaw and Marcus (1995), and Potato's conll_2003 export converts span annotations to BIO tags.

Likert scale

A Likert scale asks annotators to rate agreement or intensity on a short ordered set of labeled points, commonly five or seven. Because Likert ratings are ordinal, agreement on Likert ratings calls for ordinal alpha or weighted kappa rather than a nominal coefficient. Rating Scales and Likert Design covers the number of points and the labels.

Pairwise comparison

Pairwise comparison shows an annotator two items and asks which of the two items is better on a stated criterion. Each annotator uses an absolute scale differently, and a comparison asks only which item is better. Pairwise and Best–Worst Scaling compares the format with ratings.

Best–worst scaling (BWS)

Best–worst scaling shows an annotator a small set of items, often four, and asks for the best and the worst item on a stated criterion. Counting how often each item is chosen best and worst gives a real-valued score per item, and Kiritchenko and Mohammad (2017) found BWS more reliable than a rating scale for sentiment intensity. Potato's bws annotation type implements the format, and the best–worst scaling calculator turns choices into scores.

Bradley–Terry model

The Bradley–Terry model estimates one score per item from many pairwise comparisons. In the model, the probability that one item beats another depends only on the two items' scores. Pairwise Model Comparison uses the model to rank LLMs from human votes, and the Bradley–Terry calculator fits the model to a CSV of comparisons.

Bounding box

A bounding box is a rectangle an annotator draws around an object in an image or a video frame, stored as coordinates and a label. Image Annotation covers boxes alongside polygons and landmarks.

Intersection over union (IoU)

Intersection over union divides the area two shapes share by the area the two shapes cover together, from 0 for no overlap to 1 for identical shapes. IoU is the Jaccard index applied to regions, and temporal IoU applies the same ratio to time segments. Mean IoU between annotators reads high on large, centered objects even without agreement, and How to Measure Inter-Annotator Agreement on Bounding Boxes corrects the figure for chance.

Agreement and reliability

Inter-annotator agreement (IAA)

Inter-annotator agreement is the extent to which annotators who label the same items independently give the same labels. A study reports IAA with a chance-corrected coefficient as evidence that the task is well defined. Inter-Annotator Agreement Explained covers the coefficients, and Wikipedia treats the topic as inter-rater reliability.

Percent agreement

Percent agreement is the share of items, or of annotator pairs on an item, that received the same label. Percent agreement ignores chance, and two annotators who each pick one label for 90% of items agree on about 82% of items even when both label at random. Potato's reliability export reports percent agreement beside Krippendorff's alpha for each question.

Chance-corrected agreement

A chance-corrected coefficient compares observed agreement with the agreement expected if annotators labeled at random at the observed label frequencies. A value of 1 means perfect agreement, 0 means agreement at chance level, and a negative value means less agreement than chance. Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha are the common chance-corrected coefficients.

Cohen's kappa

Cohen's kappa measures chance-corrected agreement between exactly two annotators on the same items. Cohen (1960) introduced the coefficient for nominal categories, and weighted kappa extends kappa to ordered categories by giving partial credit to near misses. The agreement calculator computes kappa from a CSV of labels.

Fleiss' kappa

Fleiss' kappa measures chance-corrected agreement when every item receives the same number of labels, from any number of annotators. Fleiss (1971) introduced the coefficient for nominal categories. With missing labels or uneven overlap, Krippendorff's alpha fits the data better.

Krippendorff's alpha

Krippendorff's alpha measures chance-corrected agreement among any number of annotators and tolerates missing labels. Versions of alpha exist for nominal, ordinal, interval, and ratio data. Intercoder Reliability in Qualitative Research explains the conventional 0.800 and 0.667 thresholds, and the agreement guide's FAQ covers negative values.

Pairwise F1

Pairwise F1 measures agreement on spans by treating one annotator's spans as the reference and another annotator's spans as predictions, then averaging F1 over every pair of annotators. Kappa fits span tasks poorly because the text nobody marked forms no fixed set of items to count. Agreement for Spans and Structured Outputs covers exact and partial boundary matching, and the span and box agreement calculator computes the scores.

Agreement drift

Agreement drift is a gradual change in how annotators apply the guidelines over the course of a study. A single agreement check before the main study cannot detect drift. Potato's agreement report at /admin/iaa scores agreement in successive windows of finished items and prompts a re-calibration round when the latest window falls below the project baseline by the fraction set in calibration.drop_threshold.

Quality control

Gold standard

A gold-standard item has an answer verified by an expert. Mixing gold items into the annotation stream gives a running accuracy score for each annotator, and some crowdsourcing tools call gold items honeypots. Potato's gold_standards block sets how often gold items appear and the minimum accuracy (accuracy.min_threshold, 0.7 by default).

Attention check

An attention check is an item whose instructions state the correct answer, such as "Select Disagree for this item." An annotator who misses the stated answer was not reading, and Oppenheimer, Meyvis, and Davidenko (2009) named the survey version an instructional manipulation check. Potato's attention_checks block inserts a check every frequency items, warns an annotator after 2 failures, and blocks the annotator after 5 by default.

Training phase

A training phase gives annotators practice items with known answers and feedback before the main task, and admits to the main task only the annotators who meet a passing score. Potato's training block sets the practice items, the passing criteria, and the action on failure. Training Phase documents the options.

Adjudication

In adjudication, an expert reviews the items annotators labeled differently and sets the final label. Potato's adjudication block queues every item whose agreement falls below agreement_threshold (0.75 by default) for the users listed in adjudicator_users. Adjudication and Resolving Disagreement compares adjudication with aggregation.

Majority vote

Majority vote takes the label most annotators gave as an item's final label. Majority vote counts a careless annotator and a careful annotator equally, and a 2–1 split counts the same as a 3–0 sweep. Aggregating Crowd Labels covers models that weight annotators by estimated reliability.

Dawid–Skene model

The Dawid–Skene model estimates each item's true label and each annotator's confusion matrix together, from the matrix of labels alone, with the EM algorithm. Dawid and Skene (1979) introduced the model to estimate observer error rates. The Dawid–Skene calculator fits the model to a CSV of labels, and Aggregating Crowd Labels compares the model with majority vote.

MACE

MACE (Multi-Annotator Competence Estimation) models each annotator as either knowing the answer or guessing, and estimates each item's true label along with a competence score from 0 to 1 for each annotator. Hovy et al. (2013) introduced MACE. MACE documents how Potato runs the model on a study's labels.

Soft label

A soft label records a probability distribution over categories for an item in place of one category. Soft labels come from one annotator spreading points across labels or from the spread of several annotators' choices, and soft labels keep the disagreement that majority vote discards. Potato's soft_label annotation type collects the first kind.

Model-assisted annotation

Pre-annotation

Pre-annotation shows annotators labels proposed by a model, called pre-labels. Annotators accept, correct, or reject each pre-label, and checking a good pre-label takes less time than labeling from scratch. Potato's pre_annotation block loads predictions from a file or from a field in the data.

Rubber-stamping

Rubber-stamping is accepting pre-labels without checking the pre-labels, a form of automation bias. No measure computed from the labels alone separates rubber-stamped data from reviewed data, and agreement rises because annotators who accept the same pre-label agree by construction. How to Tell Review from Rubber-Stamping in Pre-Labeled Data separates the two by the timing of the work.

Active learning

Active learning orders the annotation queue by how informative each unlabeled item is to a model trained on the labels collected so far. The aim is a target model accuracy with fewer labels than random sampling needs. Active Learning for Annotation covers the loop, and Potato's active_learning block configures the loop in a study.

Uncertainty sampling

Uncertainty sampling is the simplest active-learning strategy and picks the items whose predicted labels the model is least confident about. The chosen items cluster near the decision boundary, and the labeled sample stops representing the corpus. Potato's random_sample_percent keeps a share of the queue random (0.2 by default), and the Active Learning reference documents the other strategies.

LLM and agent evaluation

LLM-as-a-judge

An LLM judge is a language model prompted to grade or compare outputs, usually against a rubric. Zheng et al. (2023) found GPT-4 as a judge agreed with both controlled and crowdsourced human preferences more than 80% of the time. How to Measure Agreement Between an LLM Judge and Human Annotators compares a judge's agreement with people to the people's agreement with each other.

Calibration

A judge or classifier is calibrated when stated confidence matches accuracy, and about 80% of a calibrated judge's verdicts at 80% confidence are correct. In annotation management, a calibration round brings annotators back to shared examples with known answers to realign the annotators' reading of the guidelines. The LLM judge guide covers the first sense and Agreement Drift covers the second.

Rubric

A rubric breaks one quality judgment into named criteria, each scored on a defined scale. Scoring each criterion separately shows which criterion an output failed, where one overall score hides the reason. Rubric-Based LLM Evaluation covers rubric design.

Preference data

Preference data records which of two or more model responses people prefer for the same prompt. Pairwise preference judgments are the usual training data for the reward model in reinforcement learning from human feedback (RLHF). Collecting RLHF and Preference Data covers collection with Potato's pairwise annotation type.

Agent trajectory

An agent trajectory records every step of one agent run, from the agent's reasoning to each tool call and tool output. Annotating a trajectory means judging the run as a whole and marking the steps that went wrong, with an error category and a severity for each. Annotating Agent Trajectories covers the labels to collect.

Process reward model (PRM)

A process reward model scores each reasoning step of a solution rather than only the final answer, and training a PRM needs step-level labels that say whether each step is correct. Lightman et al. (2023) released 800,000 step-level human labels as PRM800K and found process supervision outperformed outcome supervision on MATH problems. Process Reward Models and Step-Level Labeling covers collection with Potato's process_reward annotation type.

Hallucination

A hallucination is a statement in model output that neither the input nor known facts support. Marking the unsupported words as a span with an error label gives more usable data than one flag on the whole answer. Detecting Hallucinations with Span Annotation gives a label set for the errors.

Qualitative coding

Code and coding

In qualitative research, coding attaches short labels, called codes, to excerpts of interviews, field notes, or open-ended answers. A deductive code comes from theory or the research questions before coding starts, and an inductive code comes from the data during coding. Preparing Interview Transcripts for Qualitative Coding covers the steps from recordings to coded excerpts.

Codebook

A codebook lists every code with a definition, rules for when the code applies and when the code does not, and examples from the data. Potato stamps each saved annotation with the codebook revision in effect, and the stamp shows which version of the codebook coded which excerpt. Developing a Codebook for Qualitative Research covers writing the entries, and Shared Codebook for Qualitative Coding documents the feature.

Content analysis

Content analysis codes units such as articles or speeches against a codebook of fixed variables and categories. At least two coders code a sample, and the study reports intercoder reliability for each variable. Quantitative Content Analysis in Potato runs the method end to end.

Thematic analysis

Thematic analysis is a family of methods for developing themes across a qualitative dataset, set out for psychology by Braun and Clarke (2006). The reflexive form of thematic analysis treats coding as interpretation and leaves out an agreement coefficient. When the method argues against a coefficient sets out both positions.

Intercoder reliability (ICR)

Intercoder reliability is the term qualitative research and content analysis use for agreement between coders who apply one codebook to the same excerpts. Studies usually report Krippendorff's alpha or Cohen's kappa on a double-coded sample of 10 to 25% of the data. Intercoder Reliability in Qualitative Research matches the coefficient to the coding design.

Memo

An analytic memo is a note a researcher writes during coding about the data and the codes, such as why a code fits a passage. Potato attaches memos to an item, to a text selection, or to a shape on an image, and keeps memos in the project database. Memos documents replies, issues, and the memo export.

Dataset documentation

Datasheet

A datasheet for a dataset answers a standard set of questions about the dataset's motivation, composition, collection process, and recommended uses, among other topics. Gebru et al. (2021) proposed datasheets by analogy with the datasheet that accompanies every electronic component. Documenting Datasets and Models compares datasheets with the other two standards.

Data statement

A data statement describes a language dataset's curation rationale, language variety, speaker and annotator demographics, and guidelines. Bender and Friedman (2018) proposed data statements for natural language processing. A data statement lets a reader judge which populations a dataset represents.

Model card

A model card reports a trained model's intended use, limitations, and performance broken down across demographic and other groups. Mitchell et al. (2019) proposed model cards. The annotator demographics in a datasheet or data statement are what make the group-wise results in a model card interpretable.

Where the terms appear in a Potato config

Many of the terms above map to one config key or one annotation type in Potato. The table lists the mapping for the terms with a direct setting.

Add these blocks to a working config to collect three labels per item, score annotators on gold items, and send low-agreement items to an adjudicator:

num_annotators_per_item: 3
 
gold_standards:
  enabled: true
  items_file: "gold_standards.json"
  frequency: 20          # about one gold item per 20 items
  accuracy:
    min_threshold: 0.8   # stricter than the 0.7 default
 
adjudication:
  enabled: true
  adjudicator_users: ["lead_annotator"]
  agreement_threshold: 0.75   # items below the threshold go to the adjudicator

Each record in the gold file needs an id and a gold_label, and Quality Control documents the full format.

Further reading

• Annotation learning path, for the guides in reading order by goal
• Annotation concepts FAQ, for answers to common questions about annotation
• What Is Data Annotation?, for the main task types
• Annotation Schemes reference, for every annotation type Potato supports

References

1. Bender, E. M., & Friedman, B. (2018). Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6. https://aclanthology.org/Q18-1041/
2. Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4), 324. https://doi.org/10.2307/2334029
3. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
4. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. https://doi.org/10.1177/001316446002000104
5. Dawid, A. P., & Skene, A. M. (1979). Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, 28(1), 20. https://doi.org/10.2307/2346806
6. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382. https://doi.org/10.1037/h0031619
7. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86-92. https://doi.org/10.1145/3458723
8. Hovy, D., Berg-Kirkpatrick, T., Vaswani, A., & Hovy, E. (2013). Learning whom to trust with MACE. Proceedings of NAACL-HLT 2013. https://aclanthology.org/N13-1132/
9. Kiritchenko, S., & Mohammad, S. (2017). Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. Proceedings of ACL 2017 (Volume 2: Short Papers). https://aclanthology.org/P17-2074/
10. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., et al. (2023). Let's verify step by step. arXiv:2305.20050. https://arxiv.org/abs/2305.20050
11. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency, 220-229. https://doi.org/10.1145/3287560.3287596
12. Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867-872. https://doi.org/10.1016/j.jesp.2009.03.009
13. Ramshaw, L., & Marcus, M. (1995). Text chunking using transformation-based learning. Third Workshop on Very Large Corpora. https://aclanthology.org/W95-0107/
14. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks Track. https://arxiv.org/abs/2306.05685