Skip to content

What Is Data Annotation?

An introduction to data annotation, the main task types (classification, span labeling, ranking, free text), and how to run an annotation project in Potato.

Data annotation is the process of attaching labels to raw data, text, images, audio, video, or model outputs, so that the data can be measured, compared, or used to train and evaluate machine learning models. A label might be a sentiment category on a tweet, a highlighted name in a sentence, a 1–5 quality rating on a chatbot reply, or a bounding box around a pedestrian in a photo.

Annotation is sometimes called data labeling, tagging, or coding (the term used in the social sciences). See Data annotation and the related idea of a labeled training set on Wikipedia.

What annotation is for

Supervised machine learning learns from examples that already carry the right answer. The quality of those answers sets a ceiling on model quality, so careful annotation is often where the biggest gains are. Annotation is also how you evaluate a model, because deciding whether a system's output is correct usually takes a person reading it.

The main types of annotation task

Most projects fall into a few families. Each maps to one or more annotation controls in Potato (see Annotation Schemes).

  • Classification: pick one or more categories for a whole item, such as marking a review positive, negative, or neutral. See Text classification.
  • Span labeling: mark a region inside an item, such as a name in a sentence or a region of an audio clip. See Span Annotation.
  • Rating and scoring: place an item on a scale, such as a 1–5 quality judgment. See Rating Scales.
  • Ranking and comparison: order items or pick the better of two. See Pairwise and Best–Worst Scaling.
  • Structured annotation: link spans into relations, build coreference chains, or annotate events. See Relation and Event Extraction.
  • Free-text: write an explanation, a correction, or a transcription.

A minimal example

A sentiment task in Potato is a few lines of YAML. The annotation_schemes block defines the labels an annotator sees:

yaml
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What is the overall sentiment of this review?"
    labels:
      - Positive
      - Negative
      - Neutral

Those eight lines are the whole interface definition. You supply the data, run potato start config.yaml, and annotators label in the browser.

How a project usually goes

  1. Define the task. Write down the question and the label set.
  2. Write guidelines. Give annotators rules and examples. See Writing Annotation Guidelines.
  3. Pilot. Label a small batch, find disagreements, refine the guidelines.
  4. Annotate with overlap. Have several people label the same items so you can measure agreement.
  5. Measure agreement. See Inter-Annotator Agreement.
  6. Adjudicate and export. Resolve disagreements and export for training or analysis.

Frequently asked questions

The questions below cover how data annotation relates to data labeling and what an annotator does.

What is the difference between data annotation and data labeling?

In most usage there is no difference. Data labeling, tagging, and coding, the term used in the social sciences, are other names for attaching labels to data so that it can be measured, compared, or used to train and evaluate machine learning models.

What does a data annotator do?

A data annotator reads each item, such as a tweet, a photo, or a chatbot reply, and attaches the label the task asks for according to written guidelines. In a research study several annotators label the same items independently so that agreement between them can be measured, and disagreements are resolved before the labels are exported for training or analysis.

Further reading