Zum Inhalt springen
Diese Seite ist in Ihrer Sprache noch nicht verfügbar. Englische Version wird angezeigt.

Annotation Learning Path

Six reading orders through the Potato guides, from a first annotation study to agreement reporting, LLM and agent evaluation, and qualitative coding.

This learning path puts the Potato annotation guides in reading order for six goals, from running a first annotation study to evaluating LLM agents and coding interviews. Each track has five to eight steps, and each step names what the guide teaches.

The sidebar groups the guides by topic. The tracks below order a subset of the guides by what a reader needs first, and a guide can appear on more than one track. A reader new to annotation starts with the first track, because the later tracks assume a working scheme and guidelines. The annotation glossary defines the terms the guides use, and the Guides overview lists every guide by topic.

Running a first annotation study

The first track takes a study from a research question to exported labels. The Quick Start installs Potato and runs a sentiment task, and a reader who wants a working server before the reading can do the Quick Start first.

  1. What Is Data Annotation? Each family of annotation task, from classification to free-text answers, maps to one or more Potato annotation schemes.
  2. Choosing an Annotation Scheme. The scheme follows from the kind of answer a study needs, and a decision table narrows Potato's annotation types to the few that fit.
  3. Designing Data Formats for Annotation. Each item needs a stable unique ID and the content to label, because Potato keys every annotation to the item ID.
  4. Writing Effective Annotation Guidelines. Guidelines work when two careful annotators reading the same rules choose the same label, and a pilot-and-revise loop gets the rules there.
  5. How Many Annotators Do You Need? Overlap trades the reliability of each label against the number of items a budget covers, and subjective tasks need more overlap than clear ones.
  6. Gold Standards and Attention Checks. Mixing items with known answers into the queue measures each annotator's accuracy and catches annotators who answer without reading.
  7. Deploying an Annotation Server. A study server needs a public HTTPS address, a sign-in that admits only the study's annotators, and storage that survives a restart. Running a Crowdsourced Study on Prolific and SONA covers paid recruitment.
  8. Exporting Annotations for Machine Learning. Potato writes JSONL and CSV as well as training formats such as CoNLL and COCO, and choosing the format before labeling settles how to structure the data and item IDs.

By the end of the track, a study config holds a scheme, a data file, an overlap setting, and attention checks. The config below marks the step that adds each part, and the config passes python -m potato.validate_cli --strict in Potato 2.13.1:

yaml
annotation_task_name: "Review Sentiment"
task_dir: "."
 
data_files:
  - "data/reviews.jsonl"            # step 3: one item per line
item_properties:
  id_key: id
  text_key: text
 
annotation_schemes:                 # step 2: one category per review
  - annotation_type: radio
    name: sentiment
    description: "What is the overall sentiment of this review?"
    labels: [Positive, Negative, Mixed, Neutral]
 
num_annotators_per_item: 3          # step 5: three labels per item
 
attention_checks:                   # step 6: one check per 25 items
  enabled: true
  items_file: "data/attention_checks.json"
  frequency: 25
 
output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"   # step 8

Each record in attention_checks.json needs an id and an expected_answer. Run potato start config.yaml from the folder that holds the config, and the server starts at http://localhost:8000.

Measuring and reporting label quality

The second track turns overlapping labels into numbers a paper or a dataset card can report. The track assumes a study with at least two annotators on some items.

  1. Inter-Annotator Agreement Explained. Percent agreement overstates reliability, and Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha correct for chance in different study designs.
  2. Agreement for Spans and Structured Outputs. Span tasks give kappa no fixed set of items to count, so span agreement is reported as pairwise F1 with a stated rule for partial overlaps. How to Measure Inter-Annotator Agreement on Bounding Boxes covers boxes.
  3. Adjudication and Resolving Disagreement. Disagreements resolve by majority vote, expert adjudication, or a model of annotator reliability, and the choice depends on whether the disagreement is error or signal.
  4. Aggregating Crowd Labels. Dawid–Skene and MACE estimate each annotator's reliability from the label matrix and weight each annotator's labels by that estimate.
  5. Statistical Power and Sample Size for Annotation Studies. Overlap sets how reliable each label is, and the number of items sets whether an observed difference is larger than noise.
  6. How to Tell Review from Rubber-Stamping in Pre-Labeled Data. With pre-labels, agreement rises whether annotators review or rubber-stamp, and the timing of the work separates the two.
  7. Documenting Datasets and Models. Datasheets, data statements, and model cards record who labeled the data and under which guidelines.

Labeling text for NLP

The third track covers text tasks from whole-document labels to linked entities. Steps 2 through 6 build on one another, and each later task works on spans of the kind the earlier steps mark.

  1. Text Annotation. Whole-document labels, from categories to quality ratings, are the most common annotation task in natural language processing.
  2. Span Annotation. A span marks a labeled stretch inside the text, recorded as a start, an end, and a label.
  3. Named Entity Recognition. NER is a span task with an entity-typed label set, trimmed from a standard scheme to the domain and exported as BIO-tagged CoNLL.
  4. Coreference Resolution. Coreference groups every mention of one entity into a chain, as in "Marie Curie … she … the physicist."
  5. Relation and Event Extraction. Relations link two entity spans with a typed edge, and events add a trigger word and the roles around the trigger.
  6. Entity Linking. Entity linking resolves each mention to one entry in a knowledge base, such as "Paris" to the city in France rather than the city in Texas.
  7. Multilingual and Low-Resource Annotation. Annotation in other languages needs native speakers, categories that hold across cultures, and correct display of right-to-left and non-Latin scripts.

Annotating images, video, and audio

The fourth track moves from still images to time-based media and ends with model pre-annotation. Box and mask agreement comes before video, because the boundary rules for a shape decide most of the disagreement on spatial labels.

  1. Image Annotation. Image labels range from one category per image to boxes, polygons, and landmarks, and the right level follows from what the model being trained needs.
  2. How to Annotate Image Segmentation Masks. Most disagreement on masks comes from the boundary rule, such as whether a shadow belongs to the object, rather than from the drawing tool.
  3. How to Measure Inter-Annotator Agreement on Bounding Boxes. Mean IoU reads high on large, centered objects even for annotators who never looked at the image, and box agreement needs a chance correction.
  4. Video Annotation. A clip can be labeled whole, split into time intervals, or annotated frame by frame.
  5. How to Annotate Video Object Tracking. Tracking labels one object across frames, and model-assisted propagation carries the first mask forward through the clip.
  6. Audio Annotation. One waveform display serves clip labels, time-aligned events, transcription, and speaker work.
  7. How to Annotate Whisper Transcripts. Whisper's .json output keeps the timed segments, and Potato reads a folder of the output files with no conversion step.
  8. LLM and Vision Pre-Annotation. A vision or language model proposes labels for annotators to verify, and checking a good proposal takes less time than labeling from scratch.

Step 8 uses a model trained elsewhere. Model-Assisted Labeling trains the model on the study's own labels instead, and the Model Training section documents trainers for boxes, outlines, audio segments, and transcripts.

Evaluating LLMs and agents

The fifth track starts with judging single outputs and ends with reviewing production traces. Steps 1 through 4 apply to any generated text, and steps 5 through 8 add the agent's path.

  1. Human Evaluation of Generated Text. Automatic metrics such as BLEU and ROUGE correlate weakly with quality, and a human evaluation earns trust by defining each criterion and reporting enough detail to rerun.
  2. Rubric-Based LLM Evaluation. A rubric splits one quality judgment into scored criteria and shows which criterion an answer failed.
  3. Pairwise Model Comparison. Head-to-head judgments over many prompts rank models more reliably than absolute scores, and a Bradley–Terry or Elo fit turns the wins into a ranking.
  4. How to Measure Agreement Between an LLM Judge and Human Annotators. A judge does about as well as another annotator when the judge's kappa with people comes close to the people's kappa with each other.
  5. How to Evaluate AI Agents. Agent evaluation judges the path as well as the final answer, from reasoning steps to tool calls and actions.
  6. Annotating Agent Trajectories. Trajectory labels combine a judgment of the whole run with an error category and a severity for each failed step.
  7. Process Reward Models and Step-Level Labeling. A process reward model scores each reasoning step, and training a PRM needs step-level labels that say whether each step is correct.
  8. Human Review Queues for LLM and Agent Traces. A review queue sends traces from Langfuse, a webhook, or an export to several reviewers and writes the merged answer back to the tracing tool as a score.

Qualitative coding

The sixth track follows interview data from recordings to a reliability statistic. QDA Mode documents the Potato features the track uses.

  1. Preparing Interview Transcripts for Qualitative Coding. Recordings become coded excerpts in five steps, from transcription with speaker labels to coding and export, and the team removes names and other identifiers before loading.
  2. Developing a Codebook for Qualitative Research. Each code gets a definition, rules for when the code applies and when the code does not, and examples. Each revision of the codebook gets a number.
  3. Intercoder Reliability in Qualitative Research. The coefficient follows from the coding design, and reflexive thematic analysis argues against reporting a coefficient at all.
  4. Quantitative Content Analysis in Potato. Content analysis codes units against fixed variables and reports Krippendorff's alpha for each variable on a double-coded sample.
  5. How to Use an LLM in Qualitative Coding and Report It. A model can suggest codes, code a reliability sample, or code the corpus first, and each role calls for reporting the model's agreement with the human coders.
  6. Qualitative Data Analysis Software Compared. NVivo, ATLAS.ti, MAXQDA, Taguette, QualCoder, and Potato differ in price and in support for team coding. Potato exchanges projects with the paid programs as REFI-QDA files.

Further reading