Skip to content

Annotation Showcase

Browse 505+ ready-to-use annotation configurations. Download configs and start annotating immediately.

View on GitHub →

Showing 69 of 505 designs

Text Annotation219

Adverse Drug Event Extraction (CADEC)

Named entity recognition for adverse drug events from patient-reported experiences, based on the CADEC corpus (Karimi et al., 2015). Annotates drugs, adverse effects, symptoms, diseases, and findings from colloquial health forum posts with mapping to medical vocabularies (SNOMED-CT, MedDRA).

Annotation types
Text spans,Single choice
Domain and complexity
Biomedical NLP · Intermediate

AMI Meeting Multi-Tier Annotation

Multi-tier ELAN-style annotation of multi-party meeting recordings. Annotators segment speaker turns, head gestures, and focus of attention on parallel timeline tiers, then classify dialogue acts and topic segments. Based on the AMI Meeting Corpus.

Annotation types
Video annotation,Single choice
Domain and complexity
Discourse analysis · Advanced

Analysis of Clinical Text: Disorder Identification and Normalization

Identify disorder mentions and their attributes in clinical discharge summaries, based on SemEval-2015 Task 14 (Elhadad et al.). Annotators mark disorder spans, body locations, severity indicators, and classify the assertion status of each disorder.

Annotation types
Text spans,Single choice
Domain and complexity
Clinical NLP · Advanced

Aspect-Based Sentiment Analysis

Identification of aspect terms in review text with sentiment polarity classification for each aspect. Based on SemEval-2016 Task 5 (ABSA).

Annotation types
Text spans,Single choice
Domain and complexity
Sentiment analysis · Intermediate

Aspect-Based Sentiment Analysis (Original ABSA)

Identify aspect terms in review text and classify their sentiment polarity, based on SemEval-2014 Task 4 (Pontiki et al.). Annotators highlight aspect terms and assign sentiment labels across restaurant and laptop review domains.

Annotation types
Text spans,Single choice
Domain and complexity
Sentiment analysis · Intermediate

Biomedical Entity Linking (MedMentions)

Entity mention detection and UMLS concept linking for biomedical text based on MedMentions. Annotators identify biomedical entity mentions in PubMed abstracts and link them to UMLS Concept Unique Identifiers (CUIs), supporting large-scale biomedical knowledge base construction and clinical NLP.

Annotation types
Single choice,Text spans,Free text
Domain and complexity
Biomedical NLP · Advanced

Biomedical Named Entity Recognition (JNLPBA)

Named entity recognition for biomedical text based on the JNLPBA shared task. Annotate the five GENIA bio-entity classes (protein, DNA, RNA, cell line, cell type) in molecular-biology MEDLINE abstracts, following BioNLP community standards.

Annotation types
Text spans
Domain and complexity
Biomedical · Advanced

BioNLP 2011 - Gene Regulation Event Extraction

Biomedical event extraction for gene regulation, based on the BioNLP 2011 Shared Task (Kim et al., ACL Workshop 2011). Annotators identify biological entities and mark regulatory events such as gene expression, transcription, and protein catabolism in scientific abstracts.

Annotation types
Events,Text spans
Domain and complexity
Biomedical · Advanced

Image Annotation45

Breakfast Actions Segmentation

Fine-grained temporal action segmentation of breakfast preparation activities. Annotators label sequences of cooking actions like 'take cup', 'pour milk', 'stir'.

Annotation types
Free text,Video annotation
Domain and complexity
Computer vision · Advanced

EPIC-KITCHENS Egocentric Action Annotation

Annotate fine-grained actions in egocentric kitchen videos with verb-noun pairs. Identify cooking actions from a first-person perspective.

Annotation types
Single choice,Free text,Video annotation
Domain and complexity
Computer vision · Advanced

FineGym Dataset: Fine-Grained Gymnastics Action Recognition

FineGym is a hierarchical video dataset (CVPR 2020) for fine-grained gymnastics action recognition: events, sets, and 99-530 element classes annotated with millisecond-level temporal boundaries. See the label taxonomy and a runnable Potato config to annotate or extend it.

Annotation types
Single choice,Video annotation
Domain and complexity
Computer vision · Advanced

Harmony4D Human Interaction Tracking

Close-range human interaction tracking and annotation. Annotators track multiple people during close physical interactions (dancing, martial arts, collaborative tasks) with bounding boxes and interaction labels.

Annotation types
Multiple choice,Video annotation
Domain and complexity
Computer vision · Advanced

How2Sign Sign Language Multi-Tier Annotation

Multi-tier ELAN-style annotation of continuous American Sign Language videos. Annotators segment sign glosses, mark mouthing patterns, classify sign handedness, and provide English translations aligned to video timelines. Based on the How2Sign large-scale multimodal ASL dataset.

Annotation types
Video annotation,Single choice,Free text
Domain and complexity
Sign language · Advanced

MSAD Multi-Scenario Anomaly Detection

Video anomaly detection across multiple scenarios. Annotators watch surveillance-style videos and mark temporal segments containing anomalous events, classifying the anomaly type.

Annotation types
Video annotation,Single choice
Domain and complexity
Computer vision · Intermediate

ADE20K: Scene Parsing and Semantic Segmentation Dataset

ADE20K is a scene parsing dataset from MIT CSAIL (Zhou et al., CVPR 2017) with dense pixel labels for objects, stuff, and parts. This Potato config adapts it to an image-level category-tagging task over a curated subset of common ADE20K categories.

Annotation types
Multiple choice,Single choice
Domain and complexity
Computer vision · Advanced

BDD100K: Diverse Driving Dataset for Autonomous Driving

BDD100K is a 100,000-video driving dataset from UC Berkeley (Yu et al., CVPR 2020) spanning detection, segmentation, lane marking, and tracking. This Potato config reproduces its driving-scene labeling tasks.

Annotation types
Multiple choice,Single choice
Domain and complexity
Computer vision · Advanced

Video Annotation36

ActivityNet Captions: Dense Video Captioning Dataset

ActivityNet Captions pairs 20k untrimmed videos with 100k temporally localized sentence descriptions for dense-captioning research. This Potato config reproduces the segment-and-describe workflow.

Annotation types
Video annotation,Free text
Domain and complexity
Computer vision · Advanced

ActivityNet: Temporal Action Localization Benchmark

ActivityNet is a large-scale video benchmark for human activity understanding, with untrimmed YouTube videos labeled across 200 daily activities. This Potato config reproduces its temporal localization annotation: marking start and end times of activity instances.

Annotation types
Video annotation
Domain and complexity
Computer vision · Intermediate

AVA: Atomic Visual Actions Dataset

AVA spatio-temporally localizes 80 atomic actions on people in movie clips, labeled at 1 Hz keyframes. This Potato config reproduces the box-and-action labeling task for video clips.

Annotation types
Multiple choice,Video annotation
Domain and complexity
Computer vision · Advanced

Charades Indoor Activity Segmentation

Multi-label temporal activity segmentation in indoor home videos. Annotators identify action instances using compositional verb-object labels (e.g., 'opening door', 'sitting on chair') with precise temporal boundaries.

Annotation types
Video annotation
Domain and complexity
Computer vision · Intermediate

Charades-STA Temporal Grounding

Ground natural language descriptions to video segments. Given a sentence describing an action, identify the exact temporal boundaries where that action occurs.

Annotation types
Single choice,Video annotation
Domain and complexity
Computer vision · Intermediate

Clinical TempEval - Temporal Information Extraction from Clinical Notes

Extraction of temporal information from clinical text, identifying time expressions, event mentions, and their temporal relations. Based on SemEval-2016 Task 12 (Clinical TempEval).

Annotation types
Text spans,Single choice
Domain and complexity
Clinical NLP · Advanced

DAVIS - Video Object Segmentation with Mask Propagation

Semi-supervised video object segmentation on DAVIS (Pont-Tuset et al., 2017): mask each object on the first frame, then propagate it through the clip with SAM 2 and correct the frames where it drifts.

Annotation types
Video annotation,Multiple choice,Free text
Domain and complexity
Computer vision · Advanced

DiDeMo Moment Retrieval

Localizing natural language descriptions to specific video moments. Given a text query, annotators identify the corresponding temporal segment in the video.

Annotation types
Single choice,Video annotation
Domain and complexity
Computer vision · Intermediate

Audio Annotation38

DISPLACE 2024 - Speaker and Language Diarization

Speaker and language diarization in multilingual conversational audio. Annotators mark speaker turn boundaries, identify speakers, and label the language of each segment in conversational environments (Kalluri et al., INTERSPEECH 2024).

Annotation types
Single choice,Text spans
Domain and complexity
Speech · Advanced

Sound Event Detection

Temporal sound event annotation with strong labels following DCASE Challenge protocols.

Annotation types
Text spans,Multiple choice
Domain and complexity
Audio · Advanced

ToBI Prosodic Annotation

Multi-tier prosodic annotation following the Tones and Break Indices (ToBI) framework. Annotators label pitch accents, phrase accents, boundary tones, and break indices on speech utterances, producing a layered prosodic transcription aligned to the audio timeline (Silverman et al., ICSLP 1992).

Annotation types
Text spans,Single choice,Free text
Domain and complexity
Prosody · Advanced

Transcript Format Ingestion

Six transcript formats rendered as the same speaker bubbles, with per-turn labels and spans that work regardless of where the transcript came from.

Annotation types
Single choice,Text spans
Domain and complexity
Audio · Intermediate

Acoustic Scene Classification

Classify audio recordings by acoustic environment following the TUT/DCASE dataset format.

Annotation types
Single choice,Likert scale
Domain and complexity
Audio · Beginner

AnuraSet Frog Call Segments and Calling Intensity

Check species-labeled call segments and rate each species' calling activity from 0 to 3 on ten real frog recordings, with hf-audio-tagger training.

Annotation types
Audio annotation,Single choice
Domain and complexity
Bioacoustics · Intermediate

Audio Transcription Review

Review and correct automatic speech recognition transcriptions with waveform visualization.

Annotation types
Likert scale,Multiple choice,Single choice,1 more
Domain and complexity
Audio · Intermediate

Audio-Visual Sentiment Analysis

Rate sentiment in speech segments following CMU-MOSI and CMU-MOSEI multimodal annotation protocols.

Annotation types
Likert scale,Single choice
Domain and complexity
Audio · Intermediate

Scientific Data18

BBBC038 2018 Data Science Bowl Outlines of Every Nucleus

Draw one polygon per nucleus on ten real images from the CC0 2018 Data Science Bowl set, with Cellpose fine-tuning that writes outlines back as pre-labels.

Annotation types
Image annotation
Domain and complexity
Microscopy · Intermediate

BioLiP2 Ligand Relevance and Binding Residues on PDB Structures

View a ligand-bound PDB structure in 3D, mark the residues that touch a named ligand, and judge whether the ligand is biologically relevant.

Annotation types
Protein and genome selections,Single choice
Domain and complexity
Structural biology · Intermediate

BRACS Breast Lesion Regions on a Gigapixel Whole-Slide Image

Outline and label breast lesion regions with seven BRACS categories in the deep-zoom viewer, then train a tiled Mask R-CNN. The sample slide is illustrative.

Annotation types
Image annotation,Single choice
Domain and complexity
Pathology · Advanced

CHB-MIT Seizure Onset and End Marking in Scalp EEG

Mark the onset and end of seizures in ten real multichannel scalp EEG excerpts from one pediatric case, using the time_series display and signal-cnn.

Annotation types
Signal intervals
Domain and complexity
Neurology · Intermediate

ClinVar Variant Classification with ACMG/AMP Terms

Classify variants in igv.js with five ACMG/AMP terms and tick evidence criteria. The ten cases are illustrative, beside a live ClinVar track, and not a diagnosis.

Annotation types
Single choice,Multiple choice,Protein and genome selections
Domain and complexity
Genomics · Intermediate

Galaxy Zoo 2 Galaxy Morphology Decision Tree

Classify ten real Galaxy Zoo 2 galaxy images with an 11-question decision tree chained with display_logic, with a trainer pre-labeling the first question.

Annotation types
Single choice
Domain and complexity
Astronomy · Intermediate

Gravity Spy LIGO Glitch Classification

Classify eight real Gravity Spy spectrograms of LIGO detector glitches into 22 classes with a radio scheme, with a trainer that pre-labels new glitches.

Annotation types
Single choice
Domain and complexity
Astronomy · Intermediate

Human Protein Atlas Subcellular Localization Across 28 Labels

Select every compartment a protein localizes to on four-channel microscopy images, using 28 HPA labels. The ten sample images are synthetic and illustrative.

Annotation types
Microscopy,Multiple choice
Domain and complexity
Microscopy · Intermediate

Evaluation Tasks42

Code Review Annotation (CodeReviewer)

Annotation of code review activities based on the CodeReviewer benchmark. Annotators identify issues in code diffs, classify defect types, assign severity levels, make review decisions, and provide natural language review comments, supporting research in automated code review and software engineering.

Annotation types
Text spans,Single choice,Free text
Domain and complexity
Software engineering · Advanced

EA-MT - Entity-Aware Machine Translation

Entity-aware machine translation evaluation requiring annotators to identify entity spans, classify translation errors, and provide corrected translations. Based on SemEval-2025 Task 2.

Annotation types
Text spans,Single choice,Free text
Domain and complexity
Machine translation · Advanced

FAVA: Fine-grained Hallucination Detection and Editing for Language Models

Fine-grained hallucination span annotation. Annotators identify hallucinated spans in LLM output and classify hallucination types (entity error, relation error, contradicted, invented, subjective, unverifiable). Based on the FAVA framework for fine-grained faithfulness evaluation.

Annotation types
Text spans,Single choice
Domain and complexity
Hallucination detection · Advanced

MathDial: Math Tutoring Dialogue Dataset

MathDial is a dataset of 2,861 one-to-one math tutoring dialogues grounded in GSM8K word problems, annotated with a four-category teacher-move taxonomy (Focus, Probing, Telling, Generic). This Potato config reproduces that teacher-move labeling task.

Annotation types
Single choice,Multiple choice,Text spans
Domain and complexity
Education · Intermediate

#HashtagWars - Learning a Sense of Humor

Humor ranking of tweets submitted to Comedy Central's @midnight #HashtagWars, classifying comedic quality. Based on SemEval-2017 Task 6.

Annotation types
Single choice
Domain and complexity
Humor detection · Beginner

AgentRewardBench Trajectory Scoring

Evaluate web agent trajectories by rating step-level quality across multiple dimensions, judging overall success, and identifying where automatic evaluators disagree with human judgment.

Annotation types
Rating grid,Single choice,Free text
Domain and complexity
Agentic AI · Advanced

AITW Mobile Trajectory Review

Step-by-step review of Android device-control episodes in the style of Android in the Wild (AITW; Rawles et al., NeurIPS 2023). Each step shows the screen and the action taken (tap, scroll, type, system button, or status action), with tap coordinates rendered as a grounding marker; annotators judge each action's correctness and grounding against the natural-language instruction, then give an episode-level verdict. Complements agentic/androidworld-mobile-agent-eval: AndroidWorld is a live-environment benchmark, while AITW is a static corpus of human demonstrations.

Annotation types
GUI agent steps,Single choice
Domain and complexity
Mobile UI · Advanced

AndroidWorld: Mobile Agent Task Evaluation

Evaluation of autonomous agents performing tasks in dynamic Android environments. Annotators assess task completion, identify interaction types and touch gestures, evaluate UI understanding, and describe mobile-specific issues across diverse Android apps.

Annotation types
Single choice,Multiple choice,Free text
Domain and complexity
Mobile agents · Advanced

Preference Learning33

Interpretable Semantic Textual Similarity

Fine-grained semantic similarity assessment between sentence pairs with span alignment, combining chunk-level annotation with graded similarity scoring. Based on SemEval-2016 Task 2.

Annotation types
Text spans,Likert scale
Domain and complexity
Semantic similarity · Advanced

SaGA Gesture-Speech Alignment Multi-Tier Annotation

Multi-tier ELAN-style annotation of co-speech gestures and their alignment with spoken language. Annotators segment gesture phases and types on parallel timeline tiers, classify handedness and spatial reference frames, and transcribe concurrent speech. Based on the SaGA corpus.

Annotation types
Video annotation,Single choice,Free text
Domain and complexity
Gesture studies · Advanced

AlpacaEval: Instruction-Following Preference Evaluation

Pairwise preference annotation for instruction-following language models. Annotators compare two model responses side by side, select their preferred response, indicate preference strength, and rate individual response quality across diverse instruction categories.

Annotation types
Pairwise comparison,Single choice,Free text
Domain and complexity
AI evaluation · Intermediate

AlpacaFarm Preference Simulation

Simulate human preferences for instruction-following responses. Create preference data for efficient RLHF research and LLM evaluation.

Annotation types
Likert scale,Single choice
Domain and complexity
AI alignment · Intermediate

Arena Hard Auto - LLM Pairwise Evaluation

Pairwise evaluation of LLM responses on challenging prompts from the Arena Hard benchmark (Li et al., ICML 2025). Annotators compare two responses on a continuous scale and rate question difficulty.

Annotation types
Pairwise comparison,Likert scale
Domain and complexity
AI evaluation · Intermediate

Aya Red-Teaming - Multilingual Global and Local Harm Annotation

Multilingual safety red-teaming annotation, following the Aya Red-Teaming dataset from 'The Multilingual Alignment Prism' (Aakanksha et al., EMNLP 2024): the first human-annotated collection of harmful prompts across eight languages (English, Hindi, French, Spanish, Russian, Arabic, Serbian, Filipino). Native-speaker annotators judge whether a prompt is harmful, assign harm categories, and - the paper's key contribution - mark whether the harm is GLOBAL (universally recognized) or LOCAL (specific to a language or culture). Sample items are mild, constructed, non-operational illustrations only; they contain no actionable or graphic harmful content.

Annotation types
Single choice,Multiple choice
Domain and complexity
AI safety · Intermediate

BeaverTails Safety Preference

Annotate AI responses for safety across multiple harm categories. Identify unsafe content and rate response quality for building safer AI systems.

Annotation types
Multiple choice,Single choice
Domain and complexity
AI safety · Advanced

Chatbot Arena: Pairwise LLM Preference Evaluation

Chatbot Arena collects human pairwise preference votes between anonymous LLM responses to rank models with a Bradley-Terry leaderboard. This Potato config reproduces the pairwise and best-worst scaling judgment task.

Annotation types
Best-worst scaling,Pairwise comparison
Domain and complexity
AI evaluation · Intermediate

Comparison Tasks5

Senate Oversight of A.I. Hearings: Coding Senators' Remarks with Cases and a Code Hierarchy

Qualitative coding of senators' remarks from the four Senate Judiciary subcommittee hearings titled "Oversight of A.I." (2023-2024), in Potato's QDA Mode. The design demonstrates cases with attributes (one case per senator, with party and state), the code-by-case crosstab on the admin API, and a codebook of parent and child codes. The ten sample passages are real public-domain remarks taken from the govinfo transcripts.

Annotation types
Text spans
Domain and complexity
Qualitative research · Intermediate

Best-Worst Scaling

MaxDiff annotation where annotators select the best and worst items from a set for relative comparison.

Annotation types
Best-worst scaling
Domain and complexity
NLP · Beginner

Ranking Task

Drag-and-drop ranking interface to order items from best to worst.

Annotation types
Ranking
Domain and complexity
NLP · Beginner

SWE-Bench+ Patch Screening

Screen and compare model-generated patches against gold patches for SWE-Bench+ instances. Annotators evaluate correctness, identify specific issues, and compare model vs. gold solutions side-by-side.

Annotation types
Single choice,Multiple choice,Pairwise comparison,1 more
Domain and complexity
Software engineering · Advanced

WMT15 Relative Ranking of Machine Translations

The classic WMT relative-ranking (RR) protocol for machine translation evaluation, from 'Findings of the 2015 Workshop on Statistical Machine Translation' by Bojar et al. (WMT 2015). The annotator sees a source sentence, a human reference translation, and the outputs of up to five anonymized MT systems in random order, and ranks the outputs from best to worst with ties allowed; the joint rankings are reduced to pairwise judgments and aggregated with TrueSkill into the official system ranking.

Annotation types
Ranking
Domain and complexity
Machine translation · Intermediate

Surveys69

DimABSA: Dimensional Aspect-Based Sentiment Analysis

SemEval-2026 Task 3 (DimABSA): aspect-based sentiment analysis where sentiment toward each aspect is modeled along the continuous valence-arousal (VA) dimensions instead of categorical polarity. Annotators mark aspect terms and their opinion terms and rate the sentiment on valence and arousal scales. The task spans six languages and four domains, and includes a Dimensional Stance (DimStance) track that treats stance targets as aspects.

Annotation types
Text spans,Slider,Free text
Domain and complexity
Sentiment analysis · Advanced

ESA: Error Span Annotation for Machine Translation

Error span annotation for machine translation output. Annotators identify error spans in translations, classify error types (accuracy, fluency, terminology, style), and rate severity.

Annotation types
Text spans,Single choice,Likert scale
Domain and complexity
Machine translation · Advanced

LongEval: Faithfulness Evaluation for Long-Form Summarization

LongEval is the EACL 2023 protocol for human evaluation of faithfulness in long-form summaries (Krishna et al.). This Potato config reproduces its fine-grained, clause-level faithfulness judgments against source documents.

Annotation types
Text spans,Single choice,Likert scale
Domain and complexity
Summarization · Advanced

News Headline Emotion Roles (GoodNewsEveryone)

Annotate emotions in news headlines with semantic roles. Based on Bostan et al., LREC 2020. Identify emotion, experiencer, cause, target, and textual cue.

Annotation types
Likert scale,Single choice,Text spans
Domain and complexity
Affective computing · Advanced

NLI with Explanations (e-SNLI)

Natural language inference with human explanations. Based on e-SNLI (Camburu et al., NeurIPS 2018). Classify entailment/contradiction/neutral and provide natural language justifications.

Annotation types
Likert scale,Single choice,Text spans
Domain and complexity
Explainability · Intermediate

RT-2 - Robotic Action Annotation

Robotic manipulation task evaluation and action segmentation based on RT-2 (Brohan et al., CoRL 2023). Annotators evaluate task success, describe actions, rate execution quality, and segment video into action phases.

Annotation types
Single choice,Free text,Likert scale,1 more
Domain and complexity
Robotics · Advanced

SciFact: Scientific Claim Verification Dataset

SciFact pairs 1,409 expert-written scientific claims with 5,183 research abstracts labeled SUPPORTS, REFUTES, or NOINFO plus rationale sentences. The Potato config below reproduces the annotation task.

Annotation types
Likert scale,Single choice,Text spans
Domain and complexity
Scientific NLP · Advanced

AgentBoard Progress Scoring

Assess multi-turn LLM agent progress by identifying achieved milestones, scoring overall progress, categorizing the agent environment, and noting partial progress observations.

Annotation types
Multiple choice,Likert scale,Single choice,1 more
Domain and complexity
Agentic AI · Advanced

Have a design to share?

Contribute your annotation configurations to help the community.