Annotation Showcase
Browse 459+ ready-to-use annotation configurations. Download configs and start annotating immediately.
Showing 60 of 459 designs
Text Annotation212
Adverse Drug Event Extraction (CADEC)
Named entity recognition for adverse drug events from patient-reported experiences, based on the CADEC corpus (Karimi et al., 2015). Annotates drugs, adverse effects, symptoms, diseases, and findings from colloquial health forum posts with mapping to medical vocabularies (SNOMED-CT, MedDRA).
AMI Meeting Multi-Tier Annotation
Multi-tier ELAN-style annotation of multi-party meeting recordings. Annotators segment speaker turns, head gestures, and focus of attention on parallel timeline tiers, then classify dialogue acts and topic segments. Based on the AMI Meeting Corpus.
Analysis of Clinical Text: Disorder Identification and Normalization
Identify disorder mentions and their attributes in clinical discharge summaries, based on SemEval-2015 Task 14 (Elhadad et al.). Annotators mark disorder spans, body locations, severity indicators, and classify the assertion status of each disorder.
Aspect-Based Sentiment Analysis
Identification of aspect terms in review text with sentiment polarity classification for each aspect. Based on SemEval-2016 Task 5 (ABSA).
Aspect-Based Sentiment Analysis (Original ABSA)
Identify aspect terms in review text and classify their sentiment polarity, based on SemEval-2014 Task 4 (Pontiki et al.). Annotators highlight aspect terms and assign sentiment labels across restaurant and laptop review domains.
Biomedical Entity Linking (MedMentions)
Entity mention detection and UMLS concept linking for biomedical text based on MedMentions. Annotators identify biomedical entity mentions in PubMed abstracts and link them to UMLS Concept Unique Identifiers (CUIs), supporting large-scale biomedical knowledge base construction and clinical NLP.
Biomedical Named Entity Recognition (JNLPBA)
Named entity recognition for biomedical text based on the JNLPBA shared task. Annotate the five GENIA bio-entity classes (protein, DNA, RNA, cell line, cell type) in molecular-biology MEDLINE abstracts, following BioNLP community standards.
BioNLP 2011 - Gene Regulation Event Extraction
Biomedical event extraction for gene regulation, based on the BioNLP 2011 Shared Task (Kim et al., ACL Workshop 2011). Annotators identify biological entities and mark regulatory events such as gene expression, transcription, and protein catabolism in scientific abstracts.
Audio Annotation33
DISPLACE 2024 - Speaker and Language Diarization
Speaker and language diarization in multilingual conversational audio. Annotators mark speaker turn boundaries, identify speakers, and label the language of each segment in conversational environments (Kalluri et al., INTERSPEECH 2024).
Sound Event Detection
Temporal sound event annotation with strong labels following DCASE Challenge protocols.
ToBI Prosodic Annotation
Multi-tier prosodic annotation following the Tones and Break Indices (ToBI) framework. Annotators label pitch accents, phrase accents, boundary tones, and break indices on speech utterances, producing a layered prosodic transcription aligned to the audio timeline (Silverman et al., ICSLP 1992).
Transcript Format Ingestion
Six transcript formats rendered as the same speaker bubbles, with per-turn labels and spans that work regardless of where the transcript came from.
Acoustic Scene Classification
Classify audio recordings by acoustic environment following the TUT/DCASE dataset format.
Audio Transcription Review
Review and correct automatic speech recognition transcriptions with waveform visualization.
Audio-Visual Sentiment Analysis
Rate sentiment in speech segments following CMU-MOSI and CMU-MOSEI multimodal annotation protocols.
AudioHate - Audio Hate Speech Detection
Audio hate speech detection with explanations. Annotators classify audio clips for hate speech presence, identify target groups, and note acoustic indicators such as tone, emphasis, and prosody (An et al., SIGDIAL 2024).
Image Annotation42
Breakfast Actions Segmentation
Fine-grained temporal action segmentation of breakfast preparation activities. Annotators label sequences of cooking actions like 'take cup', 'pour milk', 'stir'.
EPIC-KITCHENS Egocentric Action Annotation
Annotate fine-grained actions in egocentric kitchen videos with verb-noun pairs. Identify cooking actions from a first-person perspective.
FineGym Dataset: Fine-Grained Gymnastics Action Recognition
FineGym is a hierarchical video dataset (CVPR 2020) for fine-grained gymnastics action recognition: events, sets, and 99-530 element classes annotated with millisecond-level temporal boundaries. See the label taxonomy and a runnable Potato config to annotate or extend it.
Harmony4D Human Interaction Tracking
Close-range human interaction tracking and annotation. Annotators track multiple people during close physical interactions (dancing, martial arts, collaborative tasks) with bounding boxes and interaction labels.
How2Sign Sign Language Multi-Tier Annotation
Multi-tier ELAN-style annotation of continuous American Sign Language videos. Annotators segment sign glosses, mark mouthing patterns, classify sign handedness, and provide English translations aligned to video timelines. Based on the How2Sign large-scale multimodal ASL dataset.
MSAD Multi-Scenario Anomaly Detection
Video anomaly detection across multiple scenarios. Annotators watch surveillance-style videos and mark temporal segments containing anomalous events, classifying the anomaly type.
ADE20K: Scene Parsing and Semantic Segmentation Dataset
ADE20K is a scene parsing dataset from MIT CSAIL (Zhou et al., CVPR 2017) with dense pixel labels for objects, stuff, and parts. This Potato config adapts it to an image-level category-tagging task over a curated subset of common ADE20K categories.
BDD100K: Diverse Driving Dataset for Autonomous Driving
BDD100K is a 100,000-video driving dataset from UC Berkeley (Yu et al., CVPR 2020) spanning detection, segmentation, lane marking, and tracking. This Potato config reproduces its driving-scene labeling tasks.
Video Annotation34
ActivityNet Captions: Dense Video Captioning Dataset
ActivityNet Captions pairs 20k untrimmed videos with 100k temporally localized sentence descriptions for dense-captioning research. This Potato config reproduces the segment-and-describe workflow.
ActivityNet: Temporal Action Localization Benchmark
ActivityNet is a large-scale video benchmark for human activity understanding, with untrimmed YouTube videos labeled across 200 daily activities. This Potato config reproduces its temporal localization annotation: marking start and end times of activity instances.
AVA: Atomic Visual Actions Dataset
AVA spatio-temporally localizes 80 atomic actions on people in movie clips, labeled at 1 Hz keyframes. This Potato config reproduces the box-and-action labeling task for video clips.
Charades Indoor Activity Segmentation
Multi-label temporal activity segmentation in indoor home videos. Annotators identify action instances using compositional verb-object labels (e.g., 'opening door', 'sitting on chair') with precise temporal boundaries.
Charades-STA Temporal Grounding
Ground natural language descriptions to video segments. Given a sentence describing an action, identify the exact temporal boundaries where that action occurs.
Clinical TempEval - Temporal Information Extraction from Clinical Notes
Extraction of temporal information from clinical text, identifying time expressions, event mentions, and their temporal relations. Based on SemEval-2016 Task 12 (Clinical TempEval).
DAVIS - Video Object Segmentation with Mask Propagation
Semi-supervised video object segmentation on DAVIS (Pont-Tuset et al., 2017): mask each object on the first frame, then propagate it through the clip with SAM 2 and correct the frames where it drifts.
DiDeMo Moment Retrieval
Localizing natural language descriptions to specific video moments. Given a text query, annotators identify the corresponding temporal segment in the video.
Comparison Tasks4
Best-Worst Scaling
MaxDiff annotation where annotators select the best and worst items from a set for relative comparison.
Ranking Task
Drag-and-drop ranking interface to order items from best to worst.
SWE-Bench+ Patch Screening
Screen and compare model-generated patches against gold patches for SWE-Bench+ instances. Annotators evaluate correctness, identify specific issues, and compare model vs. gold solutions side-by-side.
WMT15 Relative Ranking of Machine Translations
The classic WMT relative-ranking (RR) protocol for machine translation evaluation, from 'Findings of the 2015 Workshop on Statistical Machine Translation' by Bojar et al. (WMT 2015). The annotator sees a source sentence, a human reference translation, and the outputs of up to five anonymized MT systems in random order, and ranks the outputs from best to worst with ties allowed; the joint rankings are reduced to pairwise judgments and aggregated with TrueSkill into the official system ranking.
Preference Learning32
Interpretable Semantic Textual Similarity
Fine-grained semantic similarity assessment between sentence pairs with span alignment, combining chunk-level annotation with graded similarity scoring. Based on SemEval-2016 Task 2.
SaGA Gesture-Speech Alignment Multi-Tier Annotation
Multi-tier ELAN-style annotation of co-speech gestures and their alignment with spoken language. Annotators segment gesture phases and types on parallel timeline tiers, classify handedness and spatial reference frames, and transcribe concurrent speech. Based on the SaGA corpus.
AlpacaEval: Instruction-Following Preference Evaluation
Pairwise preference annotation for instruction-following language models. Annotators compare two model responses side by side, select their preferred response, indicate preference strength, and rate individual response quality across diverse instruction categories.
AlpacaFarm Preference Simulation
Simulate human preferences for instruction-following responses. Create preference data for efficient RLHF research and LLM evaluation.
Arena Hard Auto - LLM Pairwise Evaluation
Pairwise evaluation of LLM responses on challenging prompts from the Arena Hard benchmark (Li et al., ICML 2025). Annotators compare two responses on a continuous scale and rate question difficulty.
Aya Red-Teaming - Multilingual Global and Local Harm Annotation
Multilingual safety red-teaming annotation, following the Aya Red-Teaming dataset from 'The Multilingual Alignment Prism' (Aakanksha et al., EMNLP 2024): the first human-annotated collection of harmful prompts across eight languages (English, Hindi, French, Spanish, Russian, Arabic, Serbian, Filipino). Native-speaker annotators judge whether a prompt is harmful, assign harm categories, and - the paper's key contribution - mark whether the harm is GLOBAL (universally recognized) or LOCAL (specific to a language or culture). Sample items are mild, constructed, non-operational illustrations only; they contain no actionable or graphic harmful content.
BeaverTails Safety Preference
Annotate AI responses for safety across multiple harm categories. Identify unsafe content and rate response quality for building safer AI systems.
Chatbot Arena: Pairwise LLM Preference Evaluation
Chatbot Arena collects human pairwise preference votes between anonymous LLM responses to rank models with a Bradley-Terry leaderboard. This Potato config reproduces the pairwise and best-worst scaling judgment task.
Surveys63
DimABSA: Dimensional Aspect-Based Sentiment Analysis
SemEval-2026 Task 3 (DimABSA): aspect-based sentiment analysis where sentiment toward each aspect is modeled along the continuous valence-arousal (VA) dimensions instead of categorical polarity. Annotators mark aspect terms and their opinion terms and rate the sentiment on valence and arousal scales. The task spans six languages and four domains, and includes a Dimensional Stance (DimStance) track that treats stance targets as aspects.
ESA: Error Span Annotation for Machine Translation
Error span annotation for machine translation output. Annotators identify error spans in translations, classify error types (accuracy, fluency, terminology, style), and rate severity.
LongEval: Faithfulness Evaluation for Long-Form Summarization
LongEval is the EACL 2023 protocol for human evaluation of faithfulness in long-form summaries (Krishna et al.). This Potato config reproduces its fine-grained, clause-level faithfulness judgments against source documents.
News Headline Emotion Roles (GoodNewsEveryone)
Annotate emotions in news headlines with semantic roles. Based on Bostan et al., LREC 2020. Identify emotion, experiencer, cause, target, and textual cue.
NLI with Explanations (e-SNLI)
Natural language inference with human explanations. Based on e-SNLI (Camburu et al., NeurIPS 2018). Classify entailment/contradiction/neutral and provide natural language justifications.
RT-2 - Robotic Action Annotation
Robotic manipulation task evaluation and action segmentation based on RT-2 (Brohan et al., CoRL 2023). Annotators evaluate task success, describe actions, rate execution quality, and segment video into action phases.
SciFact: Scientific Claim Verification Dataset
SciFact pairs 1,409 expert-written scientific claims with 5,183 research abstracts labeled SUPPORTS, REFUTES, or NOINFO plus rationale sentences. The Potato config below reproduces the annotation task.
AgentBoard Progress Scoring
Assess multi-turn LLM agent progress by identifying achieved milestones, scoring overall progress, categorizing the agent environment, and noting partial progress observations.
Evaluation Tasks39
Code Review Annotation (CodeReviewer)
Annotation of code review activities based on the CodeReviewer benchmark. Annotators identify issues in code diffs, classify defect types, assign severity levels, make review decisions, and provide natural language review comments, supporting research in automated code review and software engineering.
EA-MT - Entity-Aware Machine Translation
Entity-aware machine translation evaluation requiring annotators to identify entity spans, classify translation errors, and provide corrected translations. Based on SemEval-2025 Task 2.
FAVA: Fine-grained Hallucination Detection and Editing for Language Models
Fine-grained hallucination span annotation. Annotators identify hallucinated spans in LLM output and classify hallucination types (entity error, relation error, contradicted, invented, subjective, unverifiable). Based on the FAVA framework for fine-grained faithfulness evaluation.
MathDial: Math Tutoring Dialogue Dataset
MathDial is a dataset of 2,861 one-to-one math tutoring dialogues grounded in GSM8K word problems, annotated with a four-category teacher-move taxonomy (Focus, Probing, Telling, Generic). This Potato config reproduces that teacher-move labeling task.
#HashtagWars - Learning a Sense of Humor
Humor ranking of tweets submitted to Comedy Central's @midnight #HashtagWars, classifying comedic quality. Based on SemEval-2017 Task 6.
AgentRewardBench Trajectory Scoring
Evaluate web agent trajectories by rating step-level quality across multiple dimensions, judging overall success, and identifying where automatic evaluators disagree with human judgment.
AITW Mobile Trajectory Review
Step-by-step review of Android device-control episodes in the style of Android in the Wild (AITW; Rawles et al., NeurIPS 2023). Each step shows the screen and the action taken (tap, scroll, type, system button, or status action), with tap coordinates rendered as a grounding marker; annotators judge each action's correctness and grounding against the natural-language instruction, then give an episode-level verdict. Complements agentic/androidworld-mobile-agent-eval: AndroidWorld is a live-environment benchmark, while AITW is a static corpus of human demonstrations.
AndroidWorld: Mobile Agent Task Evaluation
Evaluation of autonomous agents performing tasks in dynamic Android environments. Annotators assess task completion, identify interaction types and touch gestures, evaluate UI understanding, and describe mobile-specific issues across diverse Android apps.
Have a design to share?
Contribute your annotation configurations to help the community.