Annotation Showcase

Named entity recognition for adverse drug events from patient-reported experiences, based on the CADEC corpus (Karimi et al., 2015). Annotates drugs, adverse effects, symptoms, diseases, and findings from colloquial health forum posts with mapping to medical vocabularies (SNOMED-CT, MedDRA).

Identify implicit warrants in arguments. Based on Habernal et al., NAACL 2018 / SemEval 2018 Task 12. Given a claim and premise, choose the correct warrant that connects them.

Multi-label land cover classification from Sentinel-2 imagery (Sumbul et al., IGARSS 2019). Classify satellite patches into 43 land cover classes following the CORINE taxonomy.

Named entity recognition for biomedical text based on the JNLPBA shared task. Annotate entities including proteins, DNA, RNA, cell lines, and cell types following BioNLP community standards.

Extract chemical-disease relations from biomedical literature. Based on BioCreative V CDR task. Identify chemical and disease entities, then annotate causal relationships between them (chemical induces disease).

Multi-label classification of chest radiographs for 14 observations (Irvin et al., AAAI 2019). Annotate chest X-rays with pathology labels including uncertainty handling for clinical findings.

Annotate diverse perspectives on claims with stance and evidence. Based on Chen et al., NAACL 2019. Identify supporting and opposing perspectives for controversial claims.

Annotate commonsense inferences about events, mental states, and social interactions. Based on ATOMIC 2020 (Hwang et al., AAAI 2021). Generate if-then knowledge about causes, effects, intents, and reactions.

Annotate explanations for commonsense QA with positive and negative properties. Based on ECQA (Aggarwal et al., ACL 2021). Explain why an answer is correct and why others are wrong.

Recognize complex and emerging named entities. Based on SemEval 2022/2023 MultiCoNER. Identify creative works, products, groups, and other challenging entity types.

Annotate verbs for implied power and agency of participants. Based on Sap et al., EMNLP 2017. Capture how actions implicitly convey power dynamics and agency levels.

Link pronouns and noun phrases to the entities they refer to in text. Based on the OntoNotes coreference annotation guidelines and CoNLL shared tasks. Identify mention spans and cluster coreferent mentions together.

Distinguish between truthful and deceptive (fake) reviews. Based on Ott et al., ACL 2011. Identify fake reviews written to deceive vs genuine customer experiences.

Classify utterances in conversations by their communicative function (question, statement, request, etc.).

Extract relations between entities in dialogue. Based on Yu et al., ACL 2020. Identify 36 relation types between speakers and entities mentioned in conversations.

Document layout analysis benchmark (Li et al., COLING 2020). Detect and classify document elements including titles, abstracts, paragraphs, figures, tables, and captions.

Hate speech classification with fine-grained type labels based on the Dynamically Generated Hate Speech Dataset (Vidgen et al., ACL 2021). Classify content as hateful or not, then identify hate type (animosity, derogation, dehumanization, threatening, support for hateful entities) and target group.

Extract emotion causes from conversational text based on RECCON (Poria et al., EMNLP 2020). Identify which utterances and specific spans caused an emotion expressed in dialogue.

Document-level event argument extraction based on MAVEN-Arg (Wang et al., ACL 2024). Annotates event triggers with their argument roles including Agent, Patient, Location, Time, Instrument, and more. Supports both entity and non-entity arguments across document context.

Verify claims as supported, refuted, or not enough information based on provided evidence.

Span-level annotation of propaganda techniques in news articles based on SemEval 2020 Task 11 (Da San Martino et al., EMNLP 2019). Identifies 14 techniques including Loaded Language, Name Calling, Appeal to Fear, and more across rhetorical dimensions (ethos, logos, pathos).

Multi-label emotion classification with 27 emotion categories plus neutral, based on the Google Research GoEmotions dataset (Demszky et al., ACL 2020). Taxonomy covers 12 positive, 11 negative, and 4 ambiguous emotions designed for Reddit comment analysis.

Identify and categorize hate speech, offensive language, and toxic content in text.

Multi-task hate speech annotation with classification (hate/offensive/normal), target community identification, and rationale span highlighting. Based on the HateXplain benchmark (Mathew et al., AAAI 2021) - the first dataset covering classification, target identification, and rationale extraction.

Detect and categorize implicit hate speech using a six-category taxonomy. Based on ElSherief et al., EMNLP 2021. Identifies grievance, incitement, stereotypes, inferiority, irony, and threats.

Classify user utterances into intents for chatbot and virtual assistant training.

Large-scale chest radiograph classification based on MIMIC-CXR (Johnson et al., Scientific Data 2019). Multi-label classification with 14 observations derived from radiology reports.

Annotate moral reasoning in situated narratives. Based on Emelin et al., EMNLP 2021. Evaluate whether actions adhere to or diverge from social norms given situations and intentions.

Span-based entity labeling for identifying people, organizations, locations, and more.

Annotate emotions in news headlines with semantic roles. Based on Bostan et al., LREC 2020. Identify emotion, experiencer, cause, target, and textual cue.

Natural language inference with human explanations. Based on e-SNLI (Camburu et al., NeurIPS 2018). Classify entailment/contradiction/neutral and provide natural language justifications.

Multi-task annotation of political speeches covering sentiment, polarization, populism, topic identification, and named entities. Based on AgoraSpeech (Sermpezis et al., 2025), featuring human-validated labels for comprehensive political discourse analysis.

Annotate answer spans in text passages for reading comprehension tasks.

Annotate rationales (evidence spans) that justify classification decisions. Based on the ERASER benchmark (DeYoung et al., ACL 2020). Identify which parts of the text are necessary and sufficient for making a prediction.

Evaluate question-answer pairs for reading comprehension by verifying answers and rating quality.

Identify and classify relationships between entities in text (e.g., works-for, located-in, married-to).

Classify stance toward rumors in social media threads. Based on PHEME (Zubiaga et al.). Label replies as supporting, denying, querying, or commenting on rumorous claims.

5-class sentiment annotation for social media posts based on RuSentiment (Rogers et al., COLING 2018). Includes Positive, Negative, Neutral, Speech Act (greetings/thanks), and Skip categories. Achieved 0.654 Fleiss kappa with 250-350 posts/hour annotation speed.

Identify sarcastic statements and label their type and target in social media and conversational text.

Verify scientific claims against evidence from research abstracts. Based on SciFact (Wadden et al., EMNLP 2020). Classify claims as supported, refuted, or having insufficient evidence, and identify rationale sentences.

Rate the semantic similarity between pairs of sentences on a continuous scale.

Simple 3-way sentiment classification with radio buttons. Perfect for social media analysis, product reviews, and customer feedback.

Annotate social media posts for bias using structured frames. Based on Sap et al., ACL 2020. Identify offensiveness, intent, implied stereotypes, and targeted groups.

Event-based extraction of social determinants of health from clinical notes based on the n2c2 2022 Track 2 shared task and SHAC corpus. Annotates substance use (alcohol, drug, tobacco), employment, and living status with temporal and status attributes.

Detect the stance of a text toward a given topic. Based on VAST (Allaway & McKeown, EMNLP 2020) for zero-shot stance detection. Classify text as expressing favor, opposition, or neutrality toward various topics.

Dialogue act annotation following the Switchboard DAMSL tagset (Jurafsky et al., 1997). Covers statements, questions, backchannels, agreements, and more for conversational speech analysis. Achieved 0.80 inter-rater Kappa on 1,155 conversations.

Annotate temporal relations between events and time expressions following TimeML guidelines. Based on TempEval-3 shared task. Label relations as BEFORE, AFTER, SIMULTANEOUS, or VAGUE to capture how events relate in time.

Character-level toxic span annotation based on SemEval-2021 Task 5 (Pavlopoulos et al., 2021). Instead of binary toxicity classification, annotators identify the specific words/phrases that make a comment toxic, enabling more nuanced content moderation.