Causal Medical Claim Identification and PIO Frame Extraction
Identify causal medical claim spans in Reddit health posts (Subtask 1: Claim, Experience, Experience-based claim, Question) and extract PIO frames — Population, Intervention, Outcome (Subtask 2). Based on SemEval-2023 Task 8 (Khetan et al.).
Configuration Fileconfig.yaml
This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.
# Causal Medical Claim Identification and PIO Frame Extraction
# Based on Khetan et al., SemEval 2023 Task 8
# Paper: https://aclanthology.org/2023.semeval-1.311/
# Task page: https://causalclaims.github.io/
#
# The task uses Reddit health/social-media posts and has two subtasks:
#
# Subtask 1 (Causal Claim Identification): mark text spans and classify each
# into one of four categories:
# - Claim: communicates a causal interaction between an intervention and an outcome
# - Experience: relates an outcome to an intervention based on personal experience
# - Experience-based claim: a claim grounded in personal experience
# - Question: poses a question (e.g., asking whether something causes an effect)
#
# Subtask 2 (PIO Frame Extraction): from identified claims, tag tokens with the
# PIO frame elements (token-level sequence labeling):
# - Population: the demographic group affected
# - Intervention: the treatment or action applied
# - Outcome: the resulting effect or symptom
annotation_task_name: "Causal Medical Claim Identification and PIO Frame Extraction"
task_dir: "."
data_files:
- sample-data.json
item_properties:
id_key: "id"
text_key: "text"
output_annotation_dir: "annotation_output/"
output_annotation_format: "json"
port: 8000
server_name: localhost
annotation_schemes:
- annotation_type: span
name: causal_claim
description: "Subtask 1: mark spans and classify each causal-claim type"
labels:
- "Claim"
- "Experience"
- "Experience-based claim"
- "Question"
- annotation_type: span
name: pio_frame
description: "Subtask 2: tag PIO frame elements within identified claims"
labels:
- "Population"
- "Intervention"
- "Outcome"
annotation_instructions: |
You will see a social-media (Reddit) post about a health topic.
Subtask 1 - Causal Claim Identification:
1. Read the post carefully.
2. Highlight spans (which may be partial sentences) that express a causal
medical statement and label each as Claim, Experience, Experience-based
claim, or Question.
Subtask 2 - PIO Frame Extraction:
3. Within the identified claim spans, tag the Population, Intervention, and
Outcome elements.
html_layout: |
<div style="padding: 15px; max-width: 800px; margin: auto;">
<div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
<strong style="color: #0369a1;">Reddit Health Post:</strong>
<p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
</div>
</div>
allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false
Sample Datasample-data.json
[
{
"id": "causal_med_001",
"text": "I started taking magnesium glycinate at night and my leg cramps completely went away within a week. Honestly it changed my sleep too."
},
{
"id": "causal_med_002",
"text": "Has anyone else found that cutting out dairy helped with their acne? My skin cleared up so much after I stopped drinking milk."
}
]
// ... and 8 more itemsGet This Design
Clone or download from the repository
Quick start:
git clone https://github.com/davidjurgens/potato-showcase.git cd potato-showcase/semeval/2023/task08-causal-medical-claim potato start config.yaml
Dataset & paper
Khetan et al., SemEval 2023
Citation (BibTeX)
@inproceedings{khetan-etal-2023-semeval,
title = "{S}em{E}val-2023 Task 8: Causal Medical Claim Identification and Related {PIO} Frame Extraction from Social Media Posts",
author = "Khetan, Vivek and Wadhwa, Somin and Wallace, Byron and Amir, Silvio",
booktitle = "Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
pages = "2266--2274",
doi = "10.18653/v1/2023.semeval-1.311",
url = "https://aclanthology.org/2023.semeval-1.311"
}Details
Annotation Types
Domain
Use Cases
Tags
Found an issue or want to improve this design?
Open an IssueRelated Designs
Analysis of Clinical Text: Disorder Identification and Normalization
Identify disorder mentions and their attributes in clinical discharge summaries, based on SemEval-2015 Task 14 (Elhadad et al.). Annotators mark disorder spans, body locations, severity indicators, and classify the assertion status of each disorder.
Aspect-Based Sentiment Analysis
Identification of aspect terms in review text with sentiment polarity classification for each aspect. Based on SemEval-2016 Task 5 (ABSA).
BioNLP 2011 - Gene Regulation Event Extraction
Biomedical event extraction for gene regulation, based on the BioNLP 2011 Shared Task (Kim et al., ACL Workshop 2011). Annotators identify biological entities and mark regulatory events such as gene expression, transcription, and protein catabolism in scientific abstracts.