Skip to content
advancedtext

Causal Medical Claim Identification and PIO Frame Extraction

Identify causal medical claim spans in Reddit health posts (Subtask 1: Claim, Experience, Experience-based claim, Question) and extract PIO frames — Population, Intervention, Outcome (Subtask 2). Based on SemEval-2023 Task 8 (Khetan et al.).

PERORGLOCPERORGLOCDATESelect text to annotate

Configuration Fileconfig.yaml

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# Causal Medical Claim Identification and PIO Frame Extraction
# Based on Khetan et al., SemEval 2023 Task 8
# Paper: https://aclanthology.org/2023.semeval-1.311/
# Task page: https://causalclaims.github.io/
#
# The task uses Reddit health/social-media posts and has two subtasks:
#
# Subtask 1 (Causal Claim Identification): mark text spans and classify each
# into one of four categories:
# - Claim: communicates a causal interaction between an intervention and an outcome
# - Experience: relates an outcome to an intervention based on personal experience
# - Experience-based claim: a claim grounded in personal experience
# - Question: poses a question (e.g., asking whether something causes an effect)
#
# Subtask 2 (PIO Frame Extraction): from identified claims, tag tokens with the
# PIO frame elements (token-level sequence labeling):
# - Population: the demographic group affected
# - Intervention: the treatment or action applied
# - Outcome: the resulting effect or symptom

annotation_task_name: "Causal Medical Claim Identification and PIO Frame Extraction"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  - annotation_type: span
    name: causal_claim
    description: "Subtask 1: mark spans and classify each causal-claim type"
    labels:
      - "Claim"
      - "Experience"
      - "Experience-based claim"
      - "Question"

  - annotation_type: span
    name: pio_frame
    description: "Subtask 2: tag PIO frame elements within identified claims"
    labels:
      - "Population"
      - "Intervention"
      - "Outcome"

annotation_instructions: |
  You will see a social-media (Reddit) post about a health topic.
  Subtask 1 - Causal Claim Identification:
  1. Read the post carefully.
  2. Highlight spans (which may be partial sentences) that express a causal
     medical statement and label each as Claim, Experience, Experience-based
     claim, or Question.
  Subtask 2 - PIO Frame Extraction:
  3. Within the identified claim spans, tag the Population, Intervention, and
     Outcome elements.

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #0369a1;">Reddit Health Post:</strong>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false

Sample Datasample-data.json

json
[
  {
    "id": "causal_med_001",
    "text": "I started taking magnesium glycinate at night and my leg cramps completely went away within a week. Honestly it changed my sleep too."
  },
  {
    "id": "causal_med_002",
    "text": "Has anyone else found that cutting out dairy helped with their acne? My skin cleared up so much after I stopped drinking milk."
  }
]

// ... and 8 more items

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/semeval/2023/task08-causal-medical-claim
potato start config.yaml

Dataset & paper

Khetan et al., SemEval 2023

Citation (BibTeX)

bibtex
@inproceedings{khetan-etal-2023-semeval,
    title = "{S}em{E}val-2023 Task 8: Causal Medical Claim Identification and Related {PIO} Frame Extraction from Social Media Posts",
    author = "Khetan, Vivek  and Wadhwa, Somin  and Wallace, Byron  and Amir, Silvio",
    booktitle = "Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    pages = "2266--2274",
    doi = "10.18653/v1/2023.semeval-1.311",
    url = "https://aclanthology.org/2023.semeval-1.311"
}

Details

Annotation Types

span

Domain

NLPBiomedicalSemEval

Use Cases

Causal Claim DetectionPIO Frame ExtractionMedical NLP

Tags

semevalsemeval-2023shared-taskcausal-claimspiomedicalsocial-media

Found an issue or want to improve this design?

Open an Issue