Reading Comprehension QA

Evaluate question-answer pairs for reading comprehension by verifying answers and rating quality.

ملف الإعدادconfig.yaml

# Reading Comprehension QA Configuration
# Evaluate question-answer pairs for reading comprehension

task_dir: "."
annotation_task_name: "Reading Comprehension QA"

data_files:
  - "data/qa_pairs.json"

item_properties:
  id_key: "id"
  text_key: "display"
  text_display_key: "display"

user_config:
  allow_all_users: true

annotation_schemes:
  - annotation_type: "radio"
    name: "answer_correctness"
    description: "Is the provided answer correct?"
    labels:
      - name: "Correct"
        tooltip: "The answer is fully correct"
        key_value: "c"
        color: "#22c55e"
      - name: "Partially Correct"
        tooltip: "The answer is partly right but incomplete or has minor errors"
        key_value: "p"
        color: "#eab308"
      - name: "Incorrect"
        tooltip: "The answer is wrong"
        key_value: "i"
        color: "#ef4444"
      - name: "Unanswerable"
        tooltip: "The question cannot be answered from the passage"
        key_value: "u"
        color: "#6b7280"

  - annotation_type: "radio"
    name: "question_type"
    description: "What type of reasoning is needed?"
    labels:
      - name: "Explicit"
        tooltip: "Answer stated directly in text"
      - name: "Implicit"
        tooltip: "Requires inference from text"
      - name: "Multi-hop"
        tooltip: "Requires connecting multiple pieces of information"
      - name: "Numerical"
        tooltip: "Requires counting or calculation"

  - annotation_type: "text"
    name: "correct_answer"
    description: "If incorrect/partially correct, what is the right answer?"
    show_if:
      field: "answer_correctness"
      value_in: ["Partially Correct", "Incorrect"]

  - annotation_type: "text"
    name: "evidence_span"
    description: "Copy the text span that supports the answer"

  - annotation_type: "likert"
    name: "question_quality"
    description: "How good is this question?"
    size: 5
    min_label: "Poor question"
    max_label: "Excellent question"

output: "annotation_output/"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

بيانات نموذجيةsample-data.json

[
  {
    "id": "rc_001",
    "passage": "The Amazon rainforest, often referred to as the 'lungs of the Earth,' produces about 20% of the world's oxygen. Spanning across nine countries in South America, it is the world's largest tropical rainforest, covering approximately 5.5 million square kilometers. The forest is home to an estimated 10% of all species on Earth.",
    "question": "What percentage of the world's oxygen does the Amazon produce?",
    "provided_answer": "About 20%",
    "display": "**PASSAGE:**\nThe Amazon rainforest, often referred to as the 'lungs of the Earth,' produces about 20% of the world's oxygen. Spanning across nine countries in South America, it is the world's largest tropical rainforest, covering approximately 5.5 million square kilometers. The forest is home to an estimated 10% of all species on Earth.\n\n**QUESTION:** What percentage of the world's oxygen does the Amazon produce?\n\n**PROVIDED ANSWER:** About 20%"
  },
  {
    "id": "rc_002",
    "passage": "Marie Curie was a Polish-born physicist and chemist who conducted pioneering research on radioactivity. She was the first woman to win a Nobel Prize, the first person to win Nobel Prizes in two different sciences (Physics in 1903 and Chemistry in 1911), and the first woman to become a professor at the University of Paris.",
    "question": "How many Nobel Prizes did Marie Curie win?",
    "provided_answer": "She won three Nobel Prizes.",
    "display": "**PASSAGE:**\nMarie Curie was a Polish-born physicist and chemist who conducted pioneering research on radioactivity. She was the first woman to win a Nobel Prize, the first person to win Nobel Prizes in two different sciences (Physics in 1903 and Chemistry in 1911), and the first woman to become a professor at the University of Paris.\n\n**QUESTION:** How many Nobel Prizes did Marie Curie win?\n\n**PROVIDED ANSWER:** She won three Nobel Prizes."
  }
]

// ... and 1 more items

احصل على هذا التصميم

View on GitHub

Clone or download from the repository

بدء سريع:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/templates/text/reading-comprehension
potato start config.yaml

التفاصيل

أنواع التوسيم

radiospantext

المجال

nlpqa

حالات الاستخدام

question-answeringcomprehension

الوسوم

qareadingcomprehensionanswersnlp

وجدت مشكلة أو تريد تحسين هذا التصميم؟

افتح مشكلة

تصاميم ذات صلة

Biomedical Entity Linking (MedMentions)

Entity mention detection and UMLS concept linking for biomedical text based on MedMentions. Annotators identify biomedical entity mentions in PubMed abstracts and link them to UMLS Concept Unique Identifiers (CUIs), supporting large-scale biomedical knowledge base construction and clinical NLP.

radiospan

Check-COVID: Fact-Checking COVID-19 News Claims

Fact-checking COVID-19 news claims. Annotators verify claims against evidence, identify supporting/refuting spans, and provide verdicts with explanations. Based on the Check-COVID dataset targeting misinformation during the pandemic.

radiospan

Clickbait Spoiling

Classification and extraction of spoilers for clickbait posts, including spoiler type identification and span-level spoiler detection. Based on SemEval-2023 Task 5 (Hagen et al.).

textradio