Reading Comprehension QA

Evaluate question-answer pairs for reading comprehension by verifying answers and rating quality.

📝

text annotation

Configuration Fileconfig.yaml

# Reading Comprehension QA Configuration
# Evaluate question-answer pairs for reading comprehension

annotation_task_name: "Reading Comprehension QA"

data_files:
  - "data/qa_pairs.json"

item_properties:
  id_key: "id"
  text_display_key: "display"

user_config:
  allow_all_users: true

annotation_schemes:
  - annotation_type: "radio"
    name: "answer_correctness"
    description: "Is the provided answer correct?"
    labels:
      - name: "Correct"
        tooltip: "The answer is fully correct"
        key_value: "c"
        color: "#22c55e"
      - name: "Partially Correct"
        tooltip: "The answer is partly right but incomplete or has minor errors"
        key_value: "p"
        color: "#eab308"
      - name: "Incorrect"
        tooltip: "The answer is wrong"
        key_value: "i"
        color: "#ef4444"
      - name: "Unanswerable"
        tooltip: "The question cannot be answered from the passage"
        key_value: "u"
        color: "#6b7280"

  - annotation_type: "radio"
    name: "question_type"
    description: "What type of reasoning is needed?"
    labels:
      - name: "Explicit"
        tooltip: "Answer stated directly in text"
      - name: "Implicit"
        tooltip: "Requires inference from text"
      - name: "Multi-hop"
        tooltip: "Requires connecting multiple pieces of information"
      - name: "Numerical"
        tooltip: "Requires counting or calculation"

  - annotation_type: "text"
    name: "correct_answer"
    description: "If incorrect/partially correct, what is the right answer?"
    show_if:
      field: "answer_correctness"
      value_in: ["Partially Correct", "Incorrect"]

  - annotation_type: "text"
    name: "evidence_span"
    description: "Copy the text span that supports the answer"

  - annotation_type: "likert"
    name: "question_quality"
    description: "How good is this question?"
    size: 5
    min_label: "Poor question"
    max_label: "Excellent question"

output: "annotation_output/"

Sample Datasample-data.json

[
  {
    "id": "rc_001",
    "passage": "The Amazon rainforest, often referred to as the 'lungs of the Earth,' produces about 20% of the world's oxygen. Spanning across nine countries in South America, it is the world's largest tropical rainforest, covering approximately 5.5 million square kilometers. The forest is home to an estimated 10% of all species on Earth.",
    "question": "What percentage of the world's oxygen does the Amazon produce?",
    "provided_answer": "About 20%",
    "display": "**PASSAGE:**\nThe Amazon rainforest, often referred to as the 'lungs of the Earth,' produces about 20% of the world's oxygen. Spanning across nine countries in South America, it is the world's largest tropical rainforest, covering approximately 5.5 million square kilometers. The forest is home to an estimated 10% of all species on Earth.\n\n**QUESTION:** What percentage of the world's oxygen does the Amazon produce?\n\n**PROVIDED ANSWER:** About 20%"
  },
  {
    "id": "rc_002",
    "passage": "Marie Curie was a Polish-born physicist and chemist who conducted pioneering research on radioactivity. She was the first woman to win a Nobel Prize, the first person to win Nobel Prizes in two different sciences (Physics in 1903 and Chemistry in 1911), and the first woman to become a professor at the University of Paris.",
    "question": "How many Nobel Prizes did Marie Curie win?",
    "provided_answer": "She won three Nobel Prizes.",
    "display": "**PASSAGE:**\nMarie Curie was a Polish-born physicist and chemist who conducted pioneering research on radioactivity. She was the first woman to win a Nobel Prize, the first person to win Nobel Prizes in two different sciences (Physics in 1903 and Chemistry in 1911), and the first woman to become a professor at the University of Paris.\n\n**QUESTION:** How many Nobel Prizes did Marie Curie win?\n\n**PROVIDED ANSWER:** She won three Nobel Prizes."
  }
]

// ... and 1 more items

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/reading-comprehension
potato start config.yaml

Details

Annotation Types

radiospantext

Domain

nlpqa

Use Cases

question-answeringcomprehension

Related Designs

Fact Verification

Verify claims as supported, refuted, or not enough information based on provided evidence.

radiotext

Relation Extraction

Identify and classify relationships between entities in text (e.g., works-for, located-in, married-to).

radiospan

Sarcasm Detection

Identify sarcastic statements and label their type and target in social media and conversational text.