NumEval - Numeral-Aware Language Understanding and Generation

# NumEval - Numeral-Aware Language Understanding and Generation
# Based on Chen et al., SemEval-2024 Task 7 (NumEval)
# Paper: https://aclanthology.org/2024.semeval-1.213/
# Task site / datasets: https://sites.google.com/view/numeval/numeval
#
# NumEval bundles five datasets across three tasks:
#   1. Quantitative Understanding (Quantitative 101, English)
#        - QP   : Quantitative Prediction   -> predict the magnitude of a masked number
#        - QNLI : Quantitative NLI          -> entailment between number-bearing sentences
#        - QQA  : Quantitative Question Answering -> pick the correct option
#   2. Reading Comprehension of the Numerals in Text (NQuAD, Chinese)
#        - choose the number that best fills a blank in the question stem
#   3. Numeral-Aware Headline Generation (Num-HG, English)
#        - Numerical Reasoning : compute the number for the masked headline slot
#        - Headline Generation : write a numeral-aware headline
# Each sample item carries a "subtask" field; annotators fill the fields
# relevant to that subtask and may leave the others blank.

annotation_task_name: "NumEval - Numeral-Aware Language Understanding and Generation"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  - annotation_type: radio
    name: subtask
    description: "Which NumEval subtask does this item belong to?"
    labels:
      - "QP (Quantitative Prediction)"
      - "QNLI (Quantitative NLI)"
      - "QQA (Quantitative Question Answering)"
      - "Reading Comprehension (NQuAD)"
      - "Headline Generation (Num-HG)"
    tooltips:
      "QP (Quantitative Prediction)": "Predict the magnitude of the masked number in the sentence."
      "QNLI (Quantitative NLI)": "Decide the inference relation between two number-bearing sentences."
      "QQA (Quantitative Question Answering)": "Compare the numbers and choose the correct option."
      "Reading Comprehension (NQuAD)": "Select the number that best fills the blank in the question stem, given the news article."
      "Headline Generation (Num-HG)": "Reason about the numeral and/or write a numeral-aware headline for the news article."

  - annotation_type: radio
    name: qnli_label
    description: "QNLI only: inference relation between the two sentences."
    labels:
      - "Entailment"
      - "Contradiction"
      - "Neutral"
    tooltips:
      "Entailment": "The second sentence is entailed by the first."
      "Contradiction": "The two sentences numerically contradict each other."
      "Neutral": "Neither entailment nor contradiction holds."

  - annotation_type: radio
    name: qqa_answer
    description: "QQA only: which option answers the question?"
    labels:
      - "Option 1"
      - "Option 2"

  - annotation_type: radio
    name: rc_answer
    description: "Reading Comprehension (NQuAD) only: which candidate number fills the blank?"
    labels:
      - "A"
      - "B"
      - "C"
      - "D"

  - annotation_type: number
    name: predicted_number
    description: "QP / Numerical Reasoning: the predicted or computed number for the masked slot."

  - annotation_type: text
    name: generated_headline
    description: "Headline Generation only: write a concise numeral-aware headline for the article."

  - annotation_type: text
    name: reasoning
    description: "Optional: briefly explain how you derived the answer."

annotation_instructions: |
  NumEval covers five numeral-aware datasets. Each item is tagged with a subtask;
  fill in only the fields that apply and leave the rest blank.
  1. Read the text and confirm the subtask label.
  2. QP: predict the magnitude of the masked number (enter your best estimate as the number).
  3. QNLI: choose Entailment, Contradiction, or Neutral.
  4. QQA: choose Option 1 or Option 2.
  5. Reading Comprehension (NQuAD): choose the candidate number (A-D) that fills the blank.
  6. Headline Generation (Num-HG): for numerical reasoning enter the computed number;
     for headline generation write a numeral-aware headline.
  7. Optionally record your reasoning.

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #0369a1;">Subtask:</strong> <span>{{subtask}}</span>
      <p style="font-size: 16px; line-height: 1.7; margin: 12px 0 0 0; white-space: pre-wrap;">{{text}}</p>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false