BRAINTEASER - Commonsense-Defying QA

# BRAINTEASER - Commonsense-Defying QA
# Based on Jiang et al., SemEval 2024
# Paper: https://aclanthology.org/2024.semeval-1.274/
# Dataset: https://brainteasersem.github.io/
#
# This task asks annotators to solve brain teasers that defy default
# commonsense assumptions. Annotators select the best answer from four
# options and provide an explanation of their reasoning.

annotation_task_name: "BRAINTEASER - Commonsense-Defying QA"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  - annotation_type: radio
    name: answer_choice
    description: "Select the best answer to the brain teaser."
    labels:
      - "A"
      - "B"
      - "C"
      - "D"
    keyboard_shortcuts:
      "A": "1"
      "B": "2"
      "C": "3"
      "D": "4"
    tooltips:
      "A": "Option A"
      "B": "Option B"
      "C": "Option C"
      "D": "Option D"

  - annotation_type: text
    name: explanation
    description: "Explain your reasoning for the selected answer."

annotation_instructions: |
  You will be shown a brain teaser question with four answer options. The task
  covers two subtasks: Sentence Puzzles (reasoning defies commonsense at the
  sentence level) and Word Puzzles (the answer violates the default meaning of
  a word and hinges on its letter composition). Your task is to:
  1. Read the question carefully. These are lateral thinking puzzles that may
     defy your initial commonsense assumptions.
  2. Consider all four options before selecting your answer. Following the
     BRAINTEASER format, the fourth option may be "None of the above."
  3. Think creatively -- the answer may involve wordplay, unconventional logic,
     or challenging default assumptions.
  4. Provide a brief explanation of your reasoning.

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #0369a1;">Question:</strong>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
    </div>
    <div style="background: #fefce8; border: 1px solid #fde68a; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #a16207;">Options:</strong>
      <ul style="font-size: 15px; line-height: 1.8; margin: 8px 0 0 0;">
        <li><strong>A:</strong> {{option_a}}</li>
        <li><strong>B:</strong> {{option_b}}</li>
        <li><strong>C:</strong> {{option_c}}</li>
        <li><strong>D:</strong> {{option_d}}</li>
      </ul>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false