Commonsense Validation and Explanation

# Commonsense Validation and Explanation
# Based on Wang et al., SemEval 2020
# Paper: https://aclanthology.org/2020.semeval-1.39/
# Dataset: https://github.com/wangcunxiang/SemEval2020-Task4-Commonsense-Validation-and-Explanation
#
# Annotators judge which of two statements makes sense based on commonsense
# knowledge and provide a brief explanation of why the other does not.

annotation_task_name: "Commonsense Validation and Explanation"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  - annotation_type: radio
    name: sense_judgment
    description: "Which sentence makes sense according to commonsense knowledge?"
    labels:
      - "Makes Sense"
      - "Does Not Make Sense"
    keyboard_shortcuts:
      "Makes Sense": "1"
      "Does Not Make Sense": "2"
    tooltips:
      "Makes Sense": "The sentence describes a plausible situation according to commonsense"
      "Does Not Make Sense": "The sentence describes an implausible or impossible situation"

  - annotation_type: text
    name: explanation
    description: "Briefly explain why the sentence that does not make sense is implausible."

annotation_instructions: |
  You will see two sentences. Your task is to:
  1. Read both Sentence 1 and Sentence 2 carefully.
  2. Decide which sentence makes sense and which does not, based on commonsense.
  3. Select your judgment for Sentence 1 (the judgment for Sentence 2 is implied).
  4. Write a brief explanation of why the nonsensical sentence is implausible.

  Focus on commonsense knowledge about how the world works, not grammatical correctness.

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #0369a1;">Sentence 1:</strong>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
    </div>
    <div style="background: #fefce8; border: 1px solid #fde68a; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #a16207;">Sentence 2:</strong>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{sentence_2}}</p>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false