Assessing Humor in Edited News Headlines

# Assessing Humor in Edited News Headlines
# Based on Hossain et al., SemEval 2020
# Paper: https://aclanthology.org/2020.semeval-1.98/
# Dataset: https://cs.rochester.edu/u/nhossain/humicroedit.html
#
# Following the two SemEval-2020 Task 7 subtasks: annotators grade how funny
# each edited headline is on a 0-3 scale (Subtask 1), and judge which of two
# edited versions of the same headline is funnier (Subtask 2). Headlines have
# been minimally edited by replacing a single word to create humorous effect.

annotation_task_name: "Assessing Humor in Edited News Headlines"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  # Subtask 1: grade the funniness of each edited headline on the 0-3 scale
  # used in the shared task (0 = Not Funny ... 3 = Funny).
  - annotation_type: likert
    name: funniness_edit_a
    description: "Grade how funny Edit A is (0 = Not Funny, 3 = Funny)."
    min_label: "Not Funny"
    max_label: "Funny"
    size: 4

  - annotation_type: likert
    name: funniness_edit_b
    description: "Grade how funny Edit B is (0 = Not Funny, 3 = Funny)."
    min_label: "Not Funny"
    max_label: "Funny"
    size: 4

  # Subtask 2: given the two edited versions of the same headline, judge which
  # one is funnier (or that they are equally funny).
  - annotation_type: radio
    name: humor_comparison
    description: "Which edited version is funnier?"
    labels:
      - "Edit A Funnier"
      - "Edit B Funnier"
      - "Equally Funny"
    keyboard_shortcuts:
      "Edit A Funnier": "1"
      "Edit B Funnier": "2"
      "Equally Funny": "3"
    tooltips:
      "Edit A Funnier": "Edit A is more humorous than Edit B"
      "Edit B Funnier": "Edit B is more humorous than Edit A"
      "Equally Funny": "Both edits are equally funny (or equally unfunny)"

annotation_instructions: |
  You will see an original news headline together with two edited versions (Edit A
  and Edit B), each created by replacing a single word. Your task is to:
  1. Read the original headline and both edited versions.
  2. Grade how funny each edited version is on the 0-3 scale (0 = Not Funny, 3 = Funny).
  3. Decide which of the two edits is funnier (or whether they are equally funny).

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #0369a1;">Original Headline:</strong>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
    </div>
    <div style="background: #fefce8; border: 1px solid #fde68a; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #a16207;">Edit A</strong> <span style="color: #a16207;">(replaced with "<strong>{{edit_word_a}}</strong>"):</span>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{edit_a}}</p>
    </div>
    <div style="background: #fdf2f8; border: 1px solid #fbcfe8; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #be185d;">Edit B</strong> <span style="color: #be185d;">(replaced with "<strong>{{edit_word_b}}</strong>"):</span>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{edit_b}}</p>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false