Determining Sentiment Intensity of English and Arabic Phrases

# Determining Sentiment Intensity of English and Arabic Phrases
# Based on Kiritchenko et al., SemEval 2016
# Paper: https://aclanthology.org/S16-1004/
# Dataset: http://saifmohammad.com/WebPages/BestWorst.html
#
# This task uses Best-Worst Scaling (BWS, a.k.a. MaxDiff): annotators are shown
# a tuple of four phrases and select the ONE that is most positive (best) and the
# ONE that is most negative (worst). Aggregating many such judgments yields a
# real-valued sentiment intensity score in [0, 1] for each phrase (0 = most
# negative, 1 = most positive).

annotation_task_name: "Sentiment Intensity (Best-Worst Scaling)"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "prompt"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  - annotation_type: radio
    name: bws_best
    description: "Which phrase is the MOST POSITIVE?"
    horizontal: true
    labels:
      - "Phrase 1"
      - "Phrase 2"
      - "Phrase 3"
      - "Phrase 4"
    sequential_key_binding: true
  - annotation_type: radio
    name: bws_worst
    description: "Which phrase is the MOST NEGATIVE?"
    horizontal: true
    labels:
      - "Phrase 1"
      - "Phrase 2"
      - "Phrase 3"
      - "Phrase 4"
    sequential_key_binding: true

annotation_instructions: |
  You will be shown a set of four phrases at a time. For this set, make two
  judgments:
  - Best: pick the ONE phrase that is the MOST POSITIVE in sentiment.
  - Worst: pick the ONE phrase that is the MOST NEGATIVE in sentiment.
  The "best" and "worst" phrases must be different. These comparative choices are
  later aggregated (Best-Worst Scaling) into a real-valued sentiment intensity
  score in [0, 1] for each phrase.

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #0369a1;">Compare these four phrases:</strong>
      <p style="font-size: 18px; line-height: 1.7; margin: 8px 0 0 0;">{{prompt}}</p>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false