Multilingual Semantic Word Similarity

# Multilingual Semantic Word Similarity
# Based on Camacho-Collados et al., SemEval 2017
# Paper: https://aclanthology.org/S17-2002/
# Dataset: http://alt.qcri.org/semeval2017/task2/
#
# Annotators judge the degree of semantic similarity between two words
# on a graded scale. This task supports multilingual and cross-lingual
# word pair evaluation for distributional semantic models.
#
# Similarity Scale (five-point Likert, 0-4; Table 2 of the task paper):
# 4 = Very similar (the two words are synonyms, e.g., midday-noon)
# 3 = Similar (share many important ideas but differ in detail, e.g., lion-zebra)
# 2 = Slightly similar (share a common topic/domain/function, e.g., airplane-pilot)
# 1 = Dissimilar (clearly dissimilar concepts, minor shared domain, e.g., software-keyboard)
# 0 = Totally dissimilar and unrelated (e.g., pencil-frog)
# Annotators may use a step size of 0.25 to indicate a blend of two levels.
#
# Annotation Guidelines:
# 1. Read both words carefully
# 2. Consider the most common sense of each word
# 3. Rate how semantically similar the two words are (0-4)
# 4. Distinguish similarity from mere relatedness
# 5. Note the language(s) of the word pair (English, Farsi, German, Italian, Spanish)

annotation_task_name: "Multilingual Semantic Word Similarity"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  - annotation_type: likert
    name: similarity_rating
    description: "How semantically similar are these two words? (0 = totally unrelated, 4 = synonyms)"
    min_label: "Totally dissimilar / unrelated"
    max_label: "Very similar (synonyms)"
    size: 5

  - annotation_type: slider
    name: similarity_slider
    description: "Fine-grained rating: slide between 0 (totally unrelated) and 4 (synonyms); steps of 0.25 allowed"
    min_value: 0
    max_value: 4
    starting_value: 2

annotation_instructions: |
  You will be shown two words, possibly from different languages (the task covers
  English, Farsi, German, Italian, and Spanish, including cross-lingual pairs). Your
  task is to judge how semantically similar these two words are on a 0-4 scale:

    4 = Very similar (the two words are synonyms, e.g., midday-noon)
    3 = Similar (share many important ideas but differ in detail, e.g., lion-zebra)
    2 = Slightly similar (share a common topic/domain/function, e.g., airplane-pilot)
    1 = Dissimilar (clearly dissimilar concepts, minor shared domain, e.g., software-keyboard)
    0 = Totally dissimilar and unrelated (e.g., pencil-frog)

  Use the Likert scale for the categorical judgment and the slider for a fine-grained
  rating (steps of 0.25 are allowed to indicate a blend of two levels).

  Consider the most common meaning of each word, and be careful to distinguish
  semantic similarity from mere topical relatedness.

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <div style="display: flex; justify-content: center; align-items: center; gap: 40px;">
        <div style="text-align: center;">
          <span style="color: #64748b; font-size: 13px;">Word 1</span>
          <p style="font-size: 22px; font-weight: bold; margin: 4px 0;">{{text}}</p>
        </div>
        <span style="font-size: 24px; color: #94a3b8;">~</span>
        <div style="text-align: center;">
          <span style="color: #64748b; font-size: 13px;">Word 2</span>
          <p style="font-size: 22px; font-weight: bold; margin: 4px 0;">{{word_2}}</p>
        </div>
      </div>
      <p style="text-align: center; color: #64748b; margin-top: 8px;">Language: <strong>{{language}}</strong></p>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false