Predicting Multilingual and Cross-Lingual Lexical Entailment

# Predicting Multilingual and Cross-Lingual Lexical Entailment
# Based on Glavaš et al., SemEval 2020
# Paper: https://aclanthology.org/2020.semeval-1.2/
# Dataset: https://competitions.codalab.org/competitions/20865
#
# Following the HyperLex-based graded lexical entailment (LE) protocol:
# annotators rate the DEGREE to which word 1 (X) is a type of word 2 (Y)
# — the is-a / hyponym-hypernym relation — on a graded scale, and mark the
# direction of entailment. Applies across and within languages.

annotation_task_name: "Predicting Multilingual and Cross-Lingual Lexical Entailment"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  - annotation_type: radio
    name: entailment_degree
    description: "To what degree is word 1 (X) a type of word 2 (Y)?"
    labels:
      - "6 - Definitely a type of"
      - "5"
      - "4"
      - "3 - Somewhat / loosely a type of"
      - "2"
      - "1"
      - "0 - Not at all a type of"
    keyboard_shortcuts:
      "6 - Definitely a type of": "6"
      "5": "5"
      "4": "4"
      "3 - Somewhat / loosely a type of": "3"
      "2": "2"
      "1": "1"
      "0 - Not at all a type of": "0"
    tooltips:
      "6 - Definitely a type of": "X is fully a type of Y (e.g., sparrow is a type of bird)"
      "3 - Somewhat / loosely a type of": "There is a partial or weak type-of relation"
      "0 - Not at all a type of": "X is not a type of Y (no is-a / hyponym-hypernym relation)"
  - annotation_type: radio
    name: entailment_direction
    description: "Direction of the type-of (is-a) relation"
    labels:
      - "X is a type of Y (forward)"
      - "Y is a type of X (reverse)"
      - "No type-of relation"
    keyboard_shortcuts:
      "X is a type of Y (forward)": "f"
      "Y is a type of X (reverse)": "r"
      "No type-of relation": "n"
    tooltips:
      "X is a type of Y (forward)": "Word 1 is more specific (e.g., dog -> animal)"
      "Y is a type of X (reverse)": "Word 2 is more specific (e.g., animal -> dog)"
      "No type-of relation": "Neither word is a type of the other"

annotation_instructions: |
  Following the HyperLex graded lexical entailment protocol, you will see word 1
  (X, shown in an example sentence for sense disambiguation) and a candidate word 2
  (Y), together with the language. Your task is to:
  1. Read the example and understand the intended sense of word 1 (X).
  2. Rate the DEGREE to which X is a type of Y — the is-a / hyponym-hypernym
     relation — on the 0-6 scale (0 = not at all, 6 = definitely a type of).
     For example, "sparrow -> bird" is a strong (6) type-of relation, while
     "hammer -> screwdriver" has no type-of relation (0).
  3. Mark the DIRECTION of the relation: whether X is a type of Y (forward),
     Y is a type of X (reverse), or there is no type-of relation.

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #0369a1;">Word 1 in Context:</strong>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
    </div>
    <div style="background: #fefce8; border: 1px solid #fde68a; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #a16207;">Word 2:</strong>
      <span style="font-size: 18px; font-weight: bold; color: #b45309;">{{word_2}}</span>
    </div>
    <div style="background: #f0fdf4; border: 1px solid #bbf7d0; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #166534;">Language:</strong>
      <span style="font-size: 14px;">{{language}}</span>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false