Multilingual and Crosslingual Fact-Checked Claim Retrieval

# Multilingual and Crosslingual Fact-Checked Claim Retrieval
# Based on Peng et al., SemEval-2025 Task 7
# Paper: https://aclanthology.org/2025.semeval-1.323/
# Dataset: https://zenodo.org/records/14989177
#
# In the original shared task, systems retrieve/rank previously fact-checked
# claims that match a given social media post (monolingual and crosslingual).
# This annotation adaptation asks annotators to judge how well a candidate
# previously fact-checked claim matches the claim from a social media post,
# providing a graded relevance judgment and a brief justification.

annotation_task_name: "Multilingual and Crosslingual Fact-Checked Claim Retrieval"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  - annotation_type: radio
    name: match_judgment
    description: "How well does the previously fact-checked claim match the claim in the social media post?"
    labels:
      - "Exact Match"
      - "Partial Match"
      - "Related But Different"
      - "Not Related"
    keyboard_shortcuts:
      "Exact Match": "1"
      "Partial Match": "2"
      "Related But Different": "3"
      "Not Related": "4"
    tooltips:
      "Exact Match": "The fact-checked claim addresses the exact same claim made in the post"
      "Partial Match": "The fact-checked claim addresses part of the post's claim or a closely related variant"
      "Related But Different": "The fact-checked claim covers the same topic but a different specific claim"
      "Not Related": "The fact-checked claim is about a different topic or claim entirely"

  - annotation_type: text
    name: justification
    description: "Provide a brief justification for your match judgment."

annotation_instructions: |
  You will be shown a claim from a social media post and a candidate
  previously fact-checked claim. In multilingual and crosslingual settings
  the two may be written in different languages. Your task is to:
  1. Read the claim from the social media post carefully.
  2. Read the candidate previously fact-checked claim and consider its content.
  3. Judge how well the fact-checked claim matches the post's claim using the four categories.
  4. Provide a brief justification explaining your reasoning.

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #0369a1;">Claim from Social Media Post:</strong>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
    </div>
    <div style="background: #fefce8; border: 1px solid #fde68a; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #a16207;">Previously Fact-Checked Claim:</strong>
      <p style="font-size: 15px; line-height: 1.6; margin: 8px 0 0 0;">{{fact_check}}</p>
    </div>
    <div style="background: #f0fdf4; border: 1px solid #bbf7d0; border-radius: 8px; padding: 12px;">
      <strong style="color: #166534;">Source Language:</strong> <span>{{source_language}}</span>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false