SPIN Self-Play Preference Annotation

# SPIN Self-Play Preference Annotation
# Based on Chen et al., ICML 2024
# Paper: https://arxiv.org/abs/2401.01335
# Dataset: https://huggingface.co/datasets/UCLA-AGI/SPIN_iter0
#
# Annotation task for Self-Play Fine-Tuning (SPIN). The key idea is to
# train a language model to distinguish its own outputs from human-written
# responses. Annotators compare two responses to a prompt and identify
# which was written by a human versus generated by an AI model.
#
# Preference Labels:
# - Human Response: The annotator believes this is the human-written response
# - AI Response: The annotator believes this is the AI-generated response
# - Cannot Tell: The annotator cannot reliably distinguish between the two
#
# Fluency Rating:
# - Fluent: Both responses read naturally with no issues
# - Somewhat Fluent: Minor awkwardness or unnatural phrasing
# - Not Fluent: Significant issues with naturalness or coherence
#
# Annotation Guidelines:
# 1. Read the prompt carefully
# 2. Read both responses and try to identify which is human vs. AI
# 3. Select your judgment for the pairwise comparison
# 4. Rate the overall fluency of the responses

annotation_task_name: "SPIN Self-Play Preference Annotation"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  # Step 1: Identify human vs. AI response
  - annotation_type: pairwise
    name: source_preference
    description: "Which response do you believe is the human-written response?"
    mode: "binary"
    labels:
      - "Human Response"
      - "AI Response"
      - "Cannot Tell"
    keyboard_shortcuts:
      "Human Response": "a"
      "AI Response": "b"
      "Cannot Tell": "c"
    tooltips:
      "Human Response": "Response A appears to be written by a human"
      "AI Response": "Response A appears to be generated by AI (Response B is human)"
      "Cannot Tell": "Cannot reliably distinguish which is human-written"

  # Step 2: Fluency rating
  - annotation_type: radio
    name: fluency
    description: "How fluent and natural do the responses read overall?"
    labels:
      - "Fluent"
      - "Somewhat Fluent"
      - "Not Fluent"
    keyboard_shortcuts:
      "Fluent": "1"
      "Somewhat Fluent": "2"
      "Not Fluent": "3"
    tooltips:
      "Fluent": "Both responses read naturally with no awkwardness"
      "Somewhat Fluent": "Minor awkwardness or unnatural phrasing in one or both"
      "Not Fluent": "Significant issues with naturalness or coherence"

annotation_instructions: |
  You will see a prompt and two responses. One was written by a human and the other
  was generated by an AI language model. Your task is to identify which is which.

  For each item:
  1. Read the prompt carefully.
  2. Read both Response A and Response B thoroughly.
  3. Try to identify which response was written by a human and which by AI.
  4. Rate the overall fluency of both responses.

  Tips for distinguishing human vs. AI:
  - Human responses may be more concise or contain informal language
  - AI responses may be more structured, verbose, or formulaic
  - Look for patterns like numbered lists, hedging language, or overly balanced viewpoints
  - Trust your intuition - if both seem equally natural, select "Cannot Tell"

html_layout: |
  <div style="padding: 15px; max-width: 900px; margin: auto;">
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #0369a1;">Prompt:</strong>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
    </div>
    <div style="display: flex; gap: 16px;">
      <div style="flex: 1; background: #e3f2fd; border: 2px solid #1976d2; border-radius: 8px; padding: 16px;">
        <h4 style="margin-top: 0; color: #1976d2;">Response A:</h4>
        <div style="white-space: pre-wrap; font-size: 14px; line-height: 1.6;">{{response_a}}</div>
      </div>
      <div style="flex: 1; background: #fce4ec; border: 2px solid #c62828; border-radius: 8px; padding: 16px;">
        <h4 style="margin-top: 0; color: #c62828;">Response B:</h4>
        <div style="white-space: pre-wrap; font-size: 14px; line-height: 1.6;">{{response_b}}</div>
      </div>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 100
annotation_per_instance: 3
allow_skip: true
skip_reason_required: false