Arena Hard Auto - LLM Pairwise Evaluation

# Arena Hard Auto - LLM Pairwise Evaluation
# Based on Li et al., arXiv 2024
# Paper: https://arxiv.org/abs/2406.11939
# Dataset: https://github.com/lm-sys/arena-hard-auto
#
# Pairwise evaluation of LLM responses on challenging prompts from the
# Arena Hard benchmark. Uses a continuous scale for nuanced preference
# judgments rather than simple binary choices. Annotators also rate the
# difficulty of each question.
#
# Pairwise Scale:
# - -3: Response A is much better
# -  0: Tie (both responses are equally good)
# -  3: Response B is much better
#
# Difficulty Rating:
# - 1 (Very Easy) to 5 (Very Hard)
#
# Annotation Guidelines:
# 1. Read the prompt carefully
# 2. Read both responses thoroughly
# 3. Use the scale to indicate your preference strength
# 4. Rate how difficult the prompt is to answer well

annotation_task_name: "Arena Hard Auto - LLM Pairwise Evaluation"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  # Step 1: Pairwise preference on a scale
  - annotation_type: pairwise
    name: preference
    description: "Compare the two responses. Use the scale to indicate how much better one is than the other."
    mode: "scale"
    scale:
      min: -3
      max: 3
      step: 1
      labels:
        "-3": "A much better"
        "0": "Tie"
        "3": "B much better"

  # Step 2: Difficulty rating
  - annotation_type: likert
    name: difficulty
    description: "How difficult is this prompt to answer well?"
    min_label: "Very Easy"
    max_label: "Very Hard"
    size: 5

annotation_instructions: |
  You will evaluate pairs of LLM responses on challenging prompts from the Arena Hard benchmark.

  For each item:
  1. Read the user prompt carefully to understand what is being asked.
  2. Read both Response A and Response B thoroughly.
  3. Use the continuous scale (-3 to +3) to indicate your preference:
     - -3: Response A is much better
     - -2: Response A is better
     - -1: Response A is slightly better
     -  0: Tie (both are equally good or equally bad)
     - +1: Response B is slightly better
     - +2: Response B is better
     - +3: Response B is much better
  4. Rate how difficult the prompt is to answer well (1 = Very Easy, 5 = Very Hard).

  Consider: accuracy, completeness, clarity, helpfulness, and relevance.

html_layout: |
  <div style="padding: 15px; max-width: 900px; margin: auto;">
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #0369a1;">Prompt:</strong>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
    </div>
    <div style="display: flex; gap: 16px;">
      <div style="flex: 1; background: #e3f2fd; border: 2px solid #1976d2; border-radius: 8px; padding: 16px;">
        <h4 style="margin-top: 0; color: #1976d2;">Response A:</h4>
        <div style="white-space: pre-wrap; font-size: 14px; line-height: 1.6;">{{response_a}}</div>
      </div>
      <div style="flex: 1; background: #fce4ec; border: 2px solid #c62828; border-radius: 8px; padding: 16px;">
        <h4 style="margin-top: 0; color: #c62828;">Response B:</h4>
        <div style="white-space: pre-wrap; font-size: 14px; line-height: 1.6;">{{response_b}}</div>
      </div>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 3
allow_skip: true
skip_reason_required: false