HelpSteer2-Preference Seven-Point Preference Strength

# HelpSteer2-Preference: seven-point preference strength for reward-model training
#
# Dataset: HelpSteer2 (preference subset), NVIDIA, annotated through Scale AI
# Paper: Wang et al., "HelpSteer2-Preference: Complementing Ratings with
#   Preferences", arXiv 2410.01257 (2024); also an ICLR 2025 poster
# BibTeX (from the registry response, reformatted):
#   @misc{wang2024helpsteer2preference,
#     author = {Wang, Zhilin and Bukharin, Alexander and Delalleau, Olivier and Egert, Daniel and Shen, Gerald and Zeng, Jiaqi and Kuchaiev, Oleksii and Dong, Yi},
#     title = {HelpSteer2-Preference: Complementing Ratings with Preferences},
#     publisher = {arXiv},
#     year = {2024},
#     doi = {10.48550/ARXIV.2410.01257},
#     url = {https://arxiv.org/abs/2410.01257}
#   }
# DOI: 10.48550/arXiv.2410.01257
# Dataset URL: https://huggingface.co/datasets/nvidia/HelpSteer2
# Licence: CC BY 4.0. The ten sample items are real rows from
#   preference/preference.jsonl.gz (train split), with their released
#   preference strength kept in `released_preference_strength`.
#
# Preference scale, quoted from the paper (Section 2, Data Collection):
#   -3. Response 1 is much better than Response 2
#   -2. Response 1 is better than Response 2
#   -1. Response 1 is slightly better than Response 2
#    1. Response 2 is slightly better than Response 1
#    2. Response 2 is better than Response 1
#    3. Response 2 is much better than Response 1
# -100. Neither response is valid
# "we force the annotator to make a preference choice between the two
# responses (i.e. no option for Both responses are equally good), except for
# 'Neither response is valid'".
#
# Adaptations:
#   - The paper's annotators first rate each response on five Likert-5
#     dimensions (helpfulness, correctness, coherence, complexity, verbosity).
#     This design keeps only the preference, the "neither valid" flag and the
#     written justification.
#   - "Neither response is valid" (-100) is a separate radio scheme, because a
#     pairwise scale stores one number on the -3 to 3 range.
#   - Potato's scale slider starts on 0 and cannot exclude it, so the paper's
#     rule that 0 is not an answer is stated in the instructions and is not
#     enforced by the interface.
#   - The paper's justification asks for a preference statement and an
#     elaboration. This design offers one free-text box.
#   - Sample items are single-turn rows only, and `released_preference_strength`
#     (the paper's aggregate over the three most similar annotators) is stored
#     beside each item for comparison and is not shown to annotators.
#
# Model training: the `reward-model` trainer (Bradley-Terry loss,
# -log sigmoid(r_winner - r_loser)) fits a scalar reward model on the
# pairwise scale answers. The side the slider leans toward wins, weighted by
# |value| / 3, and answers at 0 are skipped. Each reward is of the response
# given the item's `prompt`. Open /admin/training with the admin key below,
# pick `reward-model` and the `preference_strength` scheme, and install
# `pip install 'potato-annotation[train-text]'` first. The default encoder is
# distilbert/distilroberta-base. The trainer learns from the labels collected
# here. It does not reproduce the paper's reward model, which was trained on
# the full released dataset with a much larger base model, and it does not
# serve or deploy a model. Predictions come back as a prelabel on the scale.

annotation_task_name: "HelpSteer2-Preference: preference strength between two responses"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: id
  text_key: prompt

output_annotation_dir: "annotation_output/"
export_annotation_format: jsonl

# Admin key for /admin/training (the reward-model run).
admin_api_key: "demo-admin-key"

# The paper asked the vendor for 3-5 independent annotators per task.
num_annotators_per_item: 3

annotation_schemes:
  - annotation_type: pairwise
    name: preference_strength
    description: "Which response is better, and by how much? Choose a side even when the difference is small."
    mode: scale
    items_key: responses
    show_labels: true
    labels:
      - "Response 1"
      - "Response 2"
    scale:
      min: -3
      max: 3
      step: 1
      default: 0
      labels:
        min: "much better"
        max: "much better"
        center: "0 is not an allowed answer"
    label_requirement:
      required: true

  - annotation_type: radio
    name: validity
    description: "Is at least one response valid? Choose the second option only when both are so bad that picking a winner is pointless."
    labels:
      - "At least one response is valid"
      - "Neither response is valid"

  - annotation_type: text
    name: justification
    description: "Explain your preference. Say which response is better and why, with reference to the prompt."
    multiline: true
    rows: 4
    cols: 60
    min_chars: 20
    placeholder: "Response 2 is better than Response 1 because ..."

model_training:
  enabled: true
  writeback:
    schemas: ["preference_strength"]
    include_unlabeled: true
    min_confidence: 0.6

user_config:
  allow_all_users: true

annotation_instructions: |
  ## Compare two responses to one prompt

  Read the prompt, then both responses. Use the slider to say which response
  is better and by how much.

  | Value | Meaning |
  |---|---|
  | -3 | Response 1 is much better than Response 2 |
  | -2 | Response 1 is better than Response 2 |
  | -1 | Response 1 is slightly better than Response 2 |
  | 1 | Response 2 is slightly better than Response 1 |
  | 2 | Response 2 is better than Response 1 |
  | 3 | Response 2 is much better than Response 1 |

  There is no "equal" answer. Leave the slider off 0 and look closer at the
  differences, even when they seem minor. If both responses are so bad that
  picking a winner is pointless, choose "Neither response is valid".

  Write a short justification that names the better response and the reason.