PRMBench Process Reward Error Types

# PRMBench Process Reward Error Types
#
# Dataset: PRMBench (preview release on Hugging Face, 6,216 instances)
# Paper: Song, Mingyang; Su, Zhaochen; Qu, Xiaoye; Zhou, Jiawei; Cheng, Yu. 2025.
#   "PRMBench: A Fine-grained and Challenging Benchmark for Process-Level
#   Reward Models." Proceedings of the 63rd Annual Meeting of the Association
#   for Computational Linguistics (Volume 1: Long Papers), pp. 25299-25346.
#   @inproceedings{song-etal-2025-prmbench,
#     title = "{PRMB}ench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models",
#     author = "Song, Mingyang and Su, Zhaochen and Qu, Xiaoye and Zhou, Jiawei and Cheng, Yu",
#     booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
#     month = jul, year = "2025", address = "Vienna, Austria",
#     publisher = "Association for Computational Linguistics",
#     url = "https://aclanthology.org/2025.acl-long.1230/",
#     doi = "10.18653/v1/2025.acl-long.1230", pages = "25299--25346" }
# DOI: 10.18653/v1/2025.acl-long.1230
# Dataset: https://huggingface.co/datasets/hitsmy/PRMBench_Preview (Apache-2.0)
# Sample items: ten real instances from prmbench_preview.jsonl, credited to
#   Song et al. 2025 under Apache-2.0. The questions and original solutions
#   come from PRM800K (Lightman et al., 2023), which PRMBench builds on.
#
# How PRMBench labels were made (Song et al. 2025, Sections 3.2 and 3.3):
# the labels are mostly synthetic. The authors took completely correct PRM800K
# solutions and queried GPT-4o with class-specific prompts to modify each one
# into a version with erroneous steps ("we query GPT-4o ... to modify the
# ground-truth reasoning process into versions containing erroneous steps").
# Multi-Solution Consistency items instead use correct solutions generated by
# QwQ-32B-Preview and filtered. People checked a sample: "we manually evaluate
# 10% of the total instances", with five volunteers reporting "over 92%
# qualification rate on the correctness metric and over 98% qualification rate
# on the difference metric" (Section 3.3, Appendix A.3.1).
#
# Error sub-categories (nine, quoted from Section 3.1) under three categories.
#   Simplicity
#     Non-Redundancy: "unnecessary steps that do not contribute to the
#       solution"
#     Non-Circular Logic: "the reasoning eventually loops back to a previous
#       step, creating an infinite cycle"
#   Soundness
#     Empirical Soundness: "counterfactual mistakes" that contradict
#       "established ground truth"
#     Step Consistency: "a conflict between a specific step and other steps
#       within a reasoning path"
#     Domain Consistency: a step that "uses a statement or theory valid in
#       other domains or cases but is not valid within the current reasoning
#       chain"
#     Confidence Invariance: "over-confident errors ... where an incorrect
#       statement is made with high confidence"
#   Sensitivity
#     Prerequisite Sensitivity: "critical premises, assumptions, or necessary
#       conditions are absent"
#     Deception Resistance: "statements that appear to be correct but are
#       subtly altered to introduce inaccuracies"
#     Multi-Solution Consistency: "multiple correct reasoning processes of the
#       same question" (these chains contain no error)
#
# Adaptations:
#   - PRMBench is a benchmark for scoring existing process reward models; this
#     design turns it into an annotation task. Annotators relabel each step and
#     name the error sub-category, which the paper's benchmark gives as a
#     generation label rather than an annotation step.
#   - The step label is binary (correct or incorrect), as in PRMBench's
#     step-level labels. Neutral is off.
#   - The sub-category radio adds "No error found" for chains without a flaw,
#     which the paper's Multi-Solution Consistency items are.
#   - Each item carries the dataset's own reference label and error step numbers
#     in reference_* fields. The display does not show them.
#   - The dataset's classification strings map to the paper's names as
#     follows: redundency to Non-Redundancy, circular to Non-Circular Logic,
#     counterfactual to Empirical Soundness, step_contradiction to Step
#     Consistency, domain_inconsistency to Domain Consistency, confidence to
#     Confidence Invariance, missing_condition to Prerequisite Sensitivity,
#     deception to Deception Resistance, multi_solutions to Multi-Solution
#     Consistency. The mapping is read from the definitions and the dataset's
#     own reason texts, not from a table the paper prints.
#
# Training loop: the prm-reward trainer (potato/training/trainers/prm_reward.py)
# fine-tunes a transformer encoder to rate each step correct or wrong from the
# step labels annotators saved, then writes predictions back as step prelabels
# for annotators to confirm. A trace longer than max_length tokens (default
# 1024, lowered to what the model can hold, often 512) is cut from the end, so
# steps past the cut get no label in training and no prediction. It trains on
# process_reward schemes only. It does not learn the sub-category radio, and it
# does not serve or deploy a model. Open /admin/training with the admin key
# below to run it. The trainer needs: pip install 'potato-annotation[train-text]'

annotation_task_name: "PRMBench Process Reward Error Types"

task_dir: "."
data_files:
  - sample-data.json

item_properties:
  id_key: id
  text_key: text
  private_keys:
    - reference_sub_category_label
    - reference_error_steps
    - source_idx

output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

require_password: false
admin_api_key: "demo-admin-key"

instance_display:
  layout:
    direction: vertical
    gap: 16px
  fields:
    - key: text
      type: text
      label: "Problem"
    - key: steps
      type: cot_trace
      label: "Solution (judge each step)"
      display_options:
        show_rail: true
        show_types: false
        collapse_long_steps: true
        clamp_lines: 8

annotation_schemes:
  - annotation_type: process_reward
    name: step_rewards
    description: "Mark each step correct or incorrect. A step that repeats, loops, contradicts, or relies on something unjustified is incorrect."
    steps_key: steps
    step_text_key: text
    mode: per_step
    allow_neutral: false
    inline_with_trace: true
    reward_labels:
      correct: "Correct"
      incorrect: "Incorrect"

  - annotation_type: radio
    name: error_sub_category
    description: "Which PRMBench error sub-category best describes the flaw in this solution?"
    labels:
      - name: "Non-Redundancy"
        tooltip: "Simplicity. Unnecessary steps that do not contribute to the solution."
      - name: "Non-Circular Logic"
        tooltip: "Simplicity. The reasoning loops back to a previous step."
      - name: "Empirical Soundness"
        tooltip: "Soundness. A counterfactual statement that contradicts established ground truth."
      - name: "Step Consistency"
        tooltip: "Soundness. A step conflicts with other steps in the same path."
      - name: "Domain Consistency"
        tooltip: "Soundness. A theory valid in other cases is used where it does not hold."
      - name: "Confidence Invariance"
        tooltip: "Soundness. An incorrect statement made with high confidence."
      - name: "Prerequisite Sensitivity"
        tooltip: "Sensitivity. A critical premise or necessary condition is missing."
      - name: "Deception Resistance"
        tooltip: "Sensitivity. A statement that looks correct but is subtly altered."
      - name: "Multi-Solution Consistency"
        tooltip: "Sensitivity. A valid alternative solution with no error."
      - name: "No error found"
        tooltip: "Adaptation. Every step is correct."

model_training:
  enabled: true
  writeback:
    schemas: ["step_rewards"]
    include_unlabeled: true
    min_confidence: 0.6

user_config:
  allow_all_users: true

num_annotators_per_item: 1
max_annotations_per_user: 10

annotation_instructions: |
  Read the problem, then judge each step of the solution. Mark a step Incorrect
  if it is wrong, redundant, circular, inconsistent, or rests on a missing or
  invalid premise. Then pick the error sub-category that fits the flaw best,
  or "No error found" when every step is correct. The sample chains were
  generated by modifying correct PRM800K solutions, so most contain a planted
  error.