ProcessBench Earliest-Error Identification

# ProcessBench Earliest-Error Identification
# Based on: Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu,
#   Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin (2025).
#   "ProcessBench: Identifying Process Errors in Mathematical Reasoning."
#   Proceedings of the 63rd Annual Meeting of the Association for Computational
#   Linguistics (Volume 1: Long Papers), pp. 1009-1024. arXiv:2412.06559.
#   Paper: https://aclanthology.org/2025.acl-long.50/
#   Dataset: https://github.com/QwenLM/ProcessBench
#
# Task: given a math problem and a step-by-step model-generated solution
# (steps indexed from 0), identify the EARLIEST step containing an error, or
# conclude that all steps are correct. This is the ProcessBench formulation:
# unlike PRM800K-style per-step rating (see agentic/prm800k-step-verification),
# only the first error matters, because the correctness of steps after the
# first error is ambiguous. As in the paper's expert annotation protocol
# (Section 3.2), the reference answer from the source dataset is shown to ease
# the annotation, but annotators must still inspect and verify the presented
# solution step by step.
#
# Annotation instructions adapted from Sections 3.1 and 3.2; the paper does
# not publish the verbatim annotator guideline document (annotators received a
# proficiency examination and annotation tutorial). The error criteria from
# Section 3.1 are quoted verbatim inside the instructions below.

annotation_task_name: "ProcessBench Earliest-Error Identification"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "problem"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_instructions: |
  You are shown a math problem and a step-by-step solution generated by a
  language model, together with the reference answer from the source dataset.
  Your task is to identify the **earliest-occurring error** in the solution,
  or to conclude that all steps are correct. You are not required to solve the
  problem from scratch, but you must inspect and verify the presented solution
  step by step.

  The paper's criteria for an erroneous step (quoted verbatim from Section 3.1):

  "Typically but non-inclusively, we consider a step as erroneous if it
  contains any of the following: (1) Mathematical errors: incorrect
  calculations, algebraic manipulations, or formula applications. (2) Logical
  errors: invalid deductions, unwarranted assumptions, or flawed reasoning
  steps. (3) Conceptual errors: misunderstanding or misapplication of
  mathematical or problem concepts. (4) Completeness errors: missing crucial
  conditions, constraints, or necessary justifications that affect the
  solution's validity. Beyond these types of errors, we encourage human
  annotators to determine the correctness of reasoning steps based on their
  own expertise. We do not require human annotators to explicitly annotate
  error types [...]"

  Mark only the earliest erroneous step: for steps after the first error,
  correctness may become ambiguous or debatable, since derivations based on
  incorrect premises can follow valid rules yet remain on a globally incorrect
  reasoning path. If every step is valid, confirm that all steps are correct.
  Note that a solution can reach the correct final answer and still contain an
  erroneous step.

# Steps are pre-segmented into a list in the data, so cot_segmentation is not
# needed; the cot_trace display and the process_reward scheme point straight
# at the step list.
instance_display:
  fields:
    - key: steps
      type: cot_trace
      display_options:
        show_rail: true
        collapse_long_steps: true
        clamp_lines: 10

annotation_schemes:
  - annotation_type: process_reward
    name: earliest_error
    description: "Work through the solution in order and mark the earliest erroneous step, or confirm all steps are correct."
    steps_key: steps
    mode: first_error
    inline_with_trace: true

html_layout: |
  <div style="padding: 15px; max-width: 860px; margin: auto;">
    <div style="background: #e8f4fd; border: 1px solid #b3d9f2; border-radius: 8px; padding: 16px 20px; margin-bottom: 14px;">
      <h3 style="margin: 0 0 8px 0; color: #1a5276; font-size: 16px;">Math Problem</h3>
      <div style="font-size: 15px; color: #2c3e50; line-height: 1.6;">{{problem}}</div>
    </div>
    <div style="display: flex; gap: 12px; margin-bottom: 14px;">
      <div style="flex: 1; background: #fef9e7; border: 1px solid #f9e79f; border-radius: 8px; padding: 12px 16px;">
        <strong style="color: #7d6608; font-size: 13px;">Solution's final answer</strong>
        <div style="font-size: 15px; font-weight: 600; color: #1a5276;">{{final_answer}}</div>
      </div>
      <div style="flex: 1; background: #eafaf1; border: 1px solid #a9dfbf; border-radius: 8px; padding: 12px 16px;">
        <strong style="color: #196f3d; font-size: 13px;">Reference answer (from source dataset)</strong>
        <div style="font-size: 15px; font-weight: 600; color: #145a32;">{{reference_answer}}</div>
      </div>
    </div>
    <p style="color: #6b7280; font-size: 13px; margin: 0;">The solution steps are shown below as a reviewable trace. Verify each step in order and mark the earliest error, or confirm all steps are correct.</p>
  </div>

allow_all_users: true
instances_per_annotator: 30
annotation_per_instance: 3
allow_skip: true