Process Reward Annotation

Name: Potato
Author: Potato

Collect per-step reward labels for process reward models with first-error and per-step modes, optional neutral labels, and LLM pre-labeling with human verification.

New in v2.4.0

Process reward models score the reasoning steps a model takes rather than only its final answer, so training one needs a label per step instead of one label per trace. The process_reward schema collects those labels.

Two modes trade speed against detail. First-error mode asks for one click: the first step that goes wrong, with everything after it treated as compromised. Per-step mode asks for a judgment on every step, which costs more and captures cases first-error cannot, such as an agent recovering from its own mistake.

Both work with the coding trace, agent trace, and chain-of-thought displays.

First-Error Mode

The annotator reads down the trace and clicks the first incorrect step. Steps before it are recorded as correct, and the clicked step and everything after it as incorrect. That produces the label sequence binary PRM training expects: 1 up to the error, then -1 from the error onward.

yaml

annotation_schemes:
  - annotation_type: process_reward
    name: step_rewards
    description: "Click the first step where the agent made a mistake"
    mode: first_error
    steps_key: structured_turns
    step_text_key: content

steps_key names the field holding the step list, and step_text_key names the field within each step to display.

Annotation Workflow

Steps start unmarked
The annotator clicks the first incorrect step
Everything before it turns green
That step is marked as the first error
Everything after it is marked as compromised

The colors make the cascade visible, which is the point of the mode: one judgment labels the whole trace.

Per-Step Mode

Every step is judged on its own. Add allow_neutral for the PRM800K-style three-way label, where a step can be neither right nor wrong:

yaml

annotation_schemes:
  - annotation_type: process_reward
    name: step_rewards
    description: "Rate each step correct, neutral, or incorrect"
    mode: per_step
    steps_key: structured_turns
    step_text_key: content
    allow_neutral: true
    inline_with_trace: true

allow_neutral applies in per-step mode only. The first-error cascade has no position for a neutral step, so the option is ignored there.

inline_with_trace: true attaches the rating controls to each step in the display instead of collecting them in a separate panel, which is what you want on a long trace where the annotator would otherwise lose their place.

Renaming the Labels

The three buttons read Correct, Neutral, and Wrong by default. reward_labels renames them for domains where those words are wrong:

yaml

annotation_schemes:
  - annotation_type: process_reward
    name: step_rewards
    description: "Rate each reasoning step"
    mode: per_step
    steps_key: cot_steps
    allow_neutral: true
    reward_labels:
      correct: "Valid"
      neutral: "Neutral"
      incorrect: "Flawed"

Segmenting a Chain of Thought

Reasoning traces usually arrive as one long string rather than a step list. cot_segmentation splits it before annotation begins:

yaml

cot_segmentation:
  source_key: reasoning      # the item field holding the long CoT string
  strategy: auto             # blank_line | numbered | markers | sentence | llm | auto
  target_key: cot_steps      # where the step list is written
  min_step_chars: 30         # merge anything shorter into the previous step
  max_steps: 200
 
instance_display:
  fields:
    - key: cot_steps
      type: cot_trace
      label: "Reasoning"
 
annotation_schemes:
  - annotation_type: process_reward
    name: step_rewards
    description: "Rate each reasoning step"
    mode: per_step
    steps_key: cot_steps
    allow_neutral: true
    inline_with_trace: true

min_step_chars is worth setting. Without it, sentence and blank-line segmentation produce one-line fragments that annotators cannot judge in isolation, and the label you get back is noise.

LLM Pre-Labeling

A model can propose the per-step labels and the annotator confirms or overrides each one, which is faster than labeling from scratch on long traces:

yaml

annotation_schemes:
  - annotation_type: process_reward
    name: step_rewards
    description: "Verify or correct each suggested reward"
    mode: per_step
    steps_key: cot_steps
    allow_neutral: true
    inline_with_trace: true
    ai_prelabel: true
    require_verification: true
 
ai_support:
  enabled: true
  endpoint_type: openai
  ai_config:
    model: gpt-4o-mini
    max_tokens: 768
    temperature: 0.1
    include:
      all: true

ai_prelabel adds the button that requests suggestions. require_verification refuses to accept the annotation until every suggestion has been confirmed or changed, which is what keeps the output a human label rather than a model's.

Give max_tokens real headroom. A long chain of thought plus a verdict per step is a large reply, and a truncated one renders as no suggestions at all.

What the Output Contains

The annotation is stored under the scheme's name as a steps list plus the mode it was collected in:

json

{
  "step_rewards": {
    "mode": "per_step",
    "steps": [
      {"reward": 1, "source": "human", "verified": true},
      {"reward": -1, "source": "human", "verified": true},
      {"reward": 0, "source": "ai", "verified": true, "ai_reward": 0},
      {"reward": null, "source": null, "verified": false}
    ]
  }
}

reward is 1 for correct, -1 for incorrect, and 0 for neutral. A skipped step is null rather than 0, so a step nobody judged stays distinguishable from one judged as neither right nor wrong. That distinction matters when the labels become training data, since treating skips as neutral silently invents signal.

source records whether the value came from a person or a model, and verified whether a person confirmed it. With ai_prelabel on, ai_reward keeps the model's original suggestion even after an annotator overrides it, which is what lets you report how often the model was right without a second pass.

Exporting

Potato has no built-in PRM or DPO export format. Export the annotations and shape them yourself:

bash

python -m potato.export -c config.yaml -f jsonl -o ./export/

Converting first-error annotations into step-level training labels is short, and writing it yourself is what lets you decide how skipped steps are handled:

python

import json
from pathlib import Path
 
SCHEME = "step_rewards"
 
examples = []
for f in Path("export/").rglob("*.jsonl"):
    with open(f) as fh:
        for line in fh:
            rec = json.loads(line)
            payload = (rec.get("labels", rec) or {}).get(SCHEME)
            if not payload:
                continue
 
            steps = payload.get("steps", [])
            # Drop traces with unjudged steps rather than guessing at them.
            if any(s.get("reward") is None for s in steps):
                continue
 
            item = items_by_id[rec["instance_id"]]      # your own data file
            examples.append({
                "trace_id": rec["instance_id"],
                "steps": [
                    {"content": src, "label": s["reward"]}
                    for src, s in zip(item["structured_turns"], steps)
                ],
            })
 
with open("prm_training_data.jsonl", "w") as out:
    for ex in examples:
        out.write(json.dumps(ex) + "\n")
 
print(f"Wrote {len(examples)} traces")

Analysis

python

import json
from collections import Counter
from pathlib import Path
 
SCHEME = "step_rewards"
 
traces = []
for f in Path("export/").rglob("*.jsonl"):
    with open(f) as fh:
        for line in fh:
            rec = json.loads(line)
            payload = (rec.get("labels", rec) or {}).get(SCHEME)
            if payload:
                traces.append(payload)
 
print(f"Traces annotated: {len(traces)}")
 
all_correct = sum(
    1 for t in traces
    if all(s.get("reward") == 1 for s in t.get("steps", []))
)
print(f"Traces with no errors: {all_correct} ({all_correct / len(traces):.1%})")
 
# Where the first error lands
first_error_positions = []
for t in traces:
    for i, s in enumerate(t.get("steps", [])):
        if s.get("reward") == -1:
            first_error_positions.append(i)
            break
 
if first_error_positions:
    avg = sum(first_error_positions) / len(first_error_positions)
    print(f"Average first-error position: step {avg:.1f}")
    print("By position:", Counter(first_error_positions).most_common(10))

When ai_prelabel is on, the same records tell you how often the model's suggestion survived review:

python

kept = changed = 0
for t in traces:
    for s in t.get("steps", []):
        if s.get("ai_reward") is None:
            continue
        if s.get("reward") == s.get("ai_reward"):
            kept += 1
        else:
            changed += 1
 
total = kept + changed
if total:
    print(f"AI suggestions kept: {kept}/{total} ({kept / total:.1%})")

That acceptance rate is the number to watch before trusting pre-labeling on a new domain. A rate near 100% usually means annotators are confirming rather than checking, not that the model is right.

Research Context

Process reward annotation exists to support work on reward models for multi-step systems, where the finding that motivates the whole approach is that a correct final answer can sit on top of faulty reasoning. Step-level labels separate the two, and the first-error and per-step modes here map onto the label formats that line of work uses.

For the design of the taxonomy and the agreement questions that come with it, see per-step error localization, which covers the same traces from the error-classification side.

Process Reward Annotation

First-Error Mode

Annotation Workflow

Per-Step Mode

Renaming the Labels

Segmenting a Chain of Thought

LLM Pre-Labeling

What the Output Contains

Exporting

Analysis

Research Context

See Also