Skip to content
advancedpreference

How to annotate CodePRM Code Process Reward

Process reward annotation for step-by-step code generation with execution feedback. Annotators verify each incremental code generation step against problem requirements, check syntactic validity, and provide feedback informed by test execution results.

Try it live — no install

Boot the real Potato server in your browser (WebAssembly) and annotate with this exact config. Nothing leaves your machine.

Run live in your browser

Get This Design

View on GitHub(opens in a new tab)

Clone or download from the repository

Quick start:

bash
git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/agentic/codeprm-code-process-reward
potato start config.yaml

Configuration File

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# CodePRM Code Process Reward
# Based on "CodePRM: Execution Feedback-enhanced Process Reward Model for Code Generation" (Li et al., ACL 2025 Findings)

annotation_task_name: "CodePRM Code Process Reward"
task_dir: "."
data_files:
  - sample-data.json
item_properties:
  id_key: "id"
  text_key: "text"
output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

annotation_schemes:
  - name: "step_correctness"
    annotation_type: radio
    description: "Verify each code generation step against the problem requirements."
    labels:
      - name: "Correct — code step is valid"
        key_value: "1"
      - name: "Neutral — step doesn't affect outcome"
        key_value: "2"
      - name: "Incorrect — step introduces a bug"
        key_value: "3"

  - name: "code_compiles"
    annotation_type: radio
    description: "Is the code in this step syntactically valid?"
    labels:
      - name: "Yes — code is syntactically valid"
        key_value: "4"
      - name: "No — syntax error present"
        key_value: "5"
      - name: "N/A — not a code step"
        key_value: "6"

  - name: "step_feedback"
    annotation_type: text
    description: "Provide feedback on the current step."

user_config:
  allow_all_users: true
max_annotations_per_user: 50
num_annotators_per_item: 2

Sample Data

json
[
  {
    "id": "codeprm-001",
    "text": "Write a function `merge_sorted(a, b)` that merges two sorted lists into one sorted list without using built-in sort functions.",
    "steps": "Step 1: Define the function signature and initialize pointers.\n  def merge_sorted(a, b):\n      result = []\n      i, j = 0, 0\n\nStep 2: Add the main merge loop comparing elements from both lists.\n      while i < len(a) and j < len(b):\n          if a[i] <= b[j]:\n              result.append(a[i])\n              i += 1\n          else:\n              result.append(b[j])\n              j += 1\n\nStep 3: Append remaining elements from both lists.\n      result.extend(a[i:])\n      result.extend(b[j:])\n      return result",
    "test_output": "test_merge_empty_lists .............. PASS\ntest_merge_single_elements .......... PASS\ntest_merge_different_lengths ........ PASS\ntest_merge_duplicates ............... PASS\n\n4 passed, 0 failed"
  },
  {
    "id": "codeprm-002",
    "text": "Implement a function `is_balanced(s)` that checks whether a string of brackets '()[]{}' is balanced.",
    "steps": "Step 1: Define the function and the bracket mappings.\n  def is_balanced(s):\n      stack = []\n      mapping = {')': '(', ']': '[', '}': '{'}\n\nStep 2: Iterate through each character and push opening brackets.\n      for char in s:\n          if char in mapping.values():\n              stack.append(char)\n\nStep 3: For closing brackets, check the stack.\n          elif char in mapping:\n              if not stack or stack.pop() != mapping[char]:\n                  return False\n\nStep 4: Return whether the stack is empty.\n      return len(stack) == 0",
    "test_output": "test_empty_string ................... PASS\ntest_simple_balanced ................ PASS\ntest_nested_balanced ................ PASS\ntest_unbalanced_open ................ PASS\ntest_unbalanced_close ............... PASS\ntest_mixed_balanced ................. PASS\n\n6 passed, 0 failed"
  }
]

// ... and 6 more items

Dataset & paper

Li et al., ACL 2025 Findings

Citation (BibTeX)

bibtex
@inproceedings{li2025codeprm, title={CodePRM: Execution Feedback-enhanced Process Reward Model for Code Generation}, author={Li, Qingyao and Dai, Xinyi and Li, Xiangyang and Zhang, Weinan and Wang, Yasheng and Tang, Ruiming and Yu, Yong}, booktitle={Findings of the Association for Computational Linguistics: ACL 2025}, year={2025}}

Details

Annotation Types

radiotext

Domain

Code GenerationProgramming

Use Cases

Process Reward ModelsCode VerificationRLHF

Tags

process-reward-modelcode-generationexecution-feedbackprogramming

Found an issue or want to improve this design?

Open an Issue(opens in a new tab)