Code Review Annotation

New in v2.4.0

Evaluating code changes produced by AI coding agents requires more than a binary pass/fail judgment. Researchers and engineering teams need to assess code quality at multiple granularities: individual lines may contain bugs or style violations, entire files may be correctly modified or unnecessary, and the overall change set may solve the problem but introduce technical debt. This is the same workflow that human code reviewers follow when reviewing pull requests on GitHub.

Potato's code review annotation mode brings the GitHub PR review experience to agent evaluation. Annotators see unified diffs for every file the agent modified. They can click any diff line to leave an inline comment with a category tag. Each file gets a correctness and quality rating. The annotator gives a final verdict: approve, request changes, or comment only. All of this is captured in structured annotation data ready for training code quality models.

Inline Comments

Annotators click any line in a diff to open an inline comment form. Each comment carries a category, the comment text, and optionally the file path and line it refers to. The comment appears anchored to that line, like a GitHub PR review comment.

Comment Categories

The default comment categories cover the most common code review feedback types:

Configuration

annotation_schemes:
  - annotation_type: code_review
    name: review
    description: "Click any diff line to add an inline comment"
 
    # Categories offered on each inline comment
    comment_categories:
      - bug
      - logic
      - security
      - performance
      - style
      - suggestion
      - question

What a comment records

Four fields, of which two are optional:

{
  "category": "bug",
  "file": "src/parser.py",
  "line": 42,
  "text": "Off-by-one error: range should be inclusive of end"
}

The file path and line are optional, so a comment can be about the change as a whole rather than one line. There is no severity field and no GitHub-style suggested replacement: a comment proposing different code says so in its text, under the suggestion category.

File-Level Ratings

Each file modified by the agent receives two independent ratings: correctness and code quality.

Configuration

annotation_schemes:
  - annotation_type: code_review
    name: review
    description: "Rate each modified file"
 
    # One 1-5 rating per dimension, per file touched by the diff
    file_rating_dimensions:
      - correctness
      - quality

Output Format

{
  "file_ratings": {
    "src/parser.py": {
      "correctness": 4,
      "quality": 3
    },
    "tests/test_parser.py": {
      "correctness": 5,
      "quality": 4
    },
    "src/utils.py": {
      "correctness": 2,
      "quality": 2
    }
  }
}

Overall Verdict

After reviewing all files and leaving inline comments, the annotator gives an overall verdict for the entire change set.

Configuration

annotation_schemes:
  - annotation_type: code_review
    name: review
    description: "Give an overall verdict on the code changes"
 
    verdict_options:
      - approve
      - request_changes
      - comment_only

Configuration Reference

Here is a complete configuration for a code review annotation task:

annotation_task_name: "Coding Agent Code Review"
task_dir: "."
 
data_files:
  - "data/coding_traces.jsonl"
 
item_properties:
  id_key: id
  text_key: task_description
 
instance_display:
  fields:
    - key: structured_turns
      type: coding_trace
      label: "Agent changes"
      display_options:
        diff_view: unified
        terminal_theme: dark
        collapse_long_outputs: true
        max_output_lines: 50
        show_file_tree: true
        show_step_numbers: true
        show_reasoning: true
 
annotation_schemes:
  # Inline comments, file ratings and the overall verdict are all one scheme
  - annotation_type: code_review
    name: review
    description: "Review the agent's code changes"
    comment_categories:
      - bug
      - logic
      - security
      - performance
      - style
      - suggestion
      - question
      - praise
    file_rating_dimensions:
      - correctness
      - quality
    verdict_options:
      - approve
      - request_changes
      - comment_only
 
  # A free-text summary is a separate scheme
  - annotation_type: text
    name: summary
    description: "Summarize your review"
    rows: 4
 
output_annotation_dir: "output/"
export_annotation_format: "jsonl"

The Annotation Workflow

Here is what annotators see and do when completing a code review annotation task:

1. Task overview: The task description appears at the top, showing what the agent was asked to do (e.g., "Fix the failing test in test_parser.py").

2. File tree navigation: The left sidebar shows all files the agent touched. Files are color-coded: green for new files, yellow for modified files, red for deleted files.

3. Diff review: The main panel shows unified diffs for each file. Annotators scroll through the diffs, reading each change.

4. Adding inline comments: Clicking a line number opens a comment form. The annotator picks a category (bug, style, suggestion, security, question), writes the comment, and optionally records the file path and line it applies to.

5. File ratings: After reviewing each file's diff, the annotator rates it on correctness (1-5) and code quality (1-5) using the rating widgets below each file's diff.

6. Overall verdict: At the bottom, the annotator selects a verdict (approve, request changes, or comment only) and writes a summary of their review.

7. Submission: The annotator clicks "Submit" to save all inline comments, file ratings, and the verdict as a single annotation record.

Data Format

The complete output for a single code review annotation:

{
  "instance_id": "trace_042",
  "annotator": "reviewer_01",
  "verdict": "request_changes",
  "comments": [
    {
      "category": "bug",
      "file": "src/parser.py",
      "line": 42,
      "text": "This will throw IndexError when tokens list is empty"
    },
    {
      "category": "style",
      "file": "src/parser.py",
      "line": 15,
      "text": "Variable name 'x' is not descriptive"
    },
    {
      "category": "praise",
      "file": "tests/test_parser.py",
      "line": 28,
      "text": "Good edge case coverage for empty input"
    }
  ],
  "file_ratings": {
    "src/parser.py": { "correctness": 3, "quality": 2 },
    "tests/test_parser.py": { "correctness": 5, "quality": 4 }
  }
}

verdict is one of the verdict_options strings, not an object. The written summary is a separate text scheme, so it arrives under its own scheme name rather than inside the review.

Export

Code review annotations come out through the coding_eval exporter, which takes the config rather than the output directory and writes into a directory you name:

python -m potato.export \
  -c config.yaml \
  -f coding_eval \
  -o results/ \
  --option types=code_review

types selects what gets written. It accepts prm, preference, swebench and code_review, and defaults to all four, so drop the --option to get the rest alongside the reviews. The exporter says so when a requested type found no data in the study, rather than writing an empty file.

Each record carries the verdict, the inline comments with their category and text, and the per-file ratings, so the comment-level training data and the rating analysis both come from the one file.

See Also

• Coding Agent Annotation -- display coding agent traces with diff rendering and file trees
• Process Reward Annotation -- per-step reward signals for PRM training
• Live Coding Agent Observation -- observe and interact with coding agents in real time
• Agentic Annotation -- general-purpose agent trace annotation
• Export Formats -- all supported export formats

For implementation details, see the source documentation.