代码评审标注

用 GitHub PR 风格的内联 diff 评论、文件级正确性评分，以及通过或打回的结论，评审 AI 编码智能体的输出，衡量代码质量。

v2.4.0 新增

评估 AI 编码智能体产出的代码改动，光靠一个通过/失败的判断远远不够。研究者和工程团队需要在多个粒度上衡量代码质量：某一行可能藏着 bug 或风格问题，某个文件可能改对了也可能根本不该改，而整套改动可能解决了问题却留下了技术债。这正是人类评审在 GitHub 上看 pull request 时走的流程。

Potato 的代码评审标注模式把 GitHub PR 评审的体验带进了智能体评估。标注者能看到智能体改动过的每个文件的 unified diff，点击任意 diff 行留下带分类标签的内联评论，为每个文件给出正确性和质量评分，最后给出结论：approve、request changes 或 comment only。全部内容都被记录为结构化的标注数据，可直接用于训练代码质量模型。

内联评论

标注者点击 diff 中的任意一行，打开内联评论表单。每条评论包含一个分类、一个严重程度和自由文本内容。评论会锚定在具体的那一行上，和 GitHub PR 评审评论一样。

评论分类

默认的评论分类覆盖了最常见的代码评审反馈类型：

分类	说明
`bug`	功能性 bug —— 代码无法正确工作
`logic`	逻辑错误 —— 语法没问题，但做法本身有缺陷
`security`	安全漏洞或不安全的做法
`performance`	性能问题 —— 多余的计算、内存泄漏等
`style`	风格问题 —— 命名、格式、是否地道
`suggestion`	有更好的替代做法
`question`	需要澄清 —— 评审者拿不准这里的意图
`praise`	正面反馈 —— 智能体做得好的地方

配置

yaml

annotation_schemes:
  - name: inline_comments
    annotation_type: code_review_comments
    description: "Click any diff line to add an inline comment"
 
    inline_comments:
      # Comment categories
      categories:
        - value: bug
          display: "Bug"
          color: "#ef4444"
          icon: "bug"
        - value: logic
          display: "Logic Error"
          color: "#f97316"
          icon: "alert-triangle"
        - value: security
          display: "Security"
          color: "#dc2626"
          icon: "shield-alert"
        - value: performance
          display: "Performance"
          color: "#eab308"
          icon: "zap"
        - value: style
          display: "Style"
          color: "#6b7280"
          icon: "palette"
        - value: suggestion
          display: "Suggestion"
          color: "#3b82f6"
          icon: "lightbulb"
        - value: question
          display: "Question"
          color: "#8b5cf6"
          icon: "help-circle"
        - value: praise
          display: "Praise"
          color: "#22c55e"
          icon: "thumbs-up"
 
      # Severity levels (optional)
      severity:
        enabled: true
        levels:
          - value: critical
            display: "Critical"
          - value: major
            display: "Major"
          - value: minor
            display: "Minor"
          - value: nit
            display: "Nit"
 
      # Behavior
      require_category: true
      require_severity: false
      allow_multi_line: true       # comments can span a range of lines
      allow_suggestions: true      # annotator can write suggested replacement code
      min_comments: 0              # minimum comments required before submission

建议的代码改动

启用 allow_suggestions 后，标注者可以为自己正在评论的代码块写一段替换建议，对应 GitHub 的 “suggestion” 功能。建议会以代码块的形式出现在评论下方，可用于训练代码修复模型。

yaml

# In inline comment output:
{
  "file": "src/parser.py",
  "line_start": 42,
  "line_end": 44,
  "category": "bug",
  "severity": "critical",
  "comment": "Off-by-one error: range should be inclusive of end",
  "suggestion": "for i in range(start, end + 1):\n    process(tokens[i])"
}

文件级评分

智能体改动过的每个文件都会得到两项彼此独立的评分：正确性和代码质量。

配置

yaml

annotation_schemes:
  - name: file_ratings
    annotation_type: code_review_file_ratings
    description: "Rate each modified file"
 
    file_ratings:
      dimensions:
        - name: correctness
          display: "Correctness"
          description: "Are the changes to this file functionally correct?"
          scale:
            min: 1
            max: 5
            labels:
              1: "Broken -- introduces bugs or breaks existing functionality"
              2: "Mostly broken -- significant functional issues"
              3: "Partially correct -- works but has edge cases or minor bugs"
              4: "Mostly correct -- minor issues only"
              5: "Fully correct -- changes work as intended"
 
        - name: quality
          display: "Code Quality"
          description: "How well-written are the changes to this file?"
          scale:
            min: 1
            max: 5
            labels:
              1: "Very poor -- unreadable, no structure"
              2: "Poor -- hard to follow, inconsistent style"
              3: "Acceptable -- works but could be cleaner"
              4: "Good -- clean, idiomatic, well-structured"
              5: "Excellent -- exemplary code, would merge as-is"
 
      # Files to rate
      include_unchanged: false     # only rate files the agent modified
      include_new_files: true      # include files the agent created
      include_deleted_files: true  # include files the agent deleted
 
      # Behavior
      require_all_files: true      # must rate every modified file

输出格式

json

{
  "file_ratings": {
    "src/parser.py": {
      "correctness": 4,
      "quality": 3
    },
    "tests/test_parser.py": {
      "correctness": 5,
      "quality": 4
    },
    "src/utils.py": {
      "correctness": 2,
      "quality": 2
    }
  }
}

总体结论

看完所有文件、留下内联评论之后，标注者为整套改动给出一个总体结论。

配置

yaml

annotation_schemes:
  - name: verdict
    annotation_type: code_review_verdict
    description: "Give an overall verdict on the code changes"
 
    verdict:
      options:
        - value: approve
          display: "Approve"
          description: "Changes are correct and ready to merge"
          color: "#22c55e"
          icon: "check-circle"
        - value: request_changes
          display: "Request Changes"
          description: "Changes need fixes before merging"
          color: "#ef4444"
          icon: "x-circle"
        - value: comment_only
          display: "Comment Only"
          description: "Leaving feedback without a verdict"
          color: "#6b7280"
          icon: "message-circle"
 
      # Optional summary text
      require_summary: true
      summary_placeholder: "Summarize your review..."
      summary_min_length: 20

配置参考

一个代码评审标注任务的完整配置：

yaml

task_name: "Coding Agent Code Review"
task_dir: "."
 
data_files:
  - "data/coding_traces.jsonl"
 
item_properties:
  id_key: id
  text_key: task_description
 
agentic:
  enabled: true
  trace_converter: claude_code
  display_type: coding_trace
 
  coding_trace_display:
    diff_style: unified
    diff_context_lines: 5
    syntax_highlight: true
    show_line_numbers: true
    terminal_theme: dark
    file_tree:
      enabled: true
      position: left
      show_operation_icons: true
      click_to_navigate: true
 
annotation_schemes:
  # Inline comments on diff lines
  - name: inline_comments
    annotation_type: code_review_comments
    inline_comments:
      categories:
        - { value: bug, display: "Bug", color: "#ef4444" }
        - { value: logic, display: "Logic Error", color: "#f97316" }
        - { value: security, display: "Security", color: "#dc2626" }
        - { value: performance, display: "Performance", color: "#eab308" }
        - { value: style, display: "Style", color: "#6b7280" }
        - { value: suggestion, display: "Suggestion", color: "#3b82f6" }
        - { value: question, display: "Question", color: "#8b5cf6" }
        - { value: praise, display: "Praise", color: "#22c55e" }
      severity:
        enabled: true
        levels:
          - { value: critical, display: "Critical" }
          - { value: major, display: "Major" }
          - { value: minor, display: "Minor" }
          - { value: nit, display: "Nit" }
      require_category: true
      allow_multi_line: true
      allow_suggestions: true
 
  # File-level correctness and quality
  - name: file_ratings
    annotation_type: code_review_file_ratings
    file_ratings:
      dimensions:
        - name: correctness
          display: "Correctness"
          scale: { min: 1, max: 5 }
        - name: quality
          display: "Code Quality"
          scale: { min: 1, max: 5 }
      require_all_files: true
 
  # Overall verdict
  - name: verdict
    annotation_type: code_review_verdict
    verdict:
      options:
        - { value: approve, display: "Approve", color: "#22c55e" }
        - { value: request_changes, display: "Request Changes", color: "#ef4444" }
        - { value: comment_only, display: "Comment Only", color: "#6b7280" }
      require_summary: true
      summary_min_length: 20
 
output_annotation_dir: "output/"
output_annotation_format: "jsonl"

标注流程

标注者在完成一个代码评审标注任务时，看到和要做的是这些：

任务概览：任务描述显示在最上方，说明智能体被要求做什么（比如“修复 test_parser.py 中失败的测试”）。
文件树导航：左侧栏列出智能体动过的所有文件，并用颜色区分：绿色是新增文件，黄色是修改的文件，红色是删除的文件。
审阅 diff：主面板按文件显示 unified diff。标注者逐段翻看，读每一处改动。
添加内联评论：点击行号会打开评论表单。标注者选择一个分类（bug、建议等），可选地选择严重程度，写下评论，也可以附上一段代码建议。
文件评分：看完某个文件的 diff 之后，标注者用该文件 diff 下方的评分组件，从正确性（1-5）和代码质量（1-5）两方面打分。
总体结论：在页面底部，标注者选择一个结论（通过、要求修改，或只留评论），并写一段评审总结。
提交：标注者点击 “Submit”，把全部内联评论、文件评分和结论作为一条标注记录保存。

数据格式

一次代码评审标注的完整输出：

json

{
  "id": "trace_042",
  "annotator": "reviewer_01",
  "timestamp": "2025-01-15T14:30:00Z",
  "annotations": {
    "inline_comments": [
      {
        "file": "src/parser.py",
        "line_start": 42,
        "line_end": 42,
        "category": "bug",
        "severity": "critical",
        "comment": "This will throw IndexError when tokens list is empty",
        "suggestion": "if tokens:\n    return tokens[0]\nreturn None"
      },
      {
        "file": "src/parser.py",
        "line_start": 15,
        "line_end": 15,
        "category": "style",
        "severity": "nit",
        "comment": "Variable name 'x' is not descriptive"
      },
      {
        "file": "tests/test_parser.py",
        "line_start": 28,
        "line_end": 30,
        "category": "praise",
        "comment": "Good edge case coverage for empty input"
      }
    ],
    "file_ratings": {
      "src/parser.py": { "correctness": 3, "quality": 2 },
      "tests/test_parser.py": { "correctness": 5, "quality": 4 }
    },
    "verdict": {
      "decision": "request_changes",
      "summary": "The core fix is on the right track but has an edge case bug with empty input. The test coverage is good. Fix the IndexError and clean up variable naming."
    }
  }
}

导出

代码评审标注可以导出为多种格式：

bash

# Export as structured code review JSON
python -m potato.export \
  -i output/ \
  -f code_review \
  -o results/reviews.jsonl
 
# Export inline comments only (for training code comment models)
python -m potato.export \
  -i output/ \
  -f code_review_comments \
  -o results/comments.jsonl
 
# Export file ratings as a CSV (for analysis)
python -m potato.export \
  -i output/ \
  -f code_review_file_ratings \
  -o results/file_ratings.csv
 
# Export verdict distribution summary
python -m potato.export \
  -i output/ \
  -f code_review_verdicts \
  -o results/verdicts.json

code_review_comments 格式尤其适合训练那些生成代码评审评论、或者预测代码问题位置与类别的模型。

另请参阅

编码智能体标注 —— 展示编码智能体 trace，带 diff 渲染和文件树
过程奖励标注 —— 为 PRM 训练收集逐步奖励信号
实时编码智能体观察 —— 实时观察编码智能体并与之交互
智能体标注 —— 通用的智能体 trace 标注
导出格式 —— 所有支持的导出格式

有关实现详情，请参阅源文档。