Skip to content
Guides8 min read

并排比较 AI 智能体：二元、量表与多维度三种模式

在 Potato 中搭建成对智能体比较，共三种模式：二元偏好、连续量表，以及需要填写理由的分维度多准则判断。

Potato Team

智能体评估为什么要用成对比较

让人给一条编码智能体 trace 打 1 到 10 分，拿到的数据噪声很大，因为每个人对这把尺子的校准都不一样。这个标注者的 7 分，可能就是另一个人的 5 分。成对比较绕开了这个问题：标注者不再孤立地给单条 trace 打分，而是并排看两条，说出哪一条更好。这种一对一的判断更容易做出，人与人之间也更一致，而且恰好就是直接偏好优化（DPO）和基于人类反馈的强化学习（RLHF）所需要的数据形式。

训练语言模型对齐用的奖励模型走的是同一条路，这套做法搬到编码智能体上也很顺：在成对的智能体轨迹之间收集人类偏好，用它们训练奖励模型，再用这个模型指导智能体训练，或者在推理时从 N 个候选里挑最好的那个。

Potato 提供三种成对比较模式，分别对应不同的评估需求和数据预算。

界面把两条 trace 并排放在一起：

Side-by-side agent comparison interfaceAnnotators compare two agent traces and select which approach was better

模式 1：二元偏好

这是最简单也最快的模式。标注者并排看到两条 trace，点击更好的那条。还有一个可选的平局按钮，用于两者一样好或一样差的情况。

什么时候用二元模式

需要在短时间内攒出大量偏好数据时，就选二元模式。它适合训练基础的奖励模型、算智能体胜率、搭 Elo 排行榜。代价是丢掉了细节：你知道哪条 trace 赢了，但不知道赢了多少，也不知道赢在哪些方面。

配置

yaml
# config.yaml
project_name: "Agent Comparison - Binary"
port: 8000
 
data:
  source: "local"
  input_path: "./data/paired_traces.jsonl"
  data_format: "paired_coding_trace"
 
coding_agent:
  display:
    diff_style: "unified"
    syntax_highlighting: true
    terminal_theme: "dark"
    file_tree:
      enabled: true
      position: "left"
    collapsible:
      auto_collapse_thinking: true
 
comparison:
  layout: "side_by_side"         # "side_by_side" or "tabbed"
  label_a: "Agent A"
  label_b: "Agent B"
  randomize_order: true          # Randomize which trace appears on which side
  show_agent_identity: false     # Hide agent names to avoid bias
  sync_scroll: false             # Independent scrolling for each trace
 
annotation_schemes:
  - annotation_type: pairwise
    name: preference
    description: "Which agent produced a better solution?"
    items_key:
      - value: "a"
        text: "Agent A is better"
        keyboard_shortcut: "1"
      - value: "b"
        text: "Agent B is better"
        keyboard_shortcut: "2"
      - value: "tie"
        text: "Tie (equally good or equally bad)"
        keyboard_shortcut: "3"
    allow_tie: true
  - annotation_type: radio
    name: confidence
    labels:
      - value: "high"
        text: "Very confident"
      - value: "medium"
        text: "Somewhat confident"
      - value: "low"
        text: "Not confident"
 
output:
  path: "./output/"
  format: "jsonl"
 
quality_control:
  inter_annotator_agreement: true
  overlap_percentage: 20
  attention_checks:
 
annotators:
  - username: "judge1"
  - username: "judge2"

标注流程

标注者看到的是分屏界面。左边是 Trace A，用完整的 CodingTraceDisplay 渲染：diff、终端块、文件读取、思考过程。右边是同一任务的 Trace B。两侧各自独立滚动。

任务描述放在两条 trace 上方，这样标注者知道两个智能体本来要做什么。

下方是三个按钮：“Agent A is better”“Agent B is better”和“Tie”。打开 randomize_order 后，哪个智能体是 A、哪个是 B 会逐条打乱，标注者就不会养成偏左或偏右的习惯。

如果需要更细的评估，界面也支持多个维度：

Pairwise preference selection interfaceBinary preference, continuous scale, and multi-dimension modes are available

模式 2：连续量表

量表模式让标注者说出一条 trace 好多少，而不只是哪条赢了。他们不是点一下，而是拖动一根滑块，从左端的“A 好得多”到右端的“B 好得多”，中间是“持平”。

什么时候用量表模式

当偏好的强度而不只是方向也重要时，用量表模式。滑块靠近端点意味着质量差距明显，靠近中间则说明两者接近。DPO 及类似流程可以按这个强度给样本加权，让区分明确的样本发挥更大作用。

配置

yaml
# config.yaml
project_name: "Agent Comparison - Scale"
port: 8000
 
data:
  source: "local"
  input_path: "./data/paired_traces.jsonl"
  data_format: "paired_coding_trace"
 
coding_agent:
  display:
    diff_style: "unified"
    syntax_highlighting: true
    terminal_theme: "dark"
    file_tree:
      enabled: true
    collapsible:
      auto_collapse_thinking: true
 
comparison:
  layout: "side_by_side"
  randomize_order: true
  show_agent_identity: false
 
annotation_schemes:
  - annotation_type: pairwise
    name: preference_scale
    description: "Which agent produced a better solution, and by how much?"
    scale:
      points: 7                  # 7-point scale
      labels:
        1: "A is much better"
        2: "A is better"
        3: "A is slightly better"
        4: "Equal"
        5: "B is slightly better"
        6: "B is better"
        7: "B is much better"
      default: 4                 # Start at "Equal"
      show_numeric_value: true
 
output:
  path: "./output/"
  format: "jsonl"
 
quality_control:
  inter_annotator_agreement: true
  overlap_percentage: 20
 
annotators:
  - username: "judge1"
  - username: "judge2"

改用 5 档量表

想标得更快、粒度稍粗一些，可以降到 5 档：

yaml
annotation_schemes:
  - annotation_type: pairwise
    name: preference_scale_5
    description: "Compare the two solutions"
    scale:
      points: 5
      labels:
        1: "A is clearly better"
        2: "A is somewhat better"
        3: "About equal"
        4: "B is somewhat better"
        5: "B is clearly better"
      default: 3

模式 3：多维度比较

这是最细的模式。标注者不给一个总体偏好，而是在若干个相互独立的维度上分别判断两条 trace。每个维度各自给出 A/B/平局，并且每次判断都要写理由。

什么时候用多维度模式

当你不仅想知道哪个智能体赢了，还想知道为什么赢时，用这个模式。一条 trace 可能代码正确但效率很差，另一条可能高效却漏掉了一个边界情况。由此得到的分维度数据既可以训练特定维度的奖励模型，也可以作为详细反馈交给做智能体的人。

配置

yaml
# config.yaml
project_name: "Agent Comparison - Multi-Dimension"
port: 8000
 
data:
  source: "local"
  input_path: "./data/paired_traces.jsonl"
  data_format: "paired_coding_trace"
 
coding_agent:
  display:
    diff_style: "unified"
    syntax_highlighting: true
    terminal_theme: "dark"
    file_tree:
      enabled: true
    collapsible:
      auto_collapse_thinking: true
 
comparison:
  layout: "side_by_side"
  randomize_order: true
  show_agent_identity: false
 
annotation_schemes:
  - annotation_type: pairwise
    name: multi_dim_comparison
    description: "Compare the two solutions along each dimension"
 
      - name: "efficiency"
        label: "Efficiency"
        description: >
          How efficient is the agent's process? Does it take unnecessary
          steps, read irrelevant files, or make redundant edits?
        options: ["A", "B", "Tie"]
        require_justification: true
        justification_placeholder: "Which agent was more efficient and why?"
        weight: 0.2
 
      - name: "code_quality"
        label: "Code Quality"
        description: >
          Is the code well-written? Consider readability, naming,
          error handling, documentation, and adherence to existing patterns.
        options: ["A", "B", "Tie"]
        require_justification: true
        justification_placeholder: "Which produces better quality code?"
        weight: 0.2
 
      - name: "communication"
        label: "Communication"
        description: >
          How well does the agent explain its reasoning? Are its thinking
          steps clear and logical? Does it identify the root cause?
        options: ["A", "B", "Tie"]
        require_justification: true
        justification_placeholder: "Which agent communicates its approach better?"
        weight: 0.1
 
      - name: "robustness"
        label: "Robustness"
        description: >
          Does the solution handle edge cases? Does the agent verify its
          changes with tests? Is the fix narrow and targeted or fragile?
        options: ["A", "B", "Tie"]
        require_justification: true
        justification_placeholder: "Which solution is more robust?"
        weight: 0.1
 
 
output:
  path: "./output/"
  format: "jsonl"
 
quality_control:
  inter_annotator_agreement: true
  overlap_percentage: 25         # Higher overlap for this detailed task
  minimum_time_per_instance: 120 # 2 minutes minimum for thorough review
 
annotators:
  - username: "judge1"
  - username: "judge2"

准备成对 trace 数据

三种模式的输入都是成对的 trace。JSONL 文件的每一行包含两条针对同一任务的 trace。

数据格式

json
{
  "id": "pair_001",
  "task_description": "Fix the IndexError in process_batch() when the input list is empty",
  "repo": "myorg/myproject",
  "trace_a": {
    "agent": "claude_code",
    "model": "claude-sonnet-4-20250514",
    "structured_turns": [
      {
        "step_idx": 0,
        "type": "file_read",
        "path": "src/batch.py",
        "content": "def process_batch(items):\n    result = items[0]\n    ...",
        "start_line": 10,
        "end_line": 25
      },
      {
        "step_idx": 1,
        "type": "file_edit",
        "path": "src/batch.py",
        "diff": "--- a/src/batch.py\n+++ b/src/batch.py\n@@ -10,3 +10,5 @@\n def process_batch(items):\n+    if not items:\n+        return []\n     result = items[0]\n"
      },
      {
        "step_idx": 2,
        "type": "bash_command",
        "command": "python -m pytest tests/test_batch.py -v",
        "output": "PASSED",
        "exit_code": 0
      }
    ]
  },
  "trace_b": {
    "agent": "swe_agent",
    "model": "gpt-4o",
    "structured_turns": [
      {
        "step_idx": 0,
        "type": "bash_command",
        "command": "find . -name '*.py' | xargs grep 'process_batch'",
        "output": "src/batch.py:def process_batch(items):\ntests/test_batch.py:    process_batch([])",
        "exit_code": 0
      },
      {
        "step_idx": 1,
        "type": "file_read",
        "path": "src/batch.py",
        "content": "def process_batch(items):\n    result = items[0]\n    ...",
        "start_line": 1,
        "end_line": 50
      },
      {
        "step_idx": 2,
        "type": "file_edit",
        "path": "src/batch.py",
        "diff": "--- a/src/batch.py\n+++ b/src/batch.py\n@@ -10,3 +10,6 @@\n def process_batch(items):\n+    if items is None or len(items) == 0:\n+        logger.warning('Empty input to process_batch')\n+        return []\n     result = items[0]\n"
      },
      {
        "step_idx": 3,
        "type": "bash_command",
        "command": "python -m pytest tests/ -v",
        "output": "PASSED (12 tests)",
        "exit_code": 0
      }
    ]
  }
}

从单条 trace 组装成对数据

如果你手上是一批针对同样任务的单条 trace，配对工具可以帮你组装：

bash
# Generate all possible pairs for each task
potato pair-traces \
  --input ./data/individual_traces.jsonl \
  --output ./data/paired_traces.jsonl \
  --pair_by "task_id" \
  --strategy "all_pairs"
 
# Or sample a fixed number of pairs per task
potato pair-traces \
  --input ./data/individual_traces.jsonl \
  --output ./data/paired_traces.jsonl \
  --pair_by "task_id" \
  --strategy "sample" \
  --pairs_per_task 3

导出比较数据

DPO/RLHF 偏好对

成对比较的主要导出格式，是用于 DPO 或 RLHF 训练的偏好对：

bash
potato export \
  --format dpo_preferences \
  --project ./output/ \
  --output ./training_data/preferences.jsonl

二元模式的输出很简单：

json
{
  "prompt": "Fix the IndexError in process_batch() when the input list is empty",
  "chosen": {"agent": "claude_code", "trace_id": "trace_a_001", "steps": [...]},
  "rejected": {"agent": "swe_agent", "trace_id": "trace_b_001", "steps": [...]},
  "annotator": "judge1",
  "confidence": "high"
}

量表模式多带一个偏好强度：

json
{
  "prompt": "Fix the IndexError in process_batch()",
  "chosen": {"agent": "claude_code", "trace_id": "trace_a_001"},
  "rejected": {"agent": "swe_agent", "trace_id": "trace_b_001"},
  "preference_strength": 0.83,
  "scale_value": 2,
  "justification": "Agent A found and fixed the bug in fewer steps with cleaner code"
}

多维度模式则带上每个维度的偏好：

json
{
  "prompt": "Fix the IndexError in process_batch()",
  "chosen": {"agent": "claude_code", "trace_id": "trace_a_001"},
  "rejected": {"agent": "swe_agent", "trace_id": "trace_b_001"},
  "overall_preference": "A",
  "dimensions": {
    "correctness": {"preference": "Tie", "justification": "Both correctly fix the bug"},
    "efficiency": {"preference": "A", "justification": "A solves it in 3 steps vs 4"},
    "code_quality": {"preference": "B", "justification": "B adds logging and handles None"},
    "communication": {"preference": "A", "justification": "A's reasoning is more focused"},
    "robustness": {"preference": "B", "justification": "B runs full test suite, not just one file"}
  },
  "weighted_score_a": 0.55,
  "weighted_score_b": 0.45
}

分析：胜率、Elo 评分和分维度拆解

计算胜率

python
import json
from collections import defaultdict
 
with open("training_data/preferences.jsonl") as f:
    prefs = [json.loads(line) for line in f]
 
wins = defaultdict(lambda: {"wins": 0, "losses": 0, "ties": 0})
 
for pref in prefs:
    agent_chosen = pref["chosen"]["agent"]
    agent_rejected = pref["rejected"]["agent"]
 
    if agent_chosen == agent_rejected:
        continue  # Skip self-comparisons
 
    if pref.get("overall_preference") == "Tie":
        wins[agent_chosen]["ties"] += 1
        wins[agent_rejected]["ties"] += 1
    else:
        wins[agent_chosen]["wins"] += 1
        wins[agent_rejected]["losses"] += 1
 
print("Agent Win Rates:")
print("-" * 55)
for agent, record in sorted(wins.items()):
    total = record["wins"] + record["losses"] + record["ties"]
    win_rate = (record["wins"] + 0.5 * record["ties"]) / total * 100
    print(f"  {agent:<20} {win_rate:5.1f}%  "
          f"(W:{record['wins']} L:{record['losses']} T:{record['ties']})")

计算 Elo 评分

python
import json
import math
from collections import defaultdict
 
def compute_elo(preferences, k=32, initial_rating=1500):
    """Compute Elo ratings from pairwise preferences."""
    ratings = defaultdict(lambda: initial_rating)
 
    for pref in preferences:
        agent_a = pref["chosen"]["agent"]
        agent_b = pref["rejected"]["agent"]
 
        ra = ratings[agent_a]
        rb = ratings[agent_b]
 
        # Expected scores
        ea = 1.0 / (1.0 + math.pow(10, (rb - ra) / 400))
        eb = 1.0 / (1.0 + math.pow(10, (ra - rb) / 400))
 
        overall = pref.get("overall_preference", "A")
        if overall == "Tie":
            sa, sb = 0.5, 0.5
        else:
            # "chosen" is the winner
            sa, sb = 1.0, 0.0
 
        ratings[agent_a] = ra + k * (sa - ea)
        ratings[agent_b] = rb + k * (sb - eb)
 
    return dict(ratings)
 
with open("training_data/preferences.jsonl") as f:
    prefs = [json.loads(line) for line in f]
 
ratings = compute_elo(prefs)
 
print("Elo Ratings:")
print("-" * 35)
for agent, rating in sorted(ratings.items(), key=lambda x: -x[1]):
    print(f"  {agent:<20} {rating:.0f}")

分维度拆解

对于多维度比较，可以看看每个智能体分别在哪些维度上表现好：

python
import json
from collections import defaultdict
 
with open("training_data/preferences.jsonl") as f:
    prefs = [json.loads(line) for line in f]
 
# Only process multi-dimension annotations
multi_dim = [p for p in prefs if "dimensions" in p]
 
dim_wins = defaultdict(lambda: defaultdict(lambda: {"A": 0, "B": 0, "Tie": 0}))
 
for pref in multi_dim:
    agent_a = pref["chosen"]["agent"]
    agent_b = pref["rejected"]["agent"]
    pair_key = f"{agent_a} vs {agent_b}"
 
    for dim_name, dim_data in pref["dimensions"].items():
        dim_wins[dim_name][pair_key][dim_data["preference"]] += 1
 
print("Per-Dimension Win Rates:")
print("=" * 60)
for dim_name, matchups in sorted(dim_wins.items()):
    print(f"\n  {dim_name.upper()}")
    print(f"  {'-' * 50}")
    for pair, counts in matchups.items():
        total = counts["A"] + counts["B"] + counts["Tie"]
        a_rate = (counts["A"] + 0.5 * counts["Tie"]) / total * 100
        print(f"    {pair}: A={a_rate:.0f}% B={100-a_rate:.0f}%  "
              f"(A:{counts['A']} B:{counts['B']} Tie:{counts['Tie']})")

实践经验

怎么选模式

想快速拿到几千条偏好、训一个通用奖励模型，或者做一个排行榜，二元模式是对的选择。按每次比较 1 到 2 分钟估算。

如果偏好强度要进你的训练流程，量表模式就值回票价。带边际加权的 DPO 在意强偏好（滑块推到端点）和弱偏好（滑块靠近中间）之间的差别。按每次比较 2 到 3 分钟估算。

当你需要知道智能体强在哪、弱在哪，或者要训练特定维度的奖励模型，又或者要给智能体开发者一份详细报告时，多维度模式值得多花的这些时间。按每次比较 4 到 6 分钟估算。

需要多少次比较

要算出可靠的胜率，每一对智能体至少收集 100 次比较。要在五个及以上智能体之间算 Elo，总共 200 到 300 次比较排名就会趋于稳定。要训 DPO 奖励模型，目标是 1,000 条以上的偏好对，并且难易任务都要覆盖到。

随机化顺序

一定要设 randomize_order: true。位置偏差，也就是倾向于选左侧或第一个标签页里那条 trace 的现象，在人类评估研究中有大量记录。把随机化和 attention_checks.type: "duplicate_reversed" 检查配合使用，可以抓出一直点同一侧的人。

处理平局

二元模式下允许平局，但要盯着平局率。如果超过 30%，说明这些智能体之间太接近，不适合做二元判断，该换到量表或多维度模式了。量表模式下平局就是中点。多维度模式下，单个维度上出现平局是正常的，而且本身就有信息量。

隐藏智能体身份

除非有充分理由，否则保持 show_agent_identity: false。标注者一旦知道某条 trace 出自哪个智能体，就会倾向于偏袒他们本来就认为更强的那个。

组合使用多种模式

想做一次彻底的评估，可以先在一大批配对上跑二元模式拿到总体排名，再在一个较小的分层子集上跑多维度模式获取诊断细节。二元比较喂给奖励模型训练，多维度比较则告诉你该往哪个方向改进智能体。

这三种模式背后的配置参考见源文档。想端到端地走一遍智能体评估流程，可以从智能体评估指南开始。