Skip to content
Guides8 min read

並排比較 AI 智慧體：二元、量表與多維度三種模式

在 Potato 中搭建成對智慧體比較，共三種模式：二元偏好、連續量表，以及需要填寫理由的分維度多準則判斷。

Potato Team

智慧體評估為什麼要用成對比較

讓人給一條編碼智慧體 trace 打 1 到 10 分，拿到的資料噪聲很大，因為每個人對這把尺子的校準都不一樣。這個標註者的 7 分，可能就是另一個人的 5 分。成對比較繞開了這個問題：標註者不再孤立地給單條 trace 打分，而是並排看兩條，說出哪一條更好。這種一對一的判斷更容易做出，人與人之間也更一致，而且恰好就是直接偏好最佳化（DPO）和基於人類反饋的強化學習（RLHF）所需要的資料形式。

訓練語言模型對齊用的獎勵模型走的是同一條路，這套做法搬到編碼智慧體上也很順：在成對的智慧體軌跡之間收集人類偏好，用它們訓練獎勵模型，再用這個模型指導智慧體訓練，或者在推理時從 N 個候選裡挑最好的那個。

Potato 提供三種成對比較模式，分別對應不同的評估需求和資料預算。

介面把兩條 trace 並排放在一起：

Side-by-side agent comparison interfaceAnnotators compare two agent traces and select which approach was better

模式 1：二元偏好

這是最簡單也最快的模式。標註者並排看到兩條 trace，點選更好的那條。還有一個可選的平局按鈕，用於兩者一樣好或一樣差的情況。

什麼時候用二元模式

需要在短時間內攢出大量偏好資料時，就選二元模式。它適合訓練基礎的獎勵模型、算智慧體勝率、搭 Elo 排行榜。代價是丟掉了細節：你知道哪條 trace 贏了，但不知道贏了多少，也不知道贏在哪些方面。

配置

yaml
# config.yaml
project_name: "Agent Comparison - Binary"
port: 8000
 
data:
  source: "local"
  input_path: "./data/paired_traces.jsonl"
  data_format: "paired_coding_trace"
 
coding_agent:
  display:
    diff_style: "unified"
    syntax_highlighting: true
    terminal_theme: "dark"
    file_tree:
      enabled: true
      position: "left"
    collapsible:
      auto_collapse_thinking: true
 
comparison:
  layout: "side_by_side"         # "side_by_side" or "tabbed"
  label_a: "Agent A"
  label_b: "Agent B"
  randomize_order: true          # Randomize which trace appears on which side
  show_agent_identity: false     # Hide agent names to avoid bias
  sync_scroll: false             # Independent scrolling for each trace
 
annotation_schemes:
  - annotation_type: pairwise
    name: preference
    description: "Which agent produced a better solution?"
    items_key:
      - value: "a"
        text: "Agent A is better"
        keyboard_shortcut: "1"
      - value: "b"
        text: "Agent B is better"
        keyboard_shortcut: "2"
      - value: "tie"
        text: "Tie (equally good or equally bad)"
        keyboard_shortcut: "3"
    allow_tie: true
  - annotation_type: radio
    name: confidence
    labels:
      - value: "high"
        text: "Very confident"
      - value: "medium"
        text: "Somewhat confident"
      - value: "low"
        text: "Not confident"
 
output:
  path: "./output/"
  format: "jsonl"
 
quality_control:
  inter_annotator_agreement: true
  overlap_percentage: 20
  attention_checks:
 
annotators:
  - username: "judge1"
  - username: "judge2"

標註流程

標註者看到的是分屏介面。左邊是 Trace A，用完整的 CodingTraceDisplay 渲染：diff、終端塊、檔案讀取、思考過程。右邊是同一任務的 Trace B。兩側各自獨立滾動。

任務描述放在兩條 trace 上方，這樣標註者知道兩個智慧體本來要做什麼。

下方是三個按鈕：“Agent A is better”“Agent B is better”和“Tie”。開啟 randomize_order 後，哪個智慧體是 A、哪個是 B 會逐條打亂，標註者就不會養成偏左或偏右的習慣。

如果需要更細的評估，介面也支援多個維度：

Pairwise preference selection interfaceBinary preference, continuous scale, and multi-dimension modes are available

模式 2：連續量表

量表模式讓標註者說出一條 trace 好多少，而不只是哪條贏了。他們不是點一下，而是拖動一根滑塊，從左端的“A 好得多”到右端的“B 好得多”，中間是“持平”。

什麼時候用量表模式

當偏好的強度而不只是方向也重要時，用量表模式。滑塊靠近端點意味著品質差距明顯，靠近中間則說明兩者接近。DPO 及類似流程可以按這個強度給樣本加權，讓區分明確的樣本發揮更大作用。

配置

yaml
# config.yaml
project_name: "Agent Comparison - Scale"
port: 8000
 
data:
  source: "local"
  input_path: "./data/paired_traces.jsonl"
  data_format: "paired_coding_trace"
 
coding_agent:
  display:
    diff_style: "unified"
    syntax_highlighting: true
    terminal_theme: "dark"
    file_tree:
      enabled: true
    collapsible:
      auto_collapse_thinking: true
 
comparison:
  layout: "side_by_side"
  randomize_order: true
  show_agent_identity: false
 
annotation_schemes:
  - annotation_type: pairwise
    name: preference_scale
    description: "Which agent produced a better solution, and by how much?"
    scale:
      points: 7                  # 7-point scale
      labels:
        1: "A is much better"
        2: "A is better"
        3: "A is slightly better"
        4: "Equal"
        5: "B is slightly better"
        6: "B is better"
        7: "B is much better"
      default: 4                 # Start at "Equal"
      show_numeric_value: true
 
output:
  path: "./output/"
  format: "jsonl"
 
quality_control:
  inter_annotator_agreement: true
  overlap_percentage: 20
 
annotators:
  - username: "judge1"
  - username: "judge2"

改用 5 檔量表

想標得更快、粒度稍粗一些，可以降到 5 檔：

yaml
annotation_schemes:
  - annotation_type: pairwise
    name: preference_scale_5
    description: "Compare the two solutions"
    scale:
      points: 5
      labels:
        1: "A is clearly better"
        2: "A is somewhat better"
        3: "About equal"
        4: "B is somewhat better"
        5: "B is clearly better"
      default: 3

模式 3：多維度比較

這是最細的模式。標註者不給一個總體偏好，而是在若干個相互獨立的維度上分別判斷兩條 trace。每個維度各自給出 A/B/平局，並且每次判斷都要寫理由。

什麼時候用多維度模式

當你不僅想知道哪個智慧體贏了，還想知道為什麼贏時，用這個模式。一條 trace 可能程式碼正確但效率很差，另一條可能高效卻漏掉了一個邊界情況。由此得到的分維度資料既可以訓練特定維度的獎勵模型，也可以作為詳細反饋交給做智慧體的人。

配置

yaml
# config.yaml
project_name: "Agent Comparison - Multi-Dimension"
port: 8000
 
data:
  source: "local"
  input_path: "./data/paired_traces.jsonl"
  data_format: "paired_coding_trace"
 
coding_agent:
  display:
    diff_style: "unified"
    syntax_highlighting: true
    terminal_theme: "dark"
    file_tree:
      enabled: true
    collapsible:
      auto_collapse_thinking: true
 
comparison:
  layout: "side_by_side"
  randomize_order: true
  show_agent_identity: false
 
annotation_schemes:
  - annotation_type: pairwise
    name: multi_dim_comparison
    description: "Compare the two solutions along each dimension"
 
      - name: "efficiency"
        label: "Efficiency"
        description: >
          How efficient is the agent's process? Does it take unnecessary
          steps, read irrelevant files, or make redundant edits?
        options: ["A", "B", "Tie"]
        require_justification: true
        justification_placeholder: "Which agent was more efficient and why?"
        weight: 0.2
 
      - name: "code_quality"
        label: "Code Quality"
        description: >
          Is the code well-written? Consider readability, naming,
          error handling, documentation, and adherence to existing patterns.
        options: ["A", "B", "Tie"]
        require_justification: true
        justification_placeholder: "Which produces better quality code?"
        weight: 0.2
 
      - name: "communication"
        label: "Communication"
        description: >
          How well does the agent explain its reasoning? Are its thinking
          steps clear and logical? Does it identify the root cause?
        options: ["A", "B", "Tie"]
        require_justification: true
        justification_placeholder: "Which agent communicates its approach better?"
        weight: 0.1
 
      - name: "robustness"
        label: "Robustness"
        description: >
          Does the solution handle edge cases? Does the agent verify its
          changes with tests? Is the fix narrow and targeted or fragile?
        options: ["A", "B", "Tie"]
        require_justification: true
        justification_placeholder: "Which solution is more robust?"
        weight: 0.1
 
 
output:
  path: "./output/"
  format: "jsonl"
 
quality_control:
  inter_annotator_agreement: true
  overlap_percentage: 25         # Higher overlap for this detailed task
  minimum_time_per_instance: 120 # 2 minutes minimum for thorough review
 
annotators:
  - username: "judge1"
  - username: "judge2"

準備成對 trace 資料

三種模式的輸入都是成對的 trace。JSONL 檔案的每一行包含兩條針對同一任務的 trace。

資料格式

json
{
  "id": "pair_001",
  "task_description": "Fix the IndexError in process_batch() when the input list is empty",
  "repo": "myorg/myproject",
  "trace_a": {
    "agent": "claude_code",
    "model": "claude-sonnet-4-20250514",
    "structured_turns": [
      {
        "step_idx": 0,
        "type": "file_read",
        "path": "src/batch.py",
        "content": "def process_batch(items):\n    result = items[0]\n    ...",
        "start_line": 10,
        "end_line": 25
      },
      {
        "step_idx": 1,
        "type": "file_edit",
        "path": "src/batch.py",
        "diff": "--- a/src/batch.py\n+++ b/src/batch.py\n@@ -10,3 +10,5 @@\n def process_batch(items):\n+    if not items:\n+        return []\n     result = items[0]\n"
      },
      {
        "step_idx": 2,
        "type": "bash_command",
        "command": "python -m pytest tests/test_batch.py -v",
        "output": "PASSED",
        "exit_code": 0
      }
    ]
  },
  "trace_b": {
    "agent": "swe_agent",
    "model": "gpt-4o",
    "structured_turns": [
      {
        "step_idx": 0,
        "type": "bash_command",
        "command": "find . -name '*.py' | xargs grep 'process_batch'",
        "output": "src/batch.py:def process_batch(items):\ntests/test_batch.py:    process_batch([])",
        "exit_code": 0
      },
      {
        "step_idx": 1,
        "type": "file_read",
        "path": "src/batch.py",
        "content": "def process_batch(items):\n    result = items[0]\n    ...",
        "start_line": 1,
        "end_line": 50
      },
      {
        "step_idx": 2,
        "type": "file_edit",
        "path": "src/batch.py",
        "diff": "--- a/src/batch.py\n+++ b/src/batch.py\n@@ -10,3 +10,6 @@\n def process_batch(items):\n+    if items is None or len(items) == 0:\n+        logger.warning('Empty input to process_batch')\n+        return []\n     result = items[0]\n"
      },
      {
        "step_idx": 3,
        "type": "bash_command",
        "command": "python -m pytest tests/ -v",
        "output": "PASSED (12 tests)",
        "exit_code": 0
      }
    ]
  }
}

從單條 trace 組裝成對資料

如果你手上是一批針對同樣任務的單條 trace，配對工具可以幫你組裝：

bash
# Generate all possible pairs for each task
potato pair-traces \
  --input ./data/individual_traces.jsonl \
  --output ./data/paired_traces.jsonl \
  --pair_by "task_id" \
  --strategy "all_pairs"
 
# Or sample a fixed number of pairs per task
potato pair-traces \
  --input ./data/individual_traces.jsonl \
  --output ./data/paired_traces.jsonl \
  --pair_by "task_id" \
  --strategy "sample" \
  --pairs_per_task 3

匯出比較資料

DPO/RLHF 偏好對

成對比較的主要匯出格式，是用於 DPO 或 RLHF 訓練的偏好對：

bash
potato export \
  --format dpo_preferences \
  --project ./output/ \
  --output ./training_data/preferences.jsonl

二元模式的輸出很簡單：

json
{
  "prompt": "Fix the IndexError in process_batch() when the input list is empty",
  "chosen": {"agent": "claude_code", "trace_id": "trace_a_001", "steps": [...]},
  "rejected": {"agent": "swe_agent", "trace_id": "trace_b_001", "steps": [...]},
  "annotator": "judge1",
  "confidence": "high"
}

量表模式多帶一個偏好強度：

json
{
  "prompt": "Fix the IndexError in process_batch()",
  "chosen": {"agent": "claude_code", "trace_id": "trace_a_001"},
  "rejected": {"agent": "swe_agent", "trace_id": "trace_b_001"},
  "preference_strength": 0.83,
  "scale_value": 2,
  "justification": "Agent A found and fixed the bug in fewer steps with cleaner code"
}

多維度模式則帶上每個維度的偏好：

json
{
  "prompt": "Fix the IndexError in process_batch()",
  "chosen": {"agent": "claude_code", "trace_id": "trace_a_001"},
  "rejected": {"agent": "swe_agent", "trace_id": "trace_b_001"},
  "overall_preference": "A",
  "dimensions": {
    "correctness": {"preference": "Tie", "justification": "Both correctly fix the bug"},
    "efficiency": {"preference": "A", "justification": "A solves it in 3 steps vs 4"},
    "code_quality": {"preference": "B", "justification": "B adds logging and handles None"},
    "communication": {"preference": "A", "justification": "A's reasoning is more focused"},
    "robustness": {"preference": "B", "justification": "B runs full test suite, not just one file"}
  },
  "weighted_score_a": 0.55,
  "weighted_score_b": 0.45
}

分析：勝率、Elo 評分和分維度拆解

計算勝率

python
import json
from collections import defaultdict
 
with open("training_data/preferences.jsonl") as f:
    prefs = [json.loads(line) for line in f]
 
wins = defaultdict(lambda: {"wins": 0, "losses": 0, "ties": 0})
 
for pref in prefs:
    agent_chosen = pref["chosen"]["agent"]
    agent_rejected = pref["rejected"]["agent"]
 
    if agent_chosen == agent_rejected:
        continue  # Skip self-comparisons
 
    if pref.get("overall_preference") == "Tie":
        wins[agent_chosen]["ties"] += 1
        wins[agent_rejected]["ties"] += 1
    else:
        wins[agent_chosen]["wins"] += 1
        wins[agent_rejected]["losses"] += 1
 
print("Agent Win Rates:")
print("-" * 55)
for agent, record in sorted(wins.items()):
    total = record["wins"] + record["losses"] + record["ties"]
    win_rate = (record["wins"] + 0.5 * record["ties"]) / total * 100
    print(f"  {agent:<20} {win_rate:5.1f}%  "
          f"(W:{record['wins']} L:{record['losses']} T:{record['ties']})")

計算 Elo 評分

python
import json
import math
from collections import defaultdict
 
def compute_elo(preferences, k=32, initial_rating=1500):
    """Compute Elo ratings from pairwise preferences."""
    ratings = defaultdict(lambda: initial_rating)
 
    for pref in preferences:
        agent_a = pref["chosen"]["agent"]
        agent_b = pref["rejected"]["agent"]
 
        ra = ratings[agent_a]
        rb = ratings[agent_b]
 
        # Expected scores
        ea = 1.0 / (1.0 + math.pow(10, (rb - ra) / 400))
        eb = 1.0 / (1.0 + math.pow(10, (ra - rb) / 400))
 
        overall = pref.get("overall_preference", "A")
        if overall == "Tie":
            sa, sb = 0.5, 0.5
        else:
            # "chosen" is the winner
            sa, sb = 1.0, 0.0
 
        ratings[agent_a] = ra + k * (sa - ea)
        ratings[agent_b] = rb + k * (sb - eb)
 
    return dict(ratings)
 
with open("training_data/preferences.jsonl") as f:
    prefs = [json.loads(line) for line in f]
 
ratings = compute_elo(prefs)
 
print("Elo Ratings:")
print("-" * 35)
for agent, rating in sorted(ratings.items(), key=lambda x: -x[1]):
    print(f"  {agent:<20} {rating:.0f}")

分維度拆解

對於多維度比較，可以看看每個智慧體分別在哪些維度上表現好：

python
import json
from collections import defaultdict
 
with open("training_data/preferences.jsonl") as f:
    prefs = [json.loads(line) for line in f]
 
# Only process multi-dimension annotations
multi_dim = [p for p in prefs if "dimensions" in p]
 
dim_wins = defaultdict(lambda: defaultdict(lambda: {"A": 0, "B": 0, "Tie": 0}))
 
for pref in multi_dim:
    agent_a = pref["chosen"]["agent"]
    agent_b = pref["rejected"]["agent"]
    pair_key = f"{agent_a} vs {agent_b}"
 
    for dim_name, dim_data in pref["dimensions"].items():
        dim_wins[dim_name][pair_key][dim_data["preference"]] += 1
 
print("Per-Dimension Win Rates:")
print("=" * 60)
for dim_name, matchups in sorted(dim_wins.items()):
    print(f"\n  {dim_name.upper()}")
    print(f"  {'-' * 50}")
    for pair, counts in matchups.items():
        total = counts["A"] + counts["B"] + counts["Tie"]
        a_rate = (counts["A"] + 0.5 * counts["Tie"]) / total * 100
        print(f"    {pair}: A={a_rate:.0f}% B={100-a_rate:.0f}%  "
              f"(A:{counts['A']} B:{counts['B']} Tie:{counts['Tie']})")

實踐經驗

怎麼選模式

想快速拿到幾千條偏好、訓一個通用獎勵模型，或者做一個排行榜，二元模式是對的選擇。按每次比較 1 到 2 分鐘估算。

如果偏好強度要進你的訓練流程，量表模式就值回票價。帶邊際加權的 DPO 在意強偏好（滑塊推到端點）和弱偏好（滑塊靠近中間）之間的差別。按每次比較 2 到 3 分鐘估算。

當你需要知道智慧體強在哪、弱在哪，或者要訓練特定維度的獎勵模型，又或者要給智慧體開發者一份詳細報告時，多維度模式值得多花的這些時間。按每次比較 4 到 6 分鐘估算。

需要多少次比較

要算出可靠的勝率，每一對智慧體至少收集 100 次比較。要在五個及以上智慧體之間算 Elo，總共 200 到 300 次比較排名就會趨於穩定。要訓 DPO 獎勵模型，目標是 1,000 條以上的偏好對，並且難易任務都要覆蓋到。

隨機化順序

一定要設 randomize_order: true。位置偏差，也就是傾向於選左側或第一個標籤頁裡那條 trace 的現象，在人類評估研究中有大量記錄。把隨機化和 attention_checks.type: "duplicate_reversed" 檢查配合使用，可以抓出一直點同一側的人。

處理平局

二元模式下允許平局，但要盯著平局率。如果超過 30%，說明這些智慧體之間太接近，不適合做二元判斷，該換到量表或多維度模式了。量表模式下平局就是中點。多維度模式下，單個維度上出現平局是正常的，而且本身就有資訊量。

隱藏智慧體身份

除非有充分理由，否則保持 show_agent_identity: false。標註者一旦知道某條 trace 出自哪個智慧體，就會傾向於偏袒他們本來就認為更強的那個。

組合使用多種模式

想做一次徹底的評估，可以先在一大批配對上跑二元模式拿到總體排名，再在一個較小的分層子集上跑多維度模式獲取診斷細節。二元比較餵給獎勵模型訓練，多維度比較則告訴你該往哪個方向改進智慧體。

這三種模式背後的配置參考見源文件。想端到端地走一遍智慧體評估流程，可以從智慧體評估指南開始。