Skip to content
Guides10 min read

Comparing AI Agents Side by Side: Binary, Scale, and Multi-Dimension Modes

Set up pairwise agent comparison in Potato across its three modes — binary preference, a preference scale, and per-dimension judgment with required justification.

Potato Team

Why pairwise comparison for agent evaluation

Asking someone to rate a coding agent trace on a scale of 1 to 10 gives you noisy data, because everyone calibrates that scale differently. One annotator's 7 is another's 5. Pairwise comparison gets around this. Instead of rating traces on their own, annotators look at two side by side and say which one is better. That head-to-head judgment is easier to make, more consistent across people, and is what Direct Preference Optimization (DPO) and Reinforcement Learning from Human Feedback (RLHF) take as input.

Potato has three pairwise modes, set with mode on the scheme. They differ in what they ask for and what they cost per comparison.

Side-by-side agent comparison interfaceAnnotators compare two agent traces and select which approach was better

The data the schema expects

All three modes read the two candidates the same way. items_key names a field on the item holding a list of two or more entries:

json
{"id": "pair_001", "task": "Fix the IndexError in process_batch() when the input list is empty", "responses": ["Agent A trace text...", "Agent B trace text..."]}

When your data names the two sides instead of listing them, Potato falls back to a left and right pair on the item, which it reads only when items_key names no field there:

json
{"id": "pair_002", "task": "Fix the IndexError", "left": "Agent A trace...", "right": "Agent B trace..."}

If neither yields two candidates, the labels render with nothing to compare, and the browser console reports it when items_key points at something that is not a list.

Mode 1: binary preference

The annotator sees both candidates and clicks the better one. An optional tie button covers cases where they are equally good or equally bad.

Reach for binary mode when you need a lot of preference data quickly, for reward model training, win rates, or an Elo leaderboard. The cost is nuance: you learn which one won, not by how much or on what grounds.

yaml
annotation_task_name: "Agent Comparison - Binary"
task_dir: "."
port: 8000
 
data_files:
  - "data/paired_traces.jsonl"
 
item_properties:
  id_key: "id"
  text_key: "task"
 
instance_display:
  fields:
    - key: "task"
      type: text
      label: "Task"
 
annotation_schemes:
  - annotation_type: pairwise
    name: preference
    description: "Which agent produced a better solution?"
    mode: binary
    items_key: "responses"
    show_labels: true
    labels:
      - "Agent A"
      - "Agent B"
    allow_tie: true
    tie_label: "Tie (equally good or equally bad)"
    randomize_order: true
    sequential_key_binding: true
    label_requirement:
      required: true
 
  - annotation_type: radio
    name: confidence
    description: "How confident are you in this judgment?"
    labels:
      - "Very confident"
      - "Somewhat confident"
      - "Not confident"
 
output_annotation_dir: "annotation_output/"
export_annotation_format: jsonl
 
user_config:
  allow_all_users: true

randomize_order: true shuffles which candidate appears on which side, per annotator and per item, so nobody settles into a left-side habit. Potato records the order each annotator actually saw, which is what lets you undo the shuffle at analysis time.

To keep annotators blind to which agent produced which trace, leave the agent name out of the text you put in responses. There is no setting that hides it for you.

Mode 2: preference scale

Scale mode replaces the two buttons with a slider, so the annotator says how much better one candidate is. Use it when preference strength feeds your pipeline, since DPO with margin weighting leans harder on the clear-cut calls.

The scale is defined by its endpoints rather than by a point count. Negative values prefer the left item, positive the right, and zero is equal:

yaml
annotation_schemes:
  - annotation_type: pairwise
    name: preference_scale
    description: "Which agent produced a better solution, and by how much?"
    mode: scale
    items_key: "responses"
    labels:
      - "Agent A"
      - "Agent B"
    scale:
      min: -3
      max: 3
      step: 1
      default: 0
      labels:
        min: "A is much better"
        max: "B is much better"
        center: "Equal"
    label_requirement:
      required: true

A narrower scale annotates faster. Setting min: -2 and max: 2 gives five positions instead of seven, which is usually enough to separate a clear preference from a marginal one.

Mode 3: multi-dimension comparison

Rather than one overall preference, the annotator judges the pair on several independent dimensions, each with its own A/B/tie row. Use it when you need to know not just which agent won but where, since one trace can have correct code and poor efficiency while the other is efficient and misses an edge case.

A dimension takes a name, an optional description shown as help text, and its own tie settings:

yaml
annotation_schemes:
  - annotation_type: pairwise
    name: multi_dim_comparison
    description: "Compare the two solutions along each dimension"
    mode: multi_dimension
    items_key: "responses"
    labels:
      - "Agent A"
      - "Agent B"
    dimensions:
      - name: correctness
        description: "Which solution actually fixes the stated problem?"
        allow_tie: true
      - name: efficiency
        description: "Which takes fewer unnecessary steps, file reads, or redundant edits?"
        allow_tie: true
      - name: code_quality
        description: "Which is more readable, better named, and closer to existing patterns?"
        allow_tie: true
      - name: communication
        description: "Which explains its reasoning more clearly and identifies the root cause?"
        allow_tie: true
      - name: robustness
        description: "Which handles edge cases and verifies its changes?"
        allow_tie: true
    justification:
      required: true
      reason_categories:
        - "More correct"
        - "More efficient"
        - "Better written"
        - "Better explained"
      min_rationale_chars: 20
      rationale_placeholder: "Which agent was better here, and why?"

Dimensions carry no weights. Weighting is a decision about what you care about, and doing it at analysis time means you can re-weight a finished study without recollecting anything.

justification is a block on the scheme rather than a per-dimension setting, and it works in all three modes. reason_categories gives annotators a quick structured reason, and min_rationale_chars sets a floor on the free text so "A is better" does not pass as an explanation.

What the output contains

Each mode stores its value under the scheme's name.

Binary mode records the selection:

json
{"preference": {"selection": "A"}}

The value is "A" or "B", and "tie" as well when the scheme sets allow_tie: true. The stored string is always the literal tie; tie_label changes the button text, not the value.

Scale mode records the slider position as a string, negative for the left item:

json
{"preference_scale": {"scale_value": "-2"}}

Multi-dimension mode records one value per dimension, plus the justification as a JSON-encoded string:

json
{
  "multi_dim_comparison": {
    "correctness": "tie",
    "efficiency": "A",
    "code_quality": "B",
    "communication": "A",
    "robustness": "B",
    "justification": "{\"reasons\": [\"More efficient\"], \"rationale\": \"A fixed it in three steps against four...\"}"
  }
}

That justification field is a string containing JSON, not a nested object. Parse it a second time before reading reasons or rationale.

Exporting

Potato exports annotations in 31 formats, and preference-pair formats for DPO are not among them. Export the annotations and build the pairs yourself:

bash
python -m potato.export -c config.yaml -f jsonl -o ./export/

Turning the exported binary preferences into DPO training pairs is a short script, and doing it yourself is what lets you decide how ties and low-confidence judgments are handled rather than inheriting someone else's choice:

python
import json
from pathlib import Path
 
SCHEME = "preference"
 
pairs = []
for f in Path("export/").rglob("*.jsonl"):
    with open(f) as fh:
        for line in fh:
            rec = json.loads(line)
            labels = rec.get("labels", rec)
            sel = (labels.get(SCHEME) or {}).get("selection")
            if sel not in ("A", "B"):
                continue  # drop ties; they carry no preference direction
 
            item = items_by_id[rec["instance_id"]]       # your own data file
            a, b = item["responses"][0], item["responses"][1]
            chosen, rejected = (a, b) if sel == "A" else (b, a)
 
            pairs.append({
                "prompt": item["task"],
                "chosen": chosen,
                "rejected": rejected,
                "annotator": rec.get("user_id"),
            })
 
with open("training_data/preferences.jsonl", "w") as out:
    for p in pairs:
        out.write(json.dumps(p) + "\n")
 
print(f"Wrote {len(pairs)} preference pairs")

Join back to your own data file on instance_id, and undo the randomize_order shuffle using the order Potato recorded before you decide which side was which.

Analysis

Win rates

python
import json
from collections import defaultdict
from pathlib import Path
 
SCHEME = "preference"
records = []
for f in Path("export/").rglob("*.jsonl"):
    with open(f) as fh:
        records += [json.loads(line) for line in fh]
 
tally = defaultdict(lambda: {"wins": 0, "losses": 0, "ties": 0})
 
for rec in records:
    sel = (rec.get("labels", rec).get(SCHEME) or {}).get("selection")
    if not sel:
        continue
    item = items_by_id[rec["instance_id"]]       # your own data file
    # `responses` deliberately carries no agent names, so a win rate per agent
    # needs fields you add yourself when building the data file.
    agent_a, agent_b = item["agent_a"], item["agent_b"]
 
    if sel == "tie":
        tally[agent_a]["ties"] += 1
        tally[agent_b]["ties"] += 1
    else:
        winner, loser = (agent_a, agent_b) if sel == "A" else (agent_b, agent_a)
        tally[winner]["wins"] += 1
        tally[loser]["losses"] += 1
 
for agent, r in sorted(tally.items()):
    total = r["wins"] + r["losses"] + r["ties"]
    rate = (r["wins"] + 0.5 * r["ties"]) / total * 100
    print(f"  {agent:<20} {rate:5.1f}%  (W:{r['wins']} L:{r['losses']} T:{r['ties']})")

Elo ratings

Elo is order-dependent, so shuffle the comparisons before running it and average over several shuffles if the ranking is close:

python
import math
from collections import defaultdict
 
def compute_elo(results, k=32, initial=1500):
    ratings = defaultdict(lambda: initial)
    for winner, loser, tied in results:
        rw, rl = ratings[winner], ratings[loser]
        ew = 1.0 / (1.0 + 10 ** ((rl - rw) / 400))
        el = 1.0 / (1.0 + 10 ** ((rw - rl) / 400))
        sw, sl = (0.5, 0.5) if tied else (1.0, 0.0)
        ratings[winner] = rw + k * (sw - ew)
        ratings[loser] = rl + k * (sl - el)
    return dict(ratings)

Per-dimension breakdown

python
from collections import defaultdict
 
SCHEME = "multi_dim_comparison"
DIMS = ["correctness", "efficiency", "code_quality", "communication", "robustness"]
 
counts = defaultdict(lambda: {"A": 0, "B": 0, "tie": 0})
for rec in records:
    vals = rec.get("labels", rec).get(SCHEME) or {}
    for d in DIMS:
        v = vals.get(d)
        if v in ("A", "B", "tie"):
            counts[d][v] += 1
 
for d in DIMS:
    c = counts[d]
    total = sum(c.values()) or 1
    a_rate = (c["A"] + 0.5 * c["tie"]) / total * 100
    print(f"  {d:<16} A={a_rate:.0f}%  B={100 - a_rate:.0f}%  (ties {c['tie']})")

The dimension where the split sits closest to 50/50 with a high tie count is usually the one whose description needs work, not the one where the agents are genuinely equal.

What works in practice

Picking a mode

Binary mode suits thousands of comparisons and a general reward model. Scale mode is worth the extra time when preference strength feeds training. Multi-dimension mode is worth it when you owe someone a diagnostic rather than a ranking, and it costs several times as long per comparison because each dimension needs a justification.

How many comparisons

For a stable win rate between two agents, collect at least 100 comparisons of that pair. Elo across five or more agents settles with a few hundred total. Reward model training generally wants preference pairs in the thousands, spanning easy and hard tasks rather than concentrating on one.

Catching inattention

randomize_order removes the side habit. To catch clicking-through, add real attention checks and a response-time floor:

yaml
attention_checks:
  enabled: true
  items_file: "data/attention_checks.json"
  frequency: 20
  min_response_time: 10.0
  failure_handling:
    warn_threshold: 2
    block_threshold: 5

min_response_time is the useful one here. Nobody reads two agent traces in ten seconds.

Ties

Allow ties in binary mode but watch the rate. Past about 30% the agents are too close for a binary call, and scale or multi-dimension mode will tell you more.

Combining modes

Run binary over a large pool for the ranking, then multi-dimension on a stratified subset for the diagnosis. The binary comparisons feed reward model training; the per-dimension ones tell you what to fix.

Further reading