Boundary Lab

Probe the decision boundary, not just the point. After each label, Potato asks whether a minimal counterfactual edit would change it — producing contrast sets and quality control from ordinary annotation.

New in v2.7.0

Every annotation tool collects point labels: item X gets label Y. Boundary Lab makes Potato collect decision boundaries. The moment an annotator commits a label, Potato shows minimal counterfactual edits of the text and asks, one probe at a time:

You said Polite. Would that survive this edit?

After each label, a counterfactual probe over a colored diff: "would that still hold?"

Each answer takes one click. Ordinary annotation then produces three artifacts that a plain label export cannot.

Contrast sets, for free. Every answered probe is a labeled (original, counterfactual) pair — the counterfactually-augmented data shown to improve model robustness (Gardner et al. 2020; Kaushik et al. 2020) — normally built as a separate and expensive effort.
Boundary rationales. When a label flips, the annotator says what crossed the line. These rationales pinpoint where your codebook is ambiguous.
Invisible quality control. Invariance probes are meaning-preserving paraphrases, and a consistent annotator never flips on them. Annotators who do are flagged on the dashboard — an attention signal collected without planting a single fake gold item.

Configuration

yaml

boundary_probing:
  enabled: true
  schema: politeness          # scheme to probe (default: first radio scheme)
  probes_per_item: 3          # probes per (instance, label), invariance included
  include_invariance: true    # add one paraphrase probe (the QC signal)
  sources:                    # probe generation tiers, in priority order
    - precomputed
    - llm
    - rules
  precomputed_key: counterfactuals   # item-data field for precomputed probes
  rationale_on_flip: true     # ask "what crossed the line?" when a label flips
  debounce_ms: 900            # delay between label selection and probe fetch

Probe sources

Probes come from three tiers. Later tiers fill slots that earlier tiers leave empty, so the feature degrades gracefully.

precomputed — ship curated counterfactuals with your data. Deterministic, and ideal for controlled studies:

json

{"id": "req_02", "text": "Send me the slides before the meeting.",
 "counterfactuals": [
   {"text": "Please send me the slides before the meeting.", "kind": "flip",
    "edit_hint": "added \"please\""},
   {"text": "Before the meeting, send me the slides.", "kind": "invariance",
    "edit_hint": "reordered clauses"}
 ]}

llm — generate probes on the fly with any configured AI endpoint (Anthropic, OpenAI, Ollama, vLLM). Probes are generated once per (instance, label) and shared across annotators, so LLM cost is bounded by dataset size rather than annotator count.

rules — deterministic lexical transforms: negation toggles, intensifier swaps, politeness markers, punctuation heat, and contraction or greeting paraphrases for invariance. No dependencies and no model calls, which is what lets Boundary Lab run with no LLM configured at all.

Probe kinds and verdicts

Kind	Meaning	Expected behavior
`flip`	Smallest edit intended to cross the label boundary	May flip or hold — both are informative
`invariance`	Meaning-preserving paraphrase	Should never flip; flips indicate inconsistency

Annotators answer each probe with holds (the label survives), flips (the label changes, and they pick the new one and optionally say why), or can't tell.

The dashboard

At /boundary/dashboard (admin access):

Boundary sensitivity by label — of the minimal edits aimed at each label, the share that flipped. A 90% flip rate means the label lives on a knife's edge; 10% means it is robust to small perturbations.
Annotator invariance consistency — per-annotator hold rate on paraphrase probes, flagged red below 60%.
Where labels flip — a gallery of confirmed flips with word-level diffs and the annotators' rationales.

Contrast-set export

GET /boundary/api/export (admin) downloads JSONL, one labeled pair per answered probe: