# Boundary Lab

Source: https://www.potatoannotator.com/docs/features/boundary-lab

*New in v2.7.0*

Every annotation tool collects point labels: item X gets label Y. Boundary Lab makes Potato collect **decision boundaries**. The moment an annotator commits a label, Potato shows minimal counterfactual edits of the text and asks, one probe at a time:

> *You said **Polite**. Would that survive this edit?*

![After each label, a counterfactual probe over a colored diff: "would that still hold?"](/images/docs/boundary-lab.png)

Each answer takes one click. Ordinary annotation then produces three artifacts that a plain label export cannot.

1. **Contrast sets, for free.** Every answered probe is a labeled *(original, counterfactual)* pair — the counterfactually-augmented data shown to improve model robustness ([Gardner et al. 2020](https://aclanthology.org/2020.findings-emnlp.117/); [Kaushik et al. 2020](https://arxiv.org/abs/1909.12434)) — normally built as a separate and expensive effort.
2. **Boundary rationales.** When a label flips, the annotator says what crossed the line. These rationales pinpoint where your codebook is ambiguous.
3. **Invisible quality control.** Invariance probes are meaning-preserving paraphrases, and a consistent annotator never flips on them. Annotators who do are flagged on the dashboard — an attention signal collected without planting a single fake gold item.

## Configuration

```yaml
boundary_probing:
  enabled: true
  schema: politeness          # scheme to probe (default: first radio scheme)
  probes_per_item: 3          # probes per (instance, label), invariance included
  include_invariance: true    # add one paraphrase probe (the QC signal)
  sources:                    # probe generation tiers, in priority order
    - precomputed
    - llm
    - rules
  precomputed_key: counterfactuals   # item-data field for precomputed probes
  rationale_on_flip: true     # ask "what crossed the line?" when a label flips
  debounce_ms: 900            # delay between label selection and probe fetch
```

## Probe sources

Probes come from three tiers. Later tiers fill slots that earlier tiers leave empty, so the feature degrades gracefully.

**`precomputed`** — ship curated counterfactuals with your data. Deterministic, and ideal for controlled studies:

```json
{"id": "req_02", "text": "Send me the slides before the meeting.",
 "counterfactuals": [
   {"text": "Please send me the slides before the meeting.", "kind": "flip",
    "edit_hint": "added \"please\""},
   {"text": "Before the meeting, send me the slides.", "kind": "invariance",
    "edit_hint": "reordered clauses"}
 ]}
```

**`llm`** — generate probes on the fly with any configured [AI endpoint](/docs/features/ai-support) (Anthropic, OpenAI, Ollama, vLLM). Probes are generated once per (instance, label) and shared across annotators, so LLM cost is bounded by dataset size rather than annotator count.

**`rules`** — deterministic lexical transforms: negation toggles, intensifier swaps, politeness markers, punctuation heat, and contraction or greeting paraphrases for invariance. No dependencies and no model calls, which is what lets Boundary Lab run with **no LLM configured at all**.

## Probe kinds and verdicts

| Kind | Meaning | Expected behavior |
|------|---------|-------------------|
| `flip` | Smallest edit intended to cross the label boundary | May flip or hold — both are informative |
| `invariance` | Meaning-preserving paraphrase | Should never flip; flips indicate inconsistency |

Annotators answer each probe with **holds** (the label survives), **flips** (the label changes, and they pick the new one and optionally say why), or **can't tell**.

## The dashboard

At `/boundary/dashboard` (admin access):

- **Boundary sensitivity by label** — of the minimal edits aimed at each label, the share that flipped. A 90% flip rate means the label lives on a knife's edge; 10% means it is robust to small perturbations.
- **Annotator invariance consistency** — per-annotator hold rate on paraphrase probes, flagged red below 60%.
- **Where labels flip** — a gallery of confirmed flips with word-level diffs and the annotators' rationales.

## Contrast-set export

`GET /boundary/api/export` (admin) downloads JSONL, one labeled pair per answered probe:

```json
{"instance_id": "req_02", "schema": "politeness",
 "original_text": "Send me the slides before the meeting.",
 "original_label": "Impolite",
 "counterfactual_text": "Please send me the slides before the meeting.",
 "counterfactual_label": "Neutral",
 "kind": "flip", "flipped": true,
 "rationale": "please softens the command",
 "edit_hint": "added \"please\"", "probe_source": "precomputed",
 "annotator": "alice", "timestamp": 1783827299.4}
```

`holds` verdicts export with `counterfactual_label` equal to the original label, which makes them useful as hard negatives. `unsure` verdicts are excluded.

## Design notes

- Probes are cached per (instance, schema, label) and shared across annotators, so invariance consistency is comparable between annotators.
- The probe panel is an overlay. It never blocks the main annotation flow, probe answers are optional, and annotators can dismiss it at any time.
- Probing currently targets one single-choice (`radio`) scheme per task.

## Further reading

- [Truth Serum](/docs/features/truth-serum) — the other no-gold-label quality signal
- [Quality control](/docs/features/quality-control) — attention checks and gold standards
- [Gold standards and attention checks](/docs/guides/gold-standards-and-attention-checks)
- [Source documentation](https://github.com/davidjurgens/potato/blob/main/docs/advanced/boundary_lab.md)