Agreement Cannot Catch Rubber-Stamping
When annotators accept model pre-labels without reading them, inter-annotator agreement goes up. Every quality measure computed from the annotations improves. Timing is the only signal left.
Here is a failure mode that every quality measure we have gets backwards.
A model pre-labels a corpus. Three annotators review it. Two of them read each suggestion and correct what is wrong; the third accepts everything without looking.
Now compute your quality metrics:
- Inter-annotator agreement goes up, because the third annotator matches the model exactly, and so does most of what the other two accepted.
- Gold-standard accuracy goes up wherever the model is right, which is most of the time.
- Adjudication volume goes down, because there is less to adjudicate.
Every dashboard improves. The annotator who did no work is indistinguishable from the two who did — and by agreement, they look like the best annotator, because they never introduce an idiosyncratic correction.
Why this is a 2026 problem specifically
Chance-corrected agreement was designed for independent judgements. That assumption is what makes the chance baseline meaningful.
Pre-labelling breaks it. Two annotators who both accepted the same suggestion did not independently arrive at the same answer; they shared a source. The coefficient is still computable and no longer means what its interpretation says it means.
This was a niche concern when pre-labelling meant an occasional model-assisted suggestion. One-click auto-labelling is now the default across the industry, and the reliability literature has not caught up.
The only place it shows
Nothing in the annotation distinguishes rubber-stamping from careful review, because in the cases where the model was right, the two produce byte-identical output. There is no content signal to find.
What differs is the process:
A human cannot inspect a mask boundary and decide in 300 ms. Once, maybe, if the object was obvious. As a median across many items, it is not fast expertise.
So Potato records the process: time per shape, stroke dynamics, revision counts, zoom behaviour, and — the signal that most often changes a decision — AI-suggestion accept latency, the time from a suggestion being rendered to being accepted.
Recording process without recording content
The obvious objection is surveillance, and it is a fair one. The design answer is that the event record cannot reconstruct the annotation.
An event carries a timestamp, an action, a geometry kind, and one integer. No coordinates, ever. That is a structural property rather than a policy: there is no field a coordinate could go in, and a test asserts it.
The same principle governs the text-side equivalent. Keystroke logging records timing and edit structure, never characters. It captures on beforeinput rather than keydown, because paste, drop, IME, dictation and autofill all mutate a field without firing a keydown — so a keydown-only logger is blind to exactly the cases it exists to detect. The gap between characters that appeared and keys actually pressed is the strongest single signal.
Annotators see a recording notice, and disclosure defaults to on. That is the line between a research instrument and surveillance.
What we deliberately did not build
No pre-trained classifier ships. There is no labelled corpus of rubber-stamping, and fitted coefficients presented as validated would be fabricated validation. Detection is a handful of transparent rules with visible evidence and per-project percentile calibration.
We also do not claim to detect AI-written annotations. "Surfaces responses that were pasted rather than typed" is what the data supports. "Detects AI-generated text" is not, and the difference matters if anyone is going to act on a flag.
How to read the numbers
As a distribution, never as a threshold on one item. Three rules that hold up:
- Compare each annotator to the project, not to a constant. An easy corpus really is fast. Percentile calibration within your own project is the only defensible baseline.
- Look for runs. Twenty consecutive sub-500 ms accepts is a much stronger signal than twenty scattered across a session.
- Split by whether a suggestion was present. An annotator's own drawing speed is the natural control for their accept speed.
And the base rate matters. In a corpus where 80% of items are genuinely trivial, fast is usually correct, and a rule tuned to catch rubber-stamping will mostly catch competence. A flag is a prompt to look rather than a finding in itself.
The uncomfortable part
If you are running a pre-labelled annotation study and not recording timing, you currently have no way to tell whether your annotators are reviewing or accepting. Your agreement statistics will not tell you, and they will look good.
Pre-labelling genuinely helps, so this is not an argument against it. It is an argument for measuring what pre-labelling changes.