# Agreement Cannot Catch Rubber-Stamping Source: https://www.potatoannotator.com/blog/agreement-cannot-catch-rubber-stamping Here is a failure mode that every quality measure we have gets backwards. A model pre-labels a corpus. Three annotators review it. Two of them read each suggestion and correct what is wrong; the third accepts everything without looking. Now compute your quality metrics: - **Inter-annotator agreement** goes *up*, because the third annotator matches the model exactly, and so does most of what the other two accepted. - **Gold-standard accuracy** goes up wherever the model is right, which is most of the time. - **Adjudication volume** goes down, because there is less to adjudicate. Every dashboard improves. The annotator who did no work is indistinguishable from the two who did — and by agreement, they look like the *best* annotator, because they never introduce an idiosyncratic correction. ## Why this is a 2026 problem specifically Chance-corrected agreement was designed for **independent** judgements. That assumption is what makes the chance baseline meaningful. Pre-labelling breaks it. Two annotators who both accepted the same suggestion did not independently arrive at the same answer; they shared a source. The coefficient is still computable and no longer means what its interpretation says it means. This was a niche concern when pre-labelling meant an occasional model-assisted suggestion. One-click auto-labelling is now the default across the industry, and the reliability literature has not caught up. ## The only place it shows Nothing in the annotation distinguishes rubber-stamping from careful review, because in the cases where the model was right, the two produce **byte-identical output**. There is no content signal to find. What differs is the process: > A human cannot inspect a mask boundary and decide in 300 ms. Once, maybe, if the object was obvious. As a **median across many items**, it is not fast expertise. So Potato records the process: time per shape, stroke dynamics, revision counts, zoom behaviour, and — the signal that most often changes a decision — **AI-suggestion accept latency**, the time from a suggestion being rendered to being accepted. ## Recording process without recording content The obvious objection is surveillance, and it is a fair one. The design answer is that the event record **cannot** reconstruct the annotation. An event carries a timestamp, an action, a geometry kind, and one integer. No coordinates, ever. That is a structural property rather than a policy: there is no field a coordinate could go in, and a test asserts it. The same principle governs the text-side equivalent. [Keystroke logging](/docs/features/keystroke-logging) records timing and edit structure, never characters. It captures on `beforeinput` rather than `keydown`, because paste, drop, IME, dictation and autofill all mutate a field without firing a keydown — so a keydown-only logger is blind to exactly the cases it exists to detect. The gap between characters that appeared and keys actually pressed is the strongest single signal. Annotators see a recording notice, and disclosure defaults to on. That is the line between a research instrument and surveillance. ## What we deliberately did not build **No pre-trained classifier ships.** There is no labelled corpus of rubber-stamping, and fitted coefficients presented as validated would be fabricated validation. Detection is a handful of transparent rules with visible evidence and per-project percentile calibration. We also do not claim to detect AI-written annotations. "Surfaces responses that were pasted rather than typed" is what the data supports. "Detects AI-generated text" is not, and the difference matters if anyone is going to act on a flag. ## How to read the numbers **As a distribution, never as a threshold on one item.** Three rules that hold up: 1. **Compare each annotator to the project, not to a constant.** An easy corpus really is fast. Percentile calibration within your own project is the only defensible baseline. 2. **Look for runs.** Twenty consecutive sub-500 ms accepts is a much stronger signal than twenty scattered across a session. 3. **Split by whether a suggestion was present.** An annotator's own drawing speed is the natural control for their accept speed. And the base rate matters. In a corpus where 80% of items are genuinely trivial, fast is usually correct, and a rule tuned to catch rubber-stamping will mostly catch competence. A flag is a prompt to look rather than a finding in itself. ## The uncomfortable part If you are running a pre-labelled annotation study and not recording timing, you currently have **no way to tell** whether your annotators are reviewing or accepting. Your agreement statistics will not tell you, and they will look good. Pre-labelling genuinely helps, so this is not an argument against it. It is an argument for measuring what pre-labelling changes. ## Further reading - [Telling review from rubber-stamping](/docs/guides/detecting-rubber-stamped-prelabels) - [Drawing telemetry](/docs/measurement/drawing-telemetry) - [Keystroke logging ethics](/docs/features/keystroke-logging-ethics)