How to Tell Review from Rubber-Stamping in Pre-Labelled Data

Pre-labelling makes annotators faster and rubber-stamping frictionless, and no measure computed from the annotations themselves can tell the two apart. The geometry is identical, and inter-annotator agreement goes up, because annotators who all accept the same model output are unanimous by construction.

The difference appears in the timing of the work.

Why quality measures fail here

Suppose a model pre-labels every image and three annotators accept nearly everything.

• Agreement rises, because they all record the model's output.
• Gold-standard accuracy rises wherever the model is right, which is most of the time.
• Adjudication volume falls, because there is nothing to adjudicate.

Every dashboard looks better. The dataset is now a record of a model agreeing with itself, and its errors are systematic rather than random — which is the worst kind, because they will not average out across annotators.

Pre-labelling is a standard feature of current annotation platforms, including Label Studio's pre-annotations, CVAT's automated annotation and V7's Auto-Annotate. Wherever it is turned on, chance-corrected agreement stops meaning what its interpretation says it means, because the coefficient was designed for independent judgements and an accepted pre-label is not one.

What to record

Timing, at the level of the individual accept:

You do not need coordinates for any of this, and you should not collect them. A process record that cannot reconstruct the annotation is much easier to justify to an ethics board, and it answers the question just as well.

Reading the numbers

A human cannot inspect a mask boundary and decide in 300 ms. Once, maybe, if the object was obvious. As a median across many items, it is not fast expertise.

Read a distribution, never a threshold on one item. Three rules that hold up:

1. Compare each annotator to the project, not to a constant. An easy corpus really is fast. Percentile calibration within your own project is the only defensible baseline.
2. Look for runs. Twenty consecutive sub-500 ms accepts is a much stronger signal than twenty scattered ones.
3. Split by whether a suggestion was present. An annotator's own drawing speed is the natural control for their accept speed.

What this does not do

It does not detect fraud, and it does not classify an annotator. A flag is a prompt to look, not a finding.

Base rates matter and are easy to forget: in a corpus where 80% of items really are trivial, fast will usually be correct, and a rule tuned to catch rubber-stamping will mostly catch competence. That is why the thresholds belong to your project rather than to a shipped model.

Doing it in Potato

annotation_telemetry:
  enabled: true
  fidelity: events
  idle_ms: 120000

For free-text answers the equivalent is keystroke logging, which distinguishes composed from transcribed from pasted text. It captures on beforeinput rather than keydown, because paste, drop, IME, dictation and autofill all change a field without firing a keydown — so a keydown-only logger is blind to exactly the cases it exists to catch.

Neither ships a pre-trained classifier. There is no labelled corpus of rubber-stamping, and fitted coefficients presented as validated would be fabricated validation.

Ethics

Annotators see a recording notice by default, and turning disclosure off with disclose_to_annotators: false logs a warning. Be explicit in your consent language about what is recorded (timing and process), what is not (content and coordinates), how long it is kept, and what it will be used for.

See keystroke logging ethics for sample consent language and the retention questions.

Further reading

• Drawing telemetry
• Keystroke logging
• Gold standards and attention checks
• LLM pre-annotation