Drawing Telemetry
How a box, polygon or mask was produced. Time per shape, stroke dynamics, revision counts and AI-accept latency separate review from rubber-stamping.
Pre-labelling makes annotators faster, and it makes rubber-stamping frictionless. A suggestion appears, the annotator clicks accept, and the resulting dataset is a record of a model agreeing with itself. Nothing in the annotation distinguishes that from careful review. The geometry is identical, and inter-annotator agreement rises while accuracy does not, because annotators who accept the same suggestion are unanimous by construction.
The difference shows up in the timing of the work.
A human cannot inspect a mask boundary and decide in 300 ms. Once, maybe, if the object was obvious. As a median across many items, it is not fast expertise.
Model pre-labelling is a standard feature of current annotation platforms, including Label Studio's pre-annotations, CVAT's automated annotation and V7's Auto-Annotate. Wherever it is turned on, accept latency is the signal that separates review from rubber-stamping.
The event stream
An event carries a timestamp, an action, a geometry kind, and one integer.
| Action | Meaning | The integer |
|---|---|---|
shape_add | A shape was committed | Vertices: 4 for a box, len(points) for a polygon, 0 for a mask |
shape_edit | An existing shape was moved or reshaped | — |
shape_remove | A shape was deleted | — |
stroke | One brush or eraser stroke finished | Stroke length in image pixels |
fill | Flood fill applied | Pixels filled |
zoom | Zoom level changed | Level × 100 |
pan | One pan drag finished | Distance in screen pixels |
tool, undo, redo | — | |
ai_suggest | A suggestion was rendered to the annotator | — |
ai_accept / ai_reject | — |
No coordinates are recorded, ever. A stream reconstructs how an annotation was made; it cannot reconstruct the annotation. That is a structural property of the event record rather than a policy. There is no field a coordinate could go in, and a test asserts it.
Stroke length is in image pixels rather than screen pixels, so the same stroke measures the same whether the annotator was zoomed in or out.
Configuration
annotation_telemetry:
enabled: true
fidelity: events # off | summary | events
store_events: true
include_schemas: [] # empty means every geometry schema
exclude_schemas: []
idle_ms: 120000 # a gap above this is idle, not active
flush_interval_ms: 10000Runnable example: examples/advanced/annotation-telemetry/.
The signal that changes decisions
AI-accept latency. A run of sub-500 ms accepts across many items is rubber-stamping, and no other figure separates that from careful agreement.
Read it as a distribution, not as a threshold. One fast accept is an easy object; a median of 300 ms across two hundred items is a different finding, and it is one you cannot reach from the annotations themselves.
Limits of the signal
This reports process signals. It does not detect fraud, and it does not classify an annotator. A flag is a prompt to look, not a finding, and the base rate matters: in a corpus where most items really are easy, fast is often correct.
The text-side twin is keystroke logging, which reports how a free-text answer was produced. Together they are the honest answer to "how do you know your crowdworkers are not pasting from an LLM" — and the honest answer is a set of process signals with per-project calibration, not a classifier.
Annotators see a recording notice. That is the difference between a research instrument and surveillance, and the disclosure defaults to on.