Drawing Telemetry
How a box, polygon or mask was produced — time per shape, stroke dynamics, revision counts, and AI-suggestion accept latency. The only signal that separates review from rubber-stamping.
Pre-labelling makes annotators faster, and it makes rubber-stamping frictionless. A suggestion appears, the annotator clicks accept, and the resulting dataset is a record of a model agreeing with itself. Nothing in the annotation distinguishes that from careful review. The geometry is identical, and every quality measure looks better, inter-annotator agreement included.
The only place the difference shows up is in the timing.
A human cannot inspect a mask boundary and decide in 300 ms. Once, maybe, if the object was obvious. As a median across many items, it is not fast expertise.
As one-click auto-labelling becomes the norm, this is the signal that separates review from rubber-stamping. It is the most under-appreciated capability in the 2.8 release.
The event stream
An event carries a timestamp, an action, a geometry kind, and one integer.
| Action | Meaning | The integer |
|---|---|---|
shape_add | A shape was committed | Vertices: 4 for a box, len(points) for a polygon, 0 for a mask |
shape_edit | An existing shape was moved or reshaped | — |
shape_remove | A shape was deleted | — |
stroke | One brush or eraser stroke finished | Stroke length in image pixels |
fill | Flood fill applied | Pixels filled |
zoom | Zoom level changed | Level × 100 |
pan | One pan drag finished | Distance in screen pixels |
tool, undo, redo | — | |
ai_suggest | A suggestion was rendered to the annotator | — |
ai_accept / ai_reject | — |
No coordinates are recorded, ever. A stream reconstructs how an annotation was made; it cannot reconstruct the annotation. That is a structural property of the event record rather than a policy. There is no field a coordinate could go in, and a test asserts it.
Stroke length is in image pixels rather than screen pixels, so the same stroke measures the same whether the annotator was zoomed in or out.
Configuration
annotation_telemetry:
enabled: true
fidelity: events # off | summary | events
store_events: true
include_schemas: [] # empty means every geometry schema
exclude_schemas: []
idle_ms: 120000 # a gap above this is idle, not active
flush_interval_ms: 10000Runnable example: examples/advanced/annotation-telemetry/.
The signal that changes decisions
AI-accept latency. A run of sub-500 ms accepts across many items is rubber-stamping, and no other figure separates that from careful agreement.
Read it as a distribution, not as a threshold. One fast accept is an easy object; a median of 300 ms across two hundred items is a different finding, and it is one you cannot reach from the annotations themselves.
Limits of the signal
This reports process signals. It does not detect fraud, and it does not classify an annotator. A flag is a prompt to look, not a finding, and the base rate matters: in a corpus where most items really are easy, fast is often correct.
The text-side twin is keystroke logging, which reports how a free-text answer was produced. Together they are the honest answer to "how do you know your crowdworkers are not pasting from an LLM" — and the honest answer is a set of process signals with per-project calibration, not a classifier.
Annotators see a recording notice. That is the difference between a research instrument and surveillance, and the disclosure defaults to on.