Reading the Writing Process: Keystroke Logging for Free-Text Annotation

Potato can now record how annotators produce free-text answers without recording what they type, and flag the responses that were pasted rather than written. Free-text fields capture a content-blind stream of how a response was produced, summarize it into about forty features, and run a small set of named rules over the result.

Consider a crowdworker who opens your annotation task, reads the passage, and switches to another tab, and twenty seconds later a 280-character rationale appears in the box in one motion. The rationale is fine. It is on topic, it is grammatical, it references the passage, and read next to forty others nothing stands out. A classifier that reads only the finished text has nothing unusual to flag in it. The finished answer does not tell you where it came from, and the twenty seconds and the single motion do.

Keystroke logging is off by default and turns on with one line:

keystroke_logging:
  enabled: true

Keystroke logging

Research on composed, transcribed, and pasted text

Four studies published between 2024 and 2026 used keystroke logs to tell composed text from copied text. Crossley and colleagues collected 500 argumentative essays, had a second set of workers transcribe them, and separated authentic writing from transcription at 99% accuracy with a random forest. Deane et al. and Zhang et al. found the same separation independently. Asher et al. built the crowdsourcing version on Prolific, flagging participants whose keystroke count was too low for the length of what they submitted.

The signature is consistent across all of these studies. Real composition has longer pauses before sentences and words, more insertions and deletions, and high variance in the intervals between keys. Copy-typing is linear, burst-oriented, and low variance, and pasting barely has intervals at all. None of these differences is visible in the finished text, and all of them are in the log.

Capturing on beforeinput rather than keydown

The choice of browser event determines whether the rest of the feature works. The obvious way to write a keystroke logger is to listen for keydown, and it fails, because paste, drag-and-drop, IME composition, dictation, autofill, and undo all change the contents of a field without firing keydown even once. A keydown-only logger is blind to the cases keystroke logging exists to catch.

Potato's primary signal is InputEvent.inputType on beforeinput, which fires for all of them and says which one happened. Potato still listens to keydown and keyup, but only to count the keys a person physically pressed.

The detection rules lean on the gap between those two numbers. Characters that appeared in the field with no corresponding keystroke are recorded as silent_insert_chars, and their share of the response as silent_insert_ratio. A paste that the page suppressed, an extension that injected text, a dictation stream, and a script filling the box all produce characters without producing a single keystroke.

Event fields and paste classification

Each event carries a timestamp, an input type, a key class, a caret position, and a length delta:

{t_ms: 1240, input_type: "insertText",            key_class: "letter", pos: 41, delta: +1}
{t_ms: 3980, input_type: "deleteContentBackward", key_class: "bksp",   pos: 42, delta: -1}
{t_ms: 9120, input_type: "insertFromPaste",       key_class: "unknown",pos: 43, delta: +287,
    meta: {paste_source: "external", paste_hash: "sekqf3"}}

Potato never stores the key itself, only its family, which is one of letter, digit, punct, space, enter, bksp, del, nav, mod, func, or unknown. Pasted text is reduced to a length, a source label, and a per-session salted hash, and password fields are refused outright. By design, you cannot reconstruct the response from the stream, which describes the process and says nothing about the content.

Paste source classification keeps ordinary behavior out of the flags. When a paste arrives, Potato compares it against the passage under annotation, any AI suggestion shown on the page, and what was already in the field, then keeps the label and discards the comparison. Quoting the passage reads as instance_text, and moving your own draft around reads as self. Neither counts as external insertion.

Detection tiers and the six flags

Detection runs server-side in three tiers, and only the first is on by default. Thresholds are evaluated on the server and never sent to the browser, because publishing them would tell an annotator precisely how slowly to paste. The first tier is six named flags, each with an explicit threshold, and each returns the feature values that fired it:

transcription_rhythm is conjunctive on purpose. Metronomic typing on its own is a fast typist, never deleting anything is a careful one, and barely pausing is a short answer. Only when all three hold at once do you have the copy-typing signature, and the rule skips responses under 80 characters entirely, where there is no rhythm to read.

The second tier is calibration. Keystroke features depend heavily on the writing task, so a threshold tuned for a one-sentence rationale is wrong for five paragraphs. python -m potato.typing_detect calibrate config.yaml refits each cutoff to a tail percentile of your own project's sessions. It needs at least 30 usable sessions, and calibrated values are clamped to within 3× the built-in defaults so a homogeneous population cannot drag a threshold onto its own median.

The third tier is supervised. If you have labels, fit_supervised() trains a real classifier on the feature matrix, and scikit-learn is imported lazily and is not a Potato dependency. No pre-trained model ships, because there is no labeled corpus in the repository, and shipping fitted coefficients derived from nothing would be inventing a validation number. Potato ships six rules you can read, disagree with, and override instead.

If you want labels, the training phase can produce them from inside your own study. Give annotators a copy-the-passage warm-up. Those sessions are genuine transcription exemplars from your annotators on your task, and their ordinary answers are the composed class. Crossley and colleagues built their corpus the same way, and the calibration example sets it up end to end.

Storage and export of keystroke data

Raw streams go to SQLite, in <task_dir>/project.sqlite, one row per session, through the same persistence layer as memos and the codebook. Events are delta-encoded and zlib-packed at a measured 1.7 bytes per event, so a 500-word response costs about 5 KB. Raw streams deliberately stay out of user_state.json, because that file is rewritten in full on every annotation save and a long response is about 3,000 events. Only the compact summary mirrors into behavioral data, keyed "{schema}:::{label}", so it travels with the annotation into the dashboard and the exports.

Free-text answers in the training phase and in prestudy or poststudy surveys are captured too. Those pages have no instance id, so their sessions bucket under the existing __phase_page__ sentinel and are identified by phase and page instead. The phase bucketing makes the copy-the-passage warm-up work, because transcription exemplars are separable from ordinary answers by phase alone.

Both exports are opt-in. export_include_typing_dynamics: true writes a typing_dynamics.csv sidecar next to your annotations, and python -m potato.export.cli <config.yaml> --format keystrokes writes the raw streams to Parquet, or to JSONL without pyarrow. Behavioral data is never included in a dataset release by accident.

False positives and disclosure to annotators

A flag is evidence for a human to review rather than proof, and it must not be wired to automatic rejection, payment withholding, or bans. writing_process_risk shows up in the admin dashboard's Writing Process panel as a ranking aid, separate from the existing suspicion_score. Neither number folds into the other, so neither changes what the other means.

The failure mode here is accusing an honest annotator, and the cases that trip the rules are not exotic. Source classification suppresses quoting the passage and rearranging your own draft, and soft keyboards and IME composition suppress silent_insertion, since neither reliably emits keydown and every insert would otherwise look silent. Dictation and grammar extensions will still flag. Some assistive technology (AT) produces untrusted events and will trip synthetic_input, which is why the documentation says plainly that if your study is open to AT users you should turn that rule off rather than explain a flag to someone who was using the tools they need to work.

If 5% of your responses are pasted and your rule flags 5% of sessions, most of what you flag can still be honest work. On a platform where misconduct is genuinely rare, a rule with even a modest false-positive rate produces more false accusations than true catches. A calibrated threshold makes the problem explicit without making it better, because a tail percentile flags its tail fraction of any population, including one where nobody did anything wrong.

Disclosure is on by default, and turning it off logs a warning at startup. The default exists because timing patterns are a behavioral biometric. They can identify a person and link accounts across contexts, and the research literature has used them to infer typing skill, second-language status, and cognitive load, which nobody who signed up to label sentences was expecting. Potato computes none of that and ships no tooling for it, but the data you keep would support the analysis, which makes it your problem to manage rather than the tool's.

typing_store.delete_for_user() removes one participant's streams, and fidelity: summary keeps the features while dropping the biometric detail. The ethics page has sample consent language, retention guidance, and notes on GDPR Article 22, IRB review, and platform rejection policies.

Turning on keystroke logging

The feature ships in Potato 2.7.2. A fuller config than the one-liner at the top looks like this:

keystroke_logging:
  enabled: true
  fidelity: events              # off | summary | events
  include_schemas: [rationale]  # empty means every free-text field
  disclose_to_annotators: true
  detection:
    enabled: true
    on_external_insert: flag    # allow | warn | block | flag

on_external_insert: block prevents pasting into instrumented fields. It also blocks legitimate quoting, and anyone determined can retype instead, so flag and review is usually the better trade. A runnable project lives at examples/advanced/keystroke-logging/, and the calibration walkthrough at examples/advanced/keystroke-calibration/.

Upgrading

Nothing changes for an existing project until you ask for it. keystroke_logging.enabled defaults to false, so upgrading never starts recording anyone. Install the release with:

pip install --upgrade potato-annotation==2.7.2

Documentation

The documentation covers each part in more depth:

• Keystroke Logging, for every captured field, the summary features, storage, and troubleshooting
• Writing-Process Detection, for the six rules, the three tiers, the false-positive table, and the citations
• Keystroke Logging Ethics, for consent, IRB, retention, and participant rights
• Behavioral Tracking, the wider interaction-tracking system this feature sits inside
• Quality Control, for attention checks and gold standards
• Admin Dashboard, where the Writing Process panel lives
• Crowdsourcing on Prolific and MTurk, for platform rules on monitoring and rejection

References

Scott Crossley et al. (2024). Plagiarism Detection Using Keystroke Logs. Proceedings of the 17th International Conference on Educational Data Mining. https://doi.org/10.5281/zenodo.12729864

Paul Deane et al. (2025). Using Keystroke Dynamics to Detect Nonoriginal Text. Journal of Educational Measurement. https://doi.org/10.1111/jedm.12431

Mo Zhang et al. (2026). Disentangling copy typing and natural writing behaviors using keystroke logs and deep learning models. Assessing Writing. https://doi.org/10.1016/j.asw.2026.101070

Michael W. Asher et al. (2026). Chatbots Are Undermining Crowdsourced Research in the Behavioral Sciences: Detecting Artificial Intelligence–Assisted Cheating With a Keystroke-Based Tool. Advances in Methods and Practices in Psychological Science. https://doi.org/10.1177/25152459261424723