# Reading the Writing Process: Keystroke Logging for Free-Text Annotation

Source: https://www.potatoannotator.com/blog/keystroke-logging-writing-process

A crowdworker opens your annotation task, reads the passage, switches to another tab, comes back twenty seconds later, and a 280-character rationale appears in the box in one motion.

The rationale is fine. It is on topic, it is grammatical, it references the passage. Read it next to forty others and nothing stands out. Every classifier you might point at it will tell you something between "probably human" and "unclear," which is what those classifiers say about most text.

The finished answer does not tell you where it came from. The twenty seconds and the single motion do.

Potato now records that. Free-text fields can capture a content-blind stream of how a response was produced, summarize it into around forty features, and run a small set of named rules over the result. It is off by default and turns on with one line:

```yaml
keystroke_logging:
  enabled: true
```

![Keystroke logging in Potato: composed, transcribed, and pasted responses look different in the process even when the finished text does not](/images/blog/keystroke-logging-banner.svg "Keystroke logging")

## Composed, transcribed, pasted

The research this rests on is fairly settled. Crossley and colleagues collected 500 argumentative essays, had a second set of workers transcribe them, and separated authentic writing from transcription at 99% accuracy with a random forest. Deane et al. and Zhang et al. found the same separation independently. Asher et al. built the crowdsourcing version on Prolific, flagging participants whose keystroke count was too low for the length of what they submitted.

The signature is consistent across all of them. Real composition has longer pauses before sentences and words, more insertions and deletions, and high variance in the intervals between keys. Copy-typing is linear, burst-oriented, and low variance. Pasting barely has intervals at all.

None of that is visible in the text. All of it is visible in the log.

## Why `beforeinput` and not `keydown`

This is the one technical decision that determines whether the rest of the feature works.

The obvious way to write a keystroke logger is to listen for `keydown`. It is also the way that fails, because paste, drag-and-drop, IME composition, dictation, autofill, and undo all change the contents of a field without firing `keydown` even once. A keydown-only logger is blind to exactly the cases this exists to catch.

Potato's primary signal is `InputEvent.inputType` on `beforeinput`, which fires for all of them and says which one happened. `keydown` and `keyup` are still listened to, but for a different purpose: counting the keys a person physically pressed.

The gap between those two numbers is the most useful thing collected. Characters that appeared in the field with no corresponding keystroke are recorded as `silent_insert_chars`, and their share of the response as `silent_insert_ratio`. A paste that the page suppressed, an extension that injected text, a dictation stream, a script filling the box: none of them produce keystrokes, all of them produce characters.

## What is recorded, and what is not

Each event carries a timestamp, an input type, a key *class*, a caret position, and a length delta:

```
{t_ms: 1240, input_type: "insertText",            key_class: "letter", pos: 41, delta: +1}
{t_ms: 3980, input_type: "deleteContentBackward", key_class: "bksp",   pos: 42, delta: -1}
{t_ms: 9120, input_type: "insertFromPaste",       key_class: "unknown",pos: 43, delta: +287,
    meta: {paste_source: "external", paste_hash: "sekqf3"}}
```

The key itself is never stored, only which family it belongs to: `letter`, `digit`, `punct`, `space`, `enter`, `bksp`, `del`, `nav`, `mod`, `func`, `unknown`. Pasted text is reduced to a length, a source label, and a per-session salted hash. Password fields are refused outright. You cannot reconstruct the response from the stream, which is the point: the stream describes the process and says nothing about the content.

Paste source classification is what keeps ordinary behavior out of the flags. When a paste arrives, Potato compares it against the passage under annotation, any AI suggestion shown on the page, and what was already in the field, then keeps the label and discards the comparison. Quoting the passage reads as `instance_text`. Moving your own draft around reads as `self`. Neither counts as external insertion.

## Six rules, and nothing pretending to be a model

Detection runs server-side in three tiers. Only the first is on by default.

The first tier is six named flags, each with an explicit threshold, each returning the feature values that fired it:

| Flag | Fires when | Severity |
|---|---|---|
| `paste_dominant` | Half or more of the final text arrived by paste | suspect |
| `silent_insertion` | ≥30% of inserted characters had no keystroke behind them | suspect |
| `transcription_rhythm` | Metronomic rhythm *and* no revision *and* almost no pauses | review |
| `offscreen_composition` | A large external insertion right after ≥10s away from the page | suspect |
| `implausible_speed` | Sustained above ~180 wpm across a whole response | review |
| `synthetic_input` | The browser reports `isTrusted === false` | suspect |

`transcription_rhythm` is conjunctive on purpose. Metronomic typing on its own is a fast typist. Never deleting anything is a careful one. Barely pausing is a short answer. Only when all three hold at once do you have the copy-typing signature, and the rule skips responses under 80 characters entirely, where there is no rhythm to read.

The second tier is calibration. Keystroke features depend heavily on the writing task, so a threshold tuned for a one-sentence rationale is wrong for five paragraphs. `python -m potato.typing_detect calibrate config.yaml` refits each cutoff to a tail percentile of your own project's sessions. It needs at least 30 usable sessions, and calibrated values are clamped to within 3× the built-in defaults so a homogeneous population cannot drag a threshold onto its own median.

The third is supervised. If you have labels, `fit_supervised()` trains a real classifier on the feature matrix. `scikit-learn` is imported lazily and is not a Potato dependency.

No pre-trained model ships. There is no labeled corpus in the repository, and shipping fitted coefficients derived from nothing would be inventing a validation number. What ships instead is six rules you can read, disagree with, and override.

If you want labels, the training phase can produce them from inside your own study. Give annotators a copy-the-passage warm-up: those sessions are genuine transcription exemplars from your annotators on your task, and their ordinary answers are the composed class. That is how the Crossley corpus was built, and the [calibration example](https://github.com/davidjurgens/potato/tree/master/examples/advanced/keystroke-calibration) sets it up end to end.

Thresholds are evaluated on the server and never sent to the browser. Publishing them would tell an annotator precisely how slowly to paste.

## Where it goes

Raw streams go to SQLite, in `<task_dir>/project.sqlite`, one row per session, through the same persistence layer as memos and the codebook. Events are delta-encoded and zlib-packed at a measured 1.7 bytes per event, so a 500-word response costs about 5 KB.

They deliberately do not go into `user_state.json`. That file is rewritten in full on every annotation save, and a long response is around 3,000 events. Only the compact summary mirrors into behavioral data, keyed `"{schema}:::{label}"`, so it travels with the annotation into the dashboard and the exports.

Free-text answers in the training phase and in prestudy or poststudy surveys are captured too. Those pages have no instance id, so their sessions bucket under the existing `__phase_page__` sentinel and are identified by `phase` and `page` instead. That is what makes the copy-the-passage trick work: transcription exemplars are separable from ordinary answers by phase alone.

Both exports are opt-in. `export_include_typing_dynamics: true` writes a `typing_dynamics.csv` sidecar next to your annotations, and `python -m potato.export.cli <config.yaml> --format keystrokes` writes the raw streams to Parquet, falling back to JSONL without `pyarrow`. Behavioral data is never included in a dataset release by accident.

## Before you point this at people

A flag is evidence for a human to review. It is not proof, and it must not be wired to automatic rejection, payment withholding, or bans.

The failure mode here is accusing an honest annotator, and the cases that trip the rules are not exotic. Some are handled: quoting the passage and rearranging your own draft are suppressed by source classification, and soft keyboards and IME composition suppress `silent_insertion`, since neither reliably emits `keydown` and every insert would otherwise look silent. Some are not. Dictation will flag. Grammar extensions will flag. Some assistive technology produces untrusted events and will trip `synthetic_input`, which is why the documentation says plainly that if your study is open to AT users you should turn that rule off rather than explain a flag to someone who was using the tools they need to work.

Then there is the arithmetic. If 5% of your responses are pasted and your rule flags 5% of sessions, most of what you flag can still be honest work. On a platform where misconduct is genuinely rare, a rule with even a modest false-positive rate produces more false accusations than true catches. A calibrated threshold makes this explicit rather than better: a tail percentile flags its tail fraction of *any* population, including one where nobody did anything wrong.

Disclosure is on by default, and turning it off logs a warning at startup. That default exists because timing patterns are a behavioral biometric. They can identify a person and link accounts across contexts, and the research literature has used them to infer typing skill, second-language status, and cognitive load. Nobody who signed up to label sentences was expecting that. Potato computes none of it and ships no tooling for it, but the data you keep would support the analysis, which is your problem to manage rather than the tool's. `typing_store.delete_for_user()` removes one participant's streams, and `fidelity: summary` keeps the features while dropping the biometric detail. The [ethics page](/docs/features/keystroke-logging-ethics) has sample consent language, retention guidance, and notes on GDPR Article 22, IRB review, and platform rejection policies.

`writing_process_risk` shows up in the admin dashboard's Writing Process panel as a ranking aid, separate from the existing `suspicion_score`. Neither number folds into the other, so neither quietly changes what the other means.

## Turning it on

The feature ships in Potato 2.7.2. A fuller config than the one-liner at the top:

```yaml
keystroke_logging:
  enabled: true
  fidelity: events              # off | summary | events
  include_schemas: [rationale]  # empty means every free-text field
  disclose_to_annotators: true
  detection:
    enabled: true
    on_external_insert: flag    # allow | warn | block | flag
```

`on_external_insert: block` prevents pasting into instrumented fields. It also blocks legitimate quoting, and anyone determined can retype instead, so `flag` and review is usually the better trade.

A runnable project lives at `examples/advanced/keystroke-logging/`, and the calibration walkthrough at `examples/advanced/keystroke-calibration/`.

## Documentation

- [Keystroke Logging](/docs/features/keystroke-logging) — every captured field, the summary features, storage, and troubleshooting
- [Writing-Process Detection](/docs/features/writing-process-detection) — the six rules, the three tiers, the false-positive table, and the citations
- [Keystroke Logging Ethics](/docs/features/keystroke-logging-ethics) — consent, IRB, retention, participant rights
- [Behavioral Tracking](/docs/features/behavioral-tracking) — the wider interaction-tracking system this sits inside
- [Quality Control](/docs/features/quality-control) — attention checks and gold standards
- [Admin Dashboard](/docs/features/admin-dashboard) — where the Writing Process panel lives
- [Crowdsourcing on Prolific and MTurk](/docs/guides/crowdsourcing-prolific-mturk) — platform rules on monitoring and rejection

## Upgrading

```bash
pip install --upgrade potato-annotation==2.7.2
```

Nothing changes for an existing project until you ask for it. `keystroke_logging.enabled` defaults to `false`, so upgrading never starts recording anyone.