# Writing-Process Detection

Source: https://www.potatoannotator.com/docs/features/writing-process-detection

Writing-process detection reads the typing dynamics captured by [Keystroke Logging](/docs/features/keystroke-logging) and produces named flags with the evidence attached, so you can see which feature values fired each one and throw out the ones you disagree with.

It needs Potato 2.7.2 or later, and it runs only where keystroke logging is already on. Since `keystroke_logging.enabled` defaults to `false`, a project that never opted in has nothing to detect on.

> **Warning:** Flags are evidence for human review. They are not proof of misconduct, and they must never be wired to automatic rejection, payment withholding, or participant bans. A fast, fluent typist producing a clean first draft genuinely resembles transcription. An annotator on a phone genuinely resembles someone pasting. The rules below are built to minimize those collisions, and the collisions still happen. The [false positives](#false-positives) section is part of the feature, not a disclaimer.

## Three tiers

| Tier | What it is | Needs labeled data? | Default |
|---|---|---|---|
| 1. Rules | Six named flags with explicit thresholds and visible evidence | No | On |
| 2. Calibration | Thresholds refitted to your project's own annotators | No | Off |
| 3. Supervised | A classifier you train on your own labels | Yes | Off |

No pre-trained model ships with Potato. There is no labeled corpus in the repository, so any coefficients we shipped would arrive with a validation number we had invented. You get six rules you can read and argue with instead. If you have real labels, tier 3 is the path that reproduces the accuracies in the literature.

## Tier 1: the rules

Every flag returns the feature values that fired it, so you can defend or discard any individual one.

### `paste_dominant`

Severity: suspect. Fires when `pasted_fraction >= 0.5`, meaning half or more of the final text arrived by paste.

Suppressed when every paste was classified as `self` (re-arranging your own draft) or `instance_text` (quoting the passage under annotation).

### `silent_insertion`

Severity: suspect. Fires when `external_insert_ratio >= 0.3`, meaning nearly a third of inserted characters appeared with no corresponding keystroke.

This is the highest-value single signal. Paste, autofill, dictation, and programmatic injection all show up here, even when the paste event itself is suppressed by the page. It is a direct operationalization of Asher et al.'s "keystroke count anomalously low relative to response length".

Suppressed on soft keyboards (`virtual_keyboard`) and during IME composition, where `keydown` is not reliably emitted and every insert would otherwise look silent. The rule uses the source-aware ratio, so legitimate quoting does not count.

### `transcription_rhythm`

Severity: review. Fires when all three of these hold:

- `iki_log_cv <= 0.06`, a metronomic typing rhythm
- `revision_ratio <= 0.02`, essentially no deletion
- fewer than 0.5 pauses of 2 seconds or longer per 100 characters

Conjunctive by design. Any one alone has an innocent reading. Together they are the copy-typing signature Crossley et al. describe: linear, burst-oriented, low variance.

Skipped entirely for responses under 80 characters or 40 keystrokes, where there is no rhythm to speak of.

### `offscreen_composition`

Severity: suspect. Fires when a large externally-sourced insertion (80 characters or more) immediately follows 10 seconds or more away from the page. This is the "switched to ChatGPT, came back, pasted" pattern.

Suppressed when the insertion's source was the passage or the annotator's own text, since stepping away and then quoting the passage is ordinary behavior.

### `implausible_speed`

Severity: review. Fires above 900 characters per minute (roughly 180 wpm) sustained across a whole response.

### `synthetic_input`

Severity: suspect. Fires on any event with `isTrusted === false`, the browser reporting that input was generated by script rather than by a person. Browser automation, injected scripts, and some accessibility tooling all produce this.

### Verdict levels

| Level | Meaning |
|---|---|
| `ok` | No flags fired |
| `review` | At least one `review`-severity flag |
| `suspect` | At least one `suspect`-severity flag |

### Overriding thresholds

```yaml
keystroke_logging:
  detection:
    thresholds:
      paste_dominant.pasted_fraction: 0.4
      silent_insertion.ratio: 0.25
      transcription_rhythm.iki_log_cv: 0.05
      transcription_rhythm.revision_ratio: 0.02
      transcription_rhythm.pause_2s_per_100_chars: 0.5
      offscreen_composition.blur_ms: 15000
      offscreen_composition.insert_chars: 100
      implausible_speed.chars_per_min: 1000
```

Thresholds are evaluated server-side and never sent to the browser. Publishing them would tell an annotator exactly how slowly to paste in order to stay under the flag.

## Tier 2: project calibration

Keystroke features vary substantially by writing task (Conijn et al. 2019), and fixed thresholds on a mixture-distributed quantity are biased (Roeser et al. 2021). A cutoff that works for a one-sentence rationale is wrong for a five-paragraph essay.

Calibration sets each threshold at a tail percentile of your own project's sessions.

```bash
# Inspect the fit without saving it
python -m potato.typing_detect calibrate config.yaml --dry-run

# Save it
python -m potato.typing_detect calibrate config.yaml
```

```yaml
keystroke_logging:
  detection:
    calibrate: true    # use the saved fit
```

Calibration requires at least 30 usable sessions (80 characters or more, non-mobile). Below that it returns `insufficient_data` and the defaults stay in force.

> **Warning:** A percentile cutoff is a relative outlier definition. By construction it flags roughly `tail_fraction` (default 5%) of sessions even in a population where nobody is doing anything wrong. It tells you where to look first. It is not evidence. Calibrated thresholds are additionally clamped to within 3x the built-in defaults, so a homogeneous population cannot drag a cutoff onto its own median and start flagging honest annotators.

An explicit `thresholds:` override always beats a calibrated value, which always beats the built-in default.

## Tier 3: supervised classifier

If you have labeled sessions, train a real model:

```python
from potato import typing_store
from potato.typing_detect import fit_supervised

rows = typing_store.feature_matrix(task_dir, project)
labels = [...]   # 1 = non-composed, 0 = composed

result = fit_supervised(rows, labels, model="random_forest")
print(result["cv_accuracy_mean"], result["feature_importances"])
```

Requires `scikit-learn`, which is imported lazily and is not a Potato dependency.

### Getting labels without an external study

Potato's [training phase](/docs/features/training-phase) can generate them for you. Ask annotators to copy a supplied passage as a warm-up task. Their copying sessions are genuine transcription exemplars and their normal answers are composed exemplars, giving you a labeled set from inside your own project, on your own task, with your own annotators. This mirrors how Crossley et al. built their corpus, and the [calibration example](https://github.com/davidjurgens/potato/tree/master/examples/advanced/keystroke-calibration) sets it up end to end.

## False positives

The failure mode of this feature is accusing an honest annotator. These are the cases that legitimately resemble the patterns above.

| Situation | Which rule it resembles | How Potato handles it |
|---|---|---|
| Quoting the passage under annotation | `paste_dominant`, `silent_insertion`, `offscreen_composition` | Suppressed via `paste_source: instance_text` |
| Moving your own draft around | Same | Suppressed via `paste_source: self` |
| Typing on a phone or tablet | `silent_insertion` | Suppressed via `virtual_keyboard` |
| Non-Latin input via an IME | `silent_insertion` | Suppressed via `composition_events` |
| Fast, fluent typist, clean draft | `transcription_rhythm`, `implausible_speed` | Conjunctive rules; `review` not `suspect`; calibration |
| Very short answers | `transcription_rhythm` | Skipped below 80 chars / 40 keystrokes |
| Dictation or speech-to-text | `silent_insertion` | Not handled. Will flag. Exclude those annotators or raise the threshold. |
| Screen readers and some assistive tech | `synthetic_input` | Not fully handled. Some tooling produces untrusted events. |
| Autocorrect on mobile | `silent_insertion` | Partly. `insertReplacementText` counts as external. |
| Browser extensions such as grammar tools | `silent_insertion`, `synthetic_input` | Not handled. Will flag. |

The last four are real limitations rather than oversights. If your participant pool includes dictation users, assistive-technology users, or people who use writing extensions, either drop those rules from your review criteria or treat every flag as a prompt to ask the annotator rather than to act.

### Accessibility

`synthetic_input` keys off `isTrusted`, which some assistive technologies also trip. Flagging a disabled annotator for using the tools they need to work is an unacceptable outcome.

That rule has no threshold, so there is no value you can set to make it unreachable. If your study is open to assistive-technology users, either exclude `synthetic_input` from whatever review criteria you write down, or turn detection off entirely and analyze the exported features yourself:

```yaml
keystroke_logging:
  enabled: true
  detection:
    enabled: false
```

Threshold-based rules can be made unreachable by setting an absurd cutoff, for example `implausible_speed.chars_per_min: 1000000`.

## Real-time intervention

```yaml
keystroke_logging:
  detection:
    on_external_insert: flag   # allow | warn | block | flag
```

| Value | Behavior |
|---|---|
| `allow` | Do nothing beyond recording |
| `warn` | Non-blocking notice on external paste, but not on self or passage quotes |
| `block` | Prevent paste and drop into instrumented fields |
| `flag` | Default. Record silently; you decide later |

`block` is a blunt instrument. It also blocks legitimate quoting, and a determined participant can retype instead. Using `flag` and then reviewing is usually better.

## Admin dashboard

The Writing Process panel under the Behavioral tab of the [Admin Dashboard](/docs/features/admin-dashboard) shows per-annotator medians, paste rates, silent-insertion rates, and every flagged session with its evidence.

It reports `writing_process_risk`, the share of a user's sessions that fired each flag, weighted toward the signals with the least innocent explanation. This is a ranking aid, deliberately kept separate from the existing `suspicion_score` so that neither number silently changes the other's meaning.

## Research grounding

Every citation below is verified against the Crossref or DataCite registry.

Crossley, Tian, Choi, Holmes and Morris (2024) collected 500 argumentative essays, had other workers transcribe them, and separated authentic from transcribed at 99% accuracy with a random forest (96-98% for other models). Their feature families are what Potato captures: pause times before sentences and words, insertion and deletion counts, product-to-process ratios, bursts, revision, and process variance. Their finding is the design target. Authentic writing shows longer pauses, more insertions and deletions, and greater variance; transcription is linear and burst-oriented.

Deane, Zhang, Hao and Li, and separately Zhang, Feng, He, Li and Zhu, independently confirm the copy-typing versus natural-writing separation, the latter with a deep-learning model. Asher, Gold, Chen and Carvalho is the crowdsourcing-specific case: a keystroke tool on Prolific flagging participants who paste into response fields or whose keystroke count is anomalously low for their response length.

For the underlying process measures, see Leijten and Van Waes (Inputlog) for the standard log measures, Chenoweth and Hayes for the burst construct, Conijn, Roeser and van Zaanen for task-dependence of keystroke features, and Roeser, De Maeyer, Leijten and Van Waes for why fixed pause thresholds are biased.

### References

1. Crossley, S., Tian, Y., Choi, J. S., Holmes, L., & Morris, W. (2024). *Plagiarism Detection Using Keystroke Logs*. EDM 2024 (Short Papers). [doi:10.5281/zenodo.12729864](https://doi.org/10.5281/zenodo.12729864)
2. Deane, P., Zhang, M., Hao, J., & Li, C. *Using Keystroke Dynamics to Detect Nonoriginal Text*. Journal of Educational Measurement, 63(1). [doi:10.1111/jedm.12431](https://doi.org/10.1111/jedm.12431)
3. Asher, M. W., Gold, G., Chen, E., & Carvalho, P. F. (2026). *Chatbots Are Undermining Crowdsourced Research in the Behavioral Sciences: Detecting Artificial Intelligence-Assisted Cheating With a Keystroke-Based Tool*. Advances in Methods and Practices in Psychological Science, 9(1). [doi:10.1177/25152459261424723](https://doi.org/10.1177/25152459261424723)
4. Zhang, M., Feng, L., He, X., Li, C., & Zhu, M. (2026). *Disentangling copy typing and natural writing behaviors using keystroke logs and deep learning model*. Assessing Writing. [doi:10.1016/j.asw.2026.101070](https://doi.org/10.1016/j.asw.2026.101070)
5. Leijten, M., & Van Waes, L. (2013). *Keystroke Logging in Writing Research: Using Inputlog to Analyze and Visualize Writing Processes*. Written Communication, 30(3), 358-392. [doi:10.1177/0741088313491692](https://doi.org/10.1177/0741088313491692)
6. Chenoweth, N. A., & Hayes, J. R. (2001). *Fluency in Writing: Generating Text in L1 and L2*. Written Communication, 18(1), 80-98. [doi:10.1177/0741088301018001004](https://doi.org/10.1177/0741088301018001004)
7. Conijn, R., Roeser, J., & van Zaanen, M. (2019). *Understanding the keystroke log: the effect of writing task on keystroke features*. Reading and Writing, 32(9), 2353-2374. [doi:10.1007/s11145-019-09953-8](https://doi.org/10.1007/s11145-019-09953-8)
8. Roeser, J., De Maeyer, S., Leijten, M., & Van Waes, L. (2021). *Modelling typing disfluencies as finite mixture process*. Reading and Writing. [doi:10.1007/s11145-021-10203-z](https://doi.org/10.1007/s11145-021-10203-z)
9. Lee, M., Liang, P., & Yang, Q. (2022). *CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities*. CHI 2022. [doi:10.1145/3491102.3502030](https://doi.org/10.1145/3491102.3502030)

## Further Reading

- [Keystroke Logging](/docs/features/keystroke-logging) - what is captured and how
- [Keystroke Logging Ethics](/docs/features/keystroke-logging-ethics) - IRB, consent, participant rights
- [Quality Control](/docs/features/quality-control) - attention checks and gold standards
- [Admin Dashboard](/docs/features/admin-dashboard) - the Writing Process panel

For implementation details, see the [source documentation](https://github.com/davidjurgens/potato/blob/main/docs/advanced/writing_process_detection.md).