Timed Trials: Fixation, Deadlines and Reaction Times

New in v2.12.0

trial_timing runs each annotation item as a timed trial: an optional fixation cross, the stimulus for a set time, a response window with a deadline, and keys mapped to labels. The reaction time is measured in the browser from the frame on which the stimulus appeared.

It suits speeded judgments on Potato's own items, such as lexical decision, grammaticality or plausibility judgments under time pressure, and rapid image categorization. Frame-critical presentation, auditory onsets and paradigms that need code on every trial belong on a jsPsych page. To show an item's text one word or region at a time, as in self-paced reading, RSVP or the maze task, add a presentation block (Self-Paced Reading and Maze).

Configuration

annotation_schemes:
  - annotation_type: radio
    name: decision
    description: Is this a word?
    labels: [word, nonword]
    label_requirement:
      required: true
 
trial_timing:
  enabled: true
  fixation_ms: 500
  stimulus_duration_ms: 200       # omit: the stimulus stays up until the response
  trial_duration_ms: 2000         # omit: no deadline
  iti_ms: 500
  response:
    schema: decision
    keys: {f: word, j: nonword}

Trial sequence

1. A blank interval, until iti_ms has passed since the last response.
2. The fixation cross, for fixation_ms.
3. The item, with a line under it listing the keys, such as F = word J = nonword.
4. A mapped key selects its label, saves, and moves to the next item. Other keys do nothing, and a mapped key pressed before the stimulus is counted as an early press and ignored.
5. At the deadline, the item is recorded as a timeout and the next item starts.

The normal annotation page's keyboard shortcuts (arrow keys and number keys) are off while a trial runs.

Trial record

Each item gets a trial record, stored in user_state.json beside the labels and written to trials.csv (or .tsv) by the CSV and TSV exports:

A timed-out item has no label, so trials.csv is the complete record of a timed task. The main annotation export lists answered items only.

Server checks

The browser reports the record, and the server compares it with its own clock and with the configuration, using the item's own timing where item_overrides set it:

• rt_exceeds_server_elapsed: the reaction time is longer than the item was out.
• timeout_too_early: a timeout arrived before the deadline could have passed.
• timeout_without_deadline: a timeout on an item that has no deadline.
• not_served: the server has no record of handing the item over, for example after a restart, so it cannot check the timing.
• key_not_mapped, label_mismatch, label_not_in_scheme: the key is not in response.keys, does not map to the label reported, or the label is not one of the scheme's.
• label_not_saved: the record's label is not the answer the same save stored.
• tab_hidden: the tab was hidden during the trial.

A timeout lets the participant move on without an answer only when it carries none of timeout_too_early, timeout_without_deadline and not_served. Otherwise the item stays unanswered and required.

Timeouts, quotas and attention checks

A timed-out item is finished for the participant: it counts toward their progress, and they are not shown it again. It is not an annotation of the item, so it does not count toward max_annotations_per_item, and other participants can still be assigned the item. The progress badge shows it as not labeled.

A trial that runs out on an attention check is not graded as a failure, and it appears in trials.csv as a timeout. A speeded task produces timeouts from attentive participants too, and blocking them for it would remove attentive people because of the deadline.

Precision

The stimulus is revealed inside a requestAnimationFrame callback, and onset is the timestamp of the next frame, the first one drawn with the stimulus. The reaction time is the key event's timestamp minus that onset, and both come from the browser's high-resolution clock.

• Onset is known to the frame: about 17 ms at 60 Hz and 8 ms at 120 Hz.
• Keyboards and displays add latency the browser cannot see, commonly 5-30 ms and different on every machine.
• Browsers that coarsen timers for privacy, such as Firefox with privacy.resistFingerprinting and Tor Browser, round to tens of milliseconds. timer_resolution_ms shows this per participant.
• Each item is a page load, so fullscreen cannot be held across items.

Compare reaction times within participants, or model them with a participant term. For millisecond-level work, or for audio, use a jsPsych page.

Limits

• Phones and tablets are sent to the touch interface (Pocket Mode), which does not run timed trials. The server logs a warning when both are on.
• The response scheme must be radio or likert, and it must be the only question on the annotation page, although display-only schemes are allowed. A second question cannot be answered while a trial holds the keyboard, so the validator reports it as an error. Ask follow-up questions on a poststudy page.
• The MySQL storage backend does not store trial records.

Example

The Potato repository ships a lexical decision task. Start it from the repository root:

python potato/flask_server.py start examples/advanced/timed-trials-lexical-decision/config.yaml -p 8000

Further Reading

• Self-Paced Reading and Maze - Word-by-word presentation with per-region reading times
• Runtime Phases - jsPsych timelines for frame-critical tasks
• Experiment Flow - Conditions, screen-outs and counterbalancing
• Behavioral Tracking - Timing and interaction data on ordinary annotation pages

For implementation details, see the source documentation.