Running Online Behavioral Experiments in Potato

An online behavioral experiment in Potato is a YAML config that lists the session's pages in order, assigns each participant to a condition, and records every trial with its timing. From Potato 2.12.0, the config covers between-subjects conditions, Latin-square lists and block orders, timed trials with reaction times, self-paced reading, screen-outs, and the return to Prolific or SONA, and a jsPsych timeline can run as one page of the session.

This guide is for researchers in psychology, cognitive science and psycholinguistics who would otherwise build the study in jsPsych, PsychoPy, Gorilla, Ibex or Qualtrics. It uses their terms:

• A trial is one presentation of a stimulus and the response to it. In Potato a trial is usually one item from a data file.
• A between-subjects factor gives each participant one level. A within-subjects factor shows every participant every level, usually in a counterbalanced order.
• Reaction time (RT) runs from stimulus onset to the response.
• A screen-out ends the session for a participant who does not meet the inclusion criteria, and a completion code tells the recruitment platform that the participant finished.

Behavioral Experiment Tools Compared sets Potato beside the other tools, and Coming from jsPsych maps jsPsych concepts to config keys.

Parts of a study and the Potato key for each

Assigning participants to conditions

conditions assigns each participant one cell of the design the first time the server sees them, and the participant keeps it. Assignment is balanced by default, which sends each arrival to the cell with the fewest participants. balance_on: completed counts only participants who finished without being screened out, so a dropout is replaced by the next arrival and the cells stay even.

For a design in which each item exists in several versions and each participant sees one version of each item, make the list a factor and give each level its own data file. The participant then sees only the items of their list:

conditions:
  factors:
    list: [list_a, list_b]
  balance_on: completed
 
assignment_strategy: batch
batch_assignment:
  auto_assign_annotators: true
  group_by_condition: list
  groups:
    - {name: list_a, data_file: data/list_a.jsonl}
    - {name: list_b, data_file: data/list_b.jsonl}

Item IDs must be unique across the files, so the two versions of one item need different IDs, such as s1_a and s1_b. An Ibex or PCIbex experiment imports into exactly this layout (Coming from Ibex).

Choosing how trials run

Potato has two ways to run trials, and the choice depends on what the trial has to do.

Timed trials on items (trial_timing) fit tasks in which each trial shows one stimulus from a data file and takes one response: lexical decision, acceptability or plausibility judgments under a deadline, rapid categorization of images. Each trial gets a fixation cross, a stimulus duration, a deadline and keys mapped to labels, and the RT is measured from the animation frame on which the stimulus was drawn. Onset is known to the frame, about 17 ms at 60 Hz. Self-paced reading and the maze task use the same mechanism, with a reading time per region.

A jsPsych page fits tasks that need code on each trial: canvas drawing, masks timed to a frame, audio scheduled with Web Audio, mouse trajectories, loops and branches inside a block, or fullscreen held across trials. The page runs jsPsych 8.3.0 alone, and the timeline reads the participant's conditions from potato.conditions.

Browser RTs carry latency from keyboards and displays that differs between machines, commonly 5-30 ms. Compare RTs within participants, or model them with a participant term.

Screening participants

A screener is a prestudy page with exit_when. When its rule holds, the participant goes to the end of the session, and with a recruitment platform configured they get the platform's screen-out outcome instead of the completion code. Previous does not take them back.

Listening studies can add a headphone check, which the server scores, so a participant cannot pass by editing what the browser sends. Second-language studies can add a cloze test or a LexTALE-style vocabulary test. Potato does not bundle the LexTALE item lists, so supply them under their authors' terms. The vocabulary score is computed at export, so screening on it means inviting qualifying participants to a second study.

Attention checks inside the task are a separate mechanism (Quality Control). A timed trial that runs out on an attention check is recorded as a timeout and is not graded as a failure, because a speeded task produces timeouts from attentive participants too.

Recruiting through Prolific or SONA

For Prolific, read the participant's ID from the study URL and give Potato a completion code and a screen-out code:

login:
  type: url_direct
  url_argument: PROLIFIC_PID
 
crowdsourcing:
  provider: prolific
  prolific:
    completion:
      code: "C1ABCDEF"          # finished
      failed_code: "C1SCREEN"   # screened out

For a SONA subject pool, crowdsourcing.provider: sona grants course credit from the server when a participant finishes, so the participant does not have to follow a link back to SONA. Running a Crowdsourced Study covers both platforms, CloudResearch Connect and the other providers.

A complete study

The config below runs an acceptability judgment study recruited on Prolific. Each participant is assigned to one of two stimulus lists, gives consent, answers a native-language screener that ends the session for non-native speakers, judges each sentence of their list under a four-second deadline, and finishes with a background questionnaire whose comment box comes last.

annotation_task_name: Acceptability judgments
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/list_a.jsonl
item_properties:
  id_key: id
  text_key: text
user_config:
  allow_all_users: true
  users: []
login:
  type: url_direct
  url_argument: PROLIFIC_PID
 
conditions:
  factors:
    list: [list_a, list_b]
  balance_on: completed
 
assignment_strategy: batch
batch_assignment:
  auto_assign_annotators: true
  group_by_condition: list
  groups:
    - {name: list_a, data_file: data/list_a.jsonl}
    - {name: list_b, data_file: data/list_b.jsonl}
 
phases:
  order: [consent, screener, annotation, background]
  consent:
    type: consent
    file: surveys/consent.json
  screener:
    type: prestudy
    file: surveys/screener.json
    exit_when:
      show_when:
        - {schema: native_english, operator: equals, value: "No"}
      message: This study is for native speakers of English. Thank you for your time.
  background:
    type: poststudy
    file: surveys/background.json
    randomize_questions:
      pin_last: [comments]
 
trial_timing:
  enabled: true
  fixation_ms: 500
  trial_duration_ms: 4000
  iti_ms: 500
  response:
    schema: judgment
    keys: {f: acceptable, j: unacceptable}
 
annotation_schemes:
  - annotation_type: radio
    name: judgment
    description: Is this sentence acceptable in English?
    labels: [acceptable, unacceptable]
    label_requirement:
      required: true
 
crowdsourcing:
  provider: prolific
  prolific:
    completion:
      code: "C1ABCDEF"
      failed_code: "C1SCREEN"

Each survey file is a JSON list of questions, each with annotation_type, name and description. The screener's question is named native_english, so exit_when can test it:

[{"annotation_type": "radio", "name": "native_english",
  "description": "Is English your native language?",
  "labels": ["Yes", "No"], "label_requirement": {"required": true}}]

The item files hold one sentence per line, with any design columns kept for the analysis:

{"id": "s1_a", "text": "Which book did you read before filing?", "condition": "gap"}

This config passes python -m potato.validate_cli config.yaml --strict on Potato 2.12.0. Start it with potato start config.yaml, and pilot it with debug mode or simulated participants before you post it.

Getting the data out

python -m potato.export --config config.yaml --format csv --output ./export/

The CSV export writes the judgments, plus three tables for the design:

• participants.csv, one row per participant, with their conditions, block and question orders, skipped pages and any screen-out;
• trials.csv, one row per timed trial, with the RT, the outcome (response or timeout), what the browser delivered against what was configured, and the server's checks;
• regions.csv, one row per region, for self-paced reading and maze.

--format runtime writes the runs and trials of jsPsych pages, headphone checks and external tasks. --format psych_ds writes a Psych-DS dataset, and --format data_package wraps the CSV export in a Frictionless Data Package with a Table Schema per file.

What Potato does not do

• Eye tracking and webcam recording. jsPsych's WebGazer extension is not shipped, because it needs 'unsafe-eval', which the jsPsych page's Content-Security-Policy does not allow.
• Timed trials on phones and tablets. They get Potato's touch interface, which does not run trial_timing. A jsPsych page runs on any device jsPsych supports.
• Fullscreen across items. Each item is a page load, and browsers leave fullscreen when the page changes. A jsPsych page holds fullscreen for its whole timeline.
• The MySQL storage backend for design data. MySQL stores neither the flow record nor trial records, so use the default storage for these studies.

Further Reading

• Experiment Flow - Conditions, skips, screen-outs and counterbalancing
• Timed Trials - Trial sequence, records and precision
• Runtime Phases - jsPsych, headphone checks and external tasks
• Coming from jsPsych - Concept map and a Stroop task inside a study
• Behavioral Experiment Tools Compared - jsPsych, PsychoPy, Gorilla, Qualtrics, lab.js and Potato
• Running a Crowdsourced Study - Prolific, SONA and CloudResearch Connect