Behavioral Experiment Tools Compared: jsPsych, PsychoPy, Gorilla, Qualtrics, lab.js and Potato

jsPsych, PsychoPy, Gorilla and lab.js are built for trial-level behavioral tasks where stimulus timing and reaction times are the data, and Qualtrics is built for questionnaires, which Gorilla also hosts. Potato is built for studies in which participants judge, rate or label items, and its config covers condition assignment, screeners, counterbalanced pages and the hand-off to Prolific or SONA. Since Potato 2.12.0, it also runs a jsPsych timeline, a headphone check or a task on another server as one page of such a study, and runs self-paced reading and maze tasks on its own items.

A behavioral experiment shows participants stimuli and records what they do, such as which key they press, how fast they press it and which option they choose. In a between-subjects design, each participant is assigned to one condition, and studies usually keep the assignment balanced as people arrive and drop out. Counterbalancing varies the order of blocks across participants so that order effects average out, often by the rows of a Latin square. Reaction time is the interval from stimulus onset to response, and in a browser its accuracy depends on the software, the browser and the participant's keyboard and display.

The tools below differ in who writes the trial logic, who hosts the study, how assignment works and what each claims about timing. Annotation Tools Compared covers annotation tools for every data type.

The six tools at a glance

The first table shows how each tool runs, who hosts the study, its license and how it is paid for. Costs are described by model only, because several of the tools quote prices on request or by package.

Each tool except Qualtrics has a paper describing it: jsPsych, PsychoPy, Gorilla, lab.js and Potato 2.0.

Which tool for which study

The table matches the part of a study that sets its requirements to the tools built for it.

Gorilla appears in many rows because it hosts tasks, questionnaires and recruitment in one product. Potato appears where participants judge items or answer questionnaires, and its timed tasks run either as trial_timing on those items, including self-paced reading and maze, or as a jsPsych page.

Assignment, screening, recruitment and questionnaires by tool

The second table compares the work around the trials. Where a tool leaves a step to its host or to custom code, the cell says so.

A dash means we found nothing for it in the tool's documentation, checked on 2026-10-22. If a cell is wrong, tell us.

Timing claims in each tool's documentation

Reaction-time accuracy in a browser has been measured across tools, for example in the timing mega-study, which compared lab-based and online experiment generators. Each tool's documentation makes its own timing claims, summarized below.

• jsPsych. The timing accuracy page says reaction times measured in JavaScript run 10-40 ms longer than in standard lab software, with similar variance, and cites the comparison with Psychophysics Toolbox among other studies.
• PsychoPy. The timing page says its timing is as good as any package the team has tested, and hosts the data tables from the mega-study.
• Gorilla. The timing page says display durations are multiples of the screen's refresh interval, USB keyboards and mice add up to 8.3 ms of error, and touchscreens are polled about every 20 ms. It cites a study of online platforms, browsers and devices.
• lab.js. The timing page says the library reaches high presentation and response accuracy with some care.
• Qualtrics. The Timing question records the first click, last click, page submit and click count on a page, as a record of page timing rather than reaction time.
• Potato. A type: jspsych page runs jsPsych alone, so jsPsych's figures apply there. trial_timing on Potato's own items reveals the stimulus inside an animation-frame callback and measures reaction time from the frame it was drawn on, so onset is known to the frame (about 17 ms at 60 Hz). Each trial_timing record stores the browser's measured frame interval and timer resolution, and the server flags records that disagree with its own clock.

Keyboards and displays add latency the browser cannot measure, and it differs between participants' machines. For every tool here, compare reaction times within participants or model them with a participant term.

Item studies and jsPsych pages in Potato

Potato started as an annotation tool, so its unit of work is an item from a data file that a participant judges. A study that collects ratings of 400 headlines from 60 participants, with each participant seeing a different 40, fits that model. Potato assigns the items, records every judgment against the item and the participant, and reports inter-annotator agreement (inter-rater reliability) on the items more than one participant rated.

The experiment flow settings apply to item studies and to studies with a jsPsych task. conditions assigns each participant to a condition when they arrive, and every rule in the config can test condition.<factor>. show_when skips a page, exit_when screens a participant out, and phases.counterbalance orders pages by participant. The survey phases around the task, the training phase with feedback, and the hand-off to Prolific or SONA are config entries.

When a study needs trial-level control, the trials stay in jsPsych. A type: jspsych page runs a jsPsych 8.3.0 timeline between Potato's other pages, passes it the participant's ID and condition, and stores the trial data in the task directory's project.sqlite. Two other runtime pages sit in the same place: a headphone check that the server scores, and an external task on oTree, Empirica or a hosted survey that returns the participant with signed results. Coming from jsPsych maps jsPsych concepts to Potato keys and walks through a Stroop task inside a Potato study.

A between-subjects rating study in Potato

The config below runs a headline credibility study for a SONA subject pool. Participants are assigned to see headlines with or without their source, and balance_on: completed counts only participants who finish, so the two conditions stay balanced when people drop out or are screened out. A screener ends the study for anyone who does not read English-language news, and the two post-study questionnaires appear in Latin-square order.

annotation_task_name: Headline credibility study
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/with_source.jsonl
item_properties:
  id_key: id
  text_key: text
user_config:
  allow_all_users: true
  users: []
 
conditions:
  factors:
    list: [with_source, without_source]
  assignment: balanced
  balance_on: completed
 
# Each condition gets its own item file, named after the factor's levels
assignment_strategy: batch
batch_assignment:
  auto_assign_annotators: true
  group_by_condition: list
  groups:
    - {name: with_source, data_file: data/with_source.jsonl}
    - {name: without_source, data_file: data/without_source.jsonl}
 
phases:
  order: [consent, screener, annotation, attitudes, media_use]
  consent:
    type: consent
    file: surveys/consent.json
  screener:
    type: prestudy
    file: surveys/screener.json
    exit_when:
      show_when:
        - {schema: reads_news, operator: equals, value: "No"}
      message: This study is for people who read English-language news. Thank you for your time.
  attitudes:
    type: poststudy
    file: surveys/attitudes.json
  media_use:
    type: poststudy
    file: surveys/media_use.json
    randomize_questions:
      pin_last: [comments]
  counterbalance:
    - pages: [attitudes, media_use]
      method: latin_square
 
crowdsourcing:
  provider: sona
  sona:
    hostname: yourdept.sona-systems.com
    experiment_id: 123
    credit_token: your-credit-token
 
annotation_schemes:
  - annotation_type: likert
    name: credibility
    description: How credible is this headline?
    size: 7                   # an odd number of points gives a neutral midpoint
    min_label: Not at all credible
    max_label: Very credible
    label_requirement:
      required: true

The two item files hold the same headlines, one with the source line and one without, and the item IDs must differ between them because Potato loads both files into one project. The CSV export writes participants.csv, with each participant's condition, questionnaire order and any screen-out, beside the ratings.

In SONA, set the study URL to https://your-server.com/?sona_code=%SURVEY_CODE%. When a participant finishes, Potato grants the credit itself through SONA's server-side credit call, so the participant does not have to follow a link back to SONA.

Limits of Potato for behavioral experiments

The limits below apply to Potato's own pages. Most of them have a way around through a jsPsych page, and the bullets say which.

• No trial code on Potato's own pages. Canvas drawing, frame-locked masks and audio scheduled with Web Audio need a jsPsych page.
• No eye tracking or webcam recording. jsPsych's WebGazer extension is not shipped, because it needs 'unsafe-eval', which the jsPsych page's Content-Security-Policy does not allow.
• No mouse trajectories on Potato's own pages. The authenticity block counts pointer moves and distance per item and keeps no positions. A jsPsych page with extensions: [mouse-tracking] records x/y samples per trial.
• No fullscreen across items. Each Potato item is a page load, and browsers leave fullscreen when the page changes. A jsPsych page holds fullscreen for its whole timeline.
• No timed trials on phones. Phones and tablets get Potato's touch interface, which does not run trial_timing.
• No trial_timing records or participants.csv data on the MySQL backend. Both need the default storage. jsPsych runs are stored in project.sqlite on either backend.

Further reading

• Coming from jsPsych, for the concept map and a jsPsych task inside a Potato study
• Multi-Phase Workflows
• Survey Instruments
• Running a Crowdsourced Study
• Rating Scales
• Annotation Tools Compared
• Experiment Flow, Runtime Phases and Timed Trials
• Online Behavioral Experiments, for a psychology or linguistics study in Potato from start to finish

References

1. de Leeuw, J. R. (2015). jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behavior Research Methods, 47(1), 1-12. https://doi.org/10.3758/s13428-014-0458-y
2. Peirce, J., Gray, J. R., Simpson, S., et al. (2019). PsychoPy2: Experiments in behavior made easy. Behavior Research Methods, 51(1), 195-203. https://doi.org/10.3758/s13428-018-01193-y
3. Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., et al. (2020). Gorilla in our midst: An online behavioral experiment builder. Behavior Research Methods, 52(1), 388-407. https://doi.org/10.3758/s13428-019-01237-x
4. Henninger, F., Shevchenko, Y., Mertens, U. K., et al. (2022). lab.js: A free, open, online study builder. Behavior Research Methods, 54(2), 556-573. https://doi.org/10.3758/s13428-019-01283-5
5. Jurgens, D., Chen, M., & Iyer, L. (2026). Potato 2.0: A Comprehensive Annotation Platform with AI-in-the-Loop Support. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), 374-386. https://aclanthology.org/2026.acl-demo.37/
6. Bridges, D., Pitiot, A., MacAskill, M. R., et al. (2020). The timing mega-study: comparing a range of experiment generators, both lab-based and online. PeerJ, 8, e9414. https://doi.org/10.7717/peerj.9414
7. de Leeuw, J. R., & Motz, B. A. (2016). Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task. Behavior Research Methods, 48(1), 1-12. https://doi.org/10.3758/s13428-015-0567-2
8. Anwyl-Irvine, A., Dalmaijer, E. S., Hodges, N., et al. (2021). Realistic precision and accuracy of online experiment platforms, web browsers, and devices. Behavior Research Methods, 53(4), 1407-1425. https://doi.org/10.3758/s13428-020-01501-5