# Behavioral Experiment Tools Compared: jsPsych, PsychoPy, Gorilla, Qualtrics, lab.js and Potato

Source: https://www.potatoannotator.com/docs/guides/behavioral-experiment-tools-compared

**jsPsych, PsychoPy, Gorilla and lab.js are built for trial-level behavioral tasks where stimulus timing and reaction times are the data, and Qualtrics is built for questionnaires, which Gorilla also hosts. Potato is built for studies in which participants judge, rate or label items, and its config covers condition assignment, screeners, counterbalanced pages and the hand-off to Prolific or SONA. Since Potato 2.12.0, it also runs a jsPsych timeline, a headphone check or a task on another server as one page of such a study, and runs self-paced reading and maze tasks on its own items.**

A behavioral experiment shows participants stimuli and records what they do, such as which key they press, how fast they press it and which option they choose. In a [between-subjects design](https://en.wikipedia.org/wiki/Between-group_design_experiment), each participant is assigned to one condition, and studies usually keep the assignment balanced as people arrive and drop out. [Counterbalancing](https://en.wikipedia.org/wiki/Counterbalancing) varies the order of blocks across participants so that order effects average out, often by the rows of a [Latin square](https://en.wikipedia.org/wiki/Latin_square). [Reaction time](https://en.wikipedia.org/wiki/Mental_chronometry) is the interval from stimulus onset to response, and in a browser its accuracy depends on the software, the browser and the participant's keyboard and display.

The tools below differ in who writes the trial logic, who hosts the study, how assignment works and what each claims about timing. [Annotation Tools Compared](/docs/guides/annotation-tools-compared) covers annotation tools for every data type.

## The six tools at a glance

The first table shows how each tool runs, who hosts the study, its license and how it is paid for. Costs are described by model only, because several of the tools quote prices on request or by package.

| Tool | Runs as | Hosting | License | Cost model |
|---|---|---|---|---|
| **[jsPsych](https://www.jspsych.org/)** | JavaScript library; you write the timeline in code | You choose a host: JATOS, Pavlovia, Cognition.run or your own server | MIT | Free; the host may charge |
| **[PsychoPy](https://www.psychopy.org/)** | Desktop Builder and Coder; online studies compile to PsychoJS | The lab computer, or [Pavlovia](https://pavlovia.org/) online | GPL-3.0-or-later (PsychoPy), MIT (PsychoJS) | Free in the lab; Pavlovia needs per-participant credits or an institutional license |
| **[Gorilla](https://gorilla.sc/)** | Hosted web platform with graphical builders | Gorilla | Proprietary | One token per participant, or a subscription or institutional license |
| **[Qualtrics](https://www.qualtrics.com/)** | Hosted survey platform | Qualtrics | Proprietary | Quoted on request; free accounts cannot redirect to a URL, and custom JavaScript is not in every license |
| **[lab.js](https://lab.js.org/)** | Graphical builder in the browser, plus a JavaScript library (latest npm release 20.2.4; latest commit October 2025, on a 23.0 alpha) | You choose a host: JATOS, Pavlovia, your own web space, or embedded in Qualtrics | Apache-2.0 (library), AGPL-3.0 (builder) | Free |
| **[Potato](/)** | Python web server configured in YAML | Your own server, or a [Hugging Face Space](/docs/deployment/deploy-huggingface) | GPL-3.0-or-later | Free |

Each tool except Qualtrics has a paper describing it: [jsPsych](https://doi.org/10.3758/s13428-014-0458-y), [PsychoPy](https://doi.org/10.3758/s13428-018-01193-y), [Gorilla](https://doi.org/10.3758/s13428-019-01237-x), [lab.js](https://doi.org/10.3758/s13428-019-01283-5) and [Potato 2.0](https://aclanthology.org/2026.acl-demo.37/).

## Which tool for which study

The table matches the part of a study that sets its requirements to the tools built for it.

| Study | Good fits |
|---|---|
| Reaction-time paradigms ([Stroop](https://en.wikipedia.org/wiki/Stroop_effect), flanker, [IAT](https://en.wikipedia.org/wiki/Implicit-association_test), visual search) | jsPsych, PsychoPy, Gorilla, lab.js |
| Lab studies with hardware eye trackers or other devices | PsychoPy |
| Webcam eye tracking in the browser | jsPsych, Gorilla, PsychoPy online (with custom code) |
| Questionnaires and survey experiments | Qualtrics, Gorilla, Potato |
| Ratings, labels or rankings of many items, split across participants | Potato |
| Speeded judgments on a list of items ([lexical decision](https://en.wikipedia.org/wiki/Lexical_decision_task), grammaticality) | jsPsych, PsychoPy, Gorilla, lab.js, Potato |
| Self-paced reading and maze tasks | jsPsych (with `jspsych-contrib` plugins), Potato |
| A timed task followed by ratings or surveys, with one completion code | Potato with a jsPsych page, Gorilla, lab.js embedded in Qualtrics |
| Building a study without programming (Potato through a YAML config) | Gorilla, Qualtrics, lab.js, PsychoPy Builder, Potato |

Gorilla appears in many rows because it hosts tasks, questionnaires and recruitment in one product. Potato appears where participants judge items or answer questionnaires, and its timed tasks run either as `trial_timing` on those items, including [self-paced reading and maze](/docs/experiments/self-paced-reading), or as a jsPsych page.

## Assignment, screening, recruitment and questionnaires by tool

The second table compares the work around the trials. Where a tool leaves a step to its host or to custom code, the cell says so.

| | jsPsych | PsychoPy and Pavlovia | Gorilla | Qualtrics | lab.js | Potato |
|---|---|---|---|---|---|---|
| Balanced assignment to conditions | From the host, e.g. a JATOS batch-session template | Counterbalance Routine (2024.1 and later), with groups set on the Pavlovia Shelf online | Randomiser node with a balanced mode, plus Quota and Allocator nodes | Randomizer with Evenly Present Elements | From the host | `conditions`, counted on assigned or on completed participants |
| Counterbalanced block order | Trial and block randomization in code | Conditions files and nested loops | Counterbalance and Order nodes | Randomizer over blocks | Shuffled loops | `phases.counterbalance`, random or Williams Latin square |
| Screen a participant out | `jsPsych.abortExperiment()` in code | Custom code | Branch node to a Reject node | Branch to an End of Survey element | Custom code | `exit_when`, which shows a message or returns the platform's screen-out outcome |
| Prolific | Documented URL-parameter and redirect recipe | Documented URL parameters and completion URL | Built-in recruitment policy | Marketplace integration published by Prolific | — | `login.type: url_direct`, or `crowdsourcing.provider: prolific` |
| SONA | Documented by Sona | Documented by Sona, linked from PsychoPy | Built-in recruitment policy, with automatic credit | Documented by Sona | — | `crowdsourcing.provider: sona`, with credit granted from the server |
| Webcam eye tracking | WebGazer extension, with calibration and validation plugins | WebGazer through custom code online; ioHub for hardware trackers in the lab | WebGazer component | — | — | — |
| Questionnaires | `survey` plugin (SurveyJS) and `survey-*` plugins | Pavlovia Surveys, online only | Questionnaire Builder | The core product | HTML forms | Survey pages, plus 55 [published instruments](/docs/features/survey-instruments) by name |

A dash means we found nothing for it in the tool's documentation, checked on 2026-10-22. If a cell is wrong, [tell us](https://github.com/davidjurgens/potato/issues/new).

## Timing claims in each tool's documentation

Reaction-time accuracy in a browser has been measured across tools, for example in the [timing mega-study](https://doi.org/10.7717/peerj.9414), which compared lab-based and online experiment generators. Each tool's documentation makes its own timing claims, summarized below.

- **jsPsych.** The [timing accuracy page](https://www.jspsych.org/latest/overview/timing-accuracy/) says reaction times measured in JavaScript run 10-40 ms longer than in standard lab software, with similar variance, and cites the [comparison with Psychophysics Toolbox](https://doi.org/10.3758/s13428-015-0567-2) among other studies.
- **PsychoPy.** The [timing page](https://www.psychopy.org/general/timing/index.html) says its timing is as good as any package the team has tested, and hosts the data tables from the mega-study.
- **Gorilla.** The [timing page](https://support.gorilla.sc/support/troubleshooting-and-technical/timing) says display durations are multiples of the screen's refresh interval, USB keyboards and mice add up to 8.3 ms of error, and touchscreens are polled about every 20 ms. It cites a [study of online platforms, browsers and devices](https://doi.org/10.3758/s13428-020-01501-5).
- **lab.js.** The [timing page](https://lab.js.org/docs/en/latest/recipes/timing-performance/index.html) says the library reaches high presentation and response accuracy with some care.
- **Qualtrics.** The [Timing question](https://www.qualtrics.com/support/survey-platform/survey-module/editing-questions/question-types-guide/advanced/timing/) records the first click, last click, page submit and click count on a page, as a record of page timing rather than reaction time.
- **Potato.** A `type: jspsych` page runs jsPsych alone, so jsPsych's figures apply there. `trial_timing` on Potato's own items reveals the stimulus inside an animation-frame callback and measures reaction time from the frame it was drawn on, so onset is known to the frame (about 17 ms at 60 Hz). Each `trial_timing` record stores the browser's measured frame interval and timer resolution, and the server flags records that disagree with its own clock.

Keyboards and displays add latency the browser cannot measure, and it differs between participants' machines. For every tool here, compare reaction times within participants or model them with a participant term.

## Item studies and jsPsych pages in Potato

Potato started as an annotation tool, so its unit of work is an item from a data file that a participant judges. A study that collects ratings of 400 headlines from 60 participants, with each participant seeing a different 40, fits that model. Potato assigns the items, records every judgment against the item and the participant, and reports [inter-annotator agreement](/docs/guides/inter-annotator-agreement) (inter-rater reliability) on the items more than one participant rated.

The [experiment flow](/docs/experiments/experiment-flow) settings apply to item studies and to studies with a jsPsych task. `conditions` assigns each participant to a condition when they arrive, and every rule in the config can test `condition.<factor>`. `show_when` skips a page, `exit_when` screens a participant out, and `phases.counterbalance` orders pages by participant. The [survey phases](/docs/features/surveyflow) around the task, the [training phase](/docs/features/training-phase) with feedback, and the hand-off to [Prolific](/docs/guides/crowdsourcing-prolific-mturk) or SONA are config entries.

When a study needs trial-level control, the trials stay in jsPsych. A `type: jspsych` page runs a jsPsych 8.3.0 timeline between Potato's other pages, passes it the participant's ID and condition, and stores the trial data in the task directory's `project.sqlite`. Two other [runtime pages](/docs/experiments/runtime-phases) sit in the same place: a headphone check that the server scores, and an external task on oTree, Empirica or a hosted survey that returns the participant with signed results. [Coming from jsPsych](/docs/guides/coming-from-jspsych) maps jsPsych concepts to Potato keys and walks through a Stroop task inside a Potato study.

## A between-subjects rating study in Potato

The config below runs a headline credibility study for a SONA subject pool. Participants are assigned to see headlines with or without their source, and `balance_on: completed` counts only participants who finish, so the two conditions stay balanced when people drop out or are screened out. A screener ends the study for anyone who does not read English-language news, and the two post-study questionnaires appear in Latin-square order.

```yaml
annotation_task_name: Headline credibility study
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/with_source.jsonl
item_properties:
  id_key: id
  text_key: text
user_config:
  allow_all_users: true
  users: []

conditions:
  factors:
    list: [with_source, without_source]
  assignment: balanced
  balance_on: completed

# Each condition gets its own item file, named after the factor's levels
assignment_strategy: batch
batch_assignment:
  auto_assign_annotators: true
  group_by_condition: list
  groups:
    - {name: with_source, data_file: data/with_source.jsonl}
    - {name: without_source, data_file: data/without_source.jsonl}

phases:
  order: [consent, screener, annotation, attitudes, media_use]
  consent:
    type: consent
    file: surveys/consent.json
  screener:
    type: prestudy
    file: surveys/screener.json
    exit_when:
      show_when:
        - {schema: reads_news, operator: equals, value: "No"}
      message: This study is for people who read English-language news. Thank you for your time.
  attitudes:
    type: poststudy
    file: surveys/attitudes.json
  media_use:
    type: poststudy
    file: surveys/media_use.json
    randomize_questions:
      pin_last: [comments]
  counterbalance:
    - pages: [attitudes, media_use]
      method: latin_square

crowdsourcing:
  provider: sona
  sona:
    hostname: yourdept.sona-systems.com
    experiment_id: 123
    credit_token: your-credit-token

annotation_schemes:
  - annotation_type: likert
    name: credibility
    description: How credible is this headline?
    size: 7                   # an odd number of points gives a neutral midpoint
    min_label: Not at all credible
    max_label: Very credible
    label_requirement:
      required: true
```

The two item files hold the same headlines, one with the source line and one without, and the item IDs must differ between them because Potato loads both files into one project. The CSV export writes `participants.csv`, with each participant's condition, questionnaire order and any screen-out, beside the ratings.

In SONA, set the study URL to `https://your-server.com/?sona_code=%SURVEY_CODE%`. When a participant finishes, Potato grants the credit itself through SONA's server-side credit call, so the participant does not have to follow a link back to SONA.

## Limits of Potato for behavioral experiments

The limits below apply to Potato's own pages. Most of them have a way around through a jsPsych page, and the bullets say which.

- **No trial code on Potato's own pages.** Canvas drawing, frame-locked masks and audio scheduled with Web Audio need a jsPsych page.
- **No eye tracking or webcam recording.** jsPsych's WebGazer extension is not shipped, because it needs `'unsafe-eval'`, which the jsPsych page's Content-Security-Policy does not allow.
- **No mouse trajectories on Potato's own pages.** The `authenticity` block counts pointer moves and distance per item and keeps no positions. A jsPsych page with `extensions: [mouse-tracking]` records x/y samples per trial.
- **No fullscreen across items.** Each Potato item is a page load, and browsers leave fullscreen when the page changes. A jsPsych page holds fullscreen for its whole timeline.
- **No timed trials on phones.** Phones and tablets get Potato's touch interface, which does not run `trial_timing`.
- **No `trial_timing` records or `participants.csv` data on the MySQL backend.** Both need the default storage. jsPsych runs are stored in `project.sqlite` on either backend.

## Further reading

- [Coming from jsPsych](/docs/guides/coming-from-jspsych), for the concept map and a jsPsych task inside a Potato study
- [Multi-Phase Workflows](/docs/features/surveyflow)
- [Survey Instruments](/docs/features/survey-instruments)
- [Running a Crowdsourced Study](/docs/guides/crowdsourcing-prolific-mturk)
- [Rating Scales](/docs/guides/rating-scales)
- [Annotation Tools Compared](/docs/guides/annotation-tools-compared)
- [Experiment Flow](/docs/experiments/experiment-flow), [Runtime Phases](/docs/experiments/runtime-phases) and [Timed Trials](/docs/experiments/timed-trials)
- [Online Behavioral Experiments](/docs/guides/online-behavioral-experiments), for a psychology or linguistics study in Potato from start to finish

## References

1. de Leeuw, J. R. (2015). jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. *Behavior Research Methods*, 47(1), 1-12. https://doi.org/10.3758/s13428-014-0458-y
2. Peirce, J., Gray, J. R., Simpson, S., et al. (2019). PsychoPy2: Experiments in behavior made easy. *Behavior Research Methods*, 51(1), 195-203. https://doi.org/10.3758/s13428-018-01193-y
3. Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., et al. (2020). Gorilla in our midst: An online behavioral experiment builder. *Behavior Research Methods*, 52(1), 388-407. https://doi.org/10.3758/s13428-019-01237-x
4. Henninger, F., Shevchenko, Y., Mertens, U. K., et al. (2022). lab.js: A free, open, online study builder. *Behavior Research Methods*, 54(2), 556-573. https://doi.org/10.3758/s13428-019-01283-5
5. Jurgens, D., Chen, M., & Iyer, L. (2026). Potato 2.0: A Comprehensive Annotation Platform with AI-in-the-Loop Support. *Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)*, 374-386. https://aclanthology.org/2026.acl-demo.37/
6. Bridges, D., Pitiot, A., MacAskill, M. R., et al. (2020). The timing mega-study: comparing a range of experiment generators, both lab-based and online. *PeerJ*, 8, e9414. https://doi.org/10.7717/peerj.9414
7. de Leeuw, J. R., & Motz, B. A. (2016). Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task. *Behavior Research Methods*, 48(1), 1-12. https://doi.org/10.3758/s13428-015-0567-2
8. Anwyl-Irvine, A., Dalmaijer, E. S., Hodges, N., et al. (2021). Realistic precision and accuracy of online experiment platforms, web browsers, and devices. *Behavior Research Methods*, 53(4), 1407-1425. https://doi.org/10.3758/s13428-020-01501-5