Tutorials15 min read

Run a Stroop Task on Your Own Server With jsPsych

Host a jsPsych Stroop task on a university server with Potato, add a color-vision screen-out and Prolific completion codes, and analyze reaction times in R.

Potato TeamNovember 1, 2026

Cognitive psychologists who write tasks in jsPsych can follow this tutorial to run a Stroop task on a server under the lab's control, with Potato 2.13.1 serving the pages and storing the data. The study has a consent page, a color-vision screener that ends the session for anyone who has trouble telling red, green, and blue apart, the jsPsych timeline with trial-by-trial feedback for half of the participants, two ratings of the task, and a debrief. Participants arrive from Prolific and leave with a completion code, and the trial data goes into R. The configs, commands, and export outputs on this page come from runs of Potato 2.12.1 with simulated participants in a headless browser.

Hosting options for a jsPsych study

jsPsych runs in the participant's browser and hosts nothing itself. A jsPsych study needs a server to deliver the page and receive the data. PsychoPy's pricing page, updated October 21, 2026, lists Pavlovia credits at £0.26 per participant and a university license at £2,000 a year. Gorilla hosts the whole study, and its pricing FAQ lists academic pay-as-you-go tokens at £1.09 per participant, with a minimum purchase of 25 tokens. Both services keep the data on their own servers. In the PsychoPy forum thread "Running Experiments on Own Server", PsychoPy staff wrote in 2021 that a local install of Pavlovia was available to institutions with site licenses.

JATOS is free and open source under the Apache 2.0 license, installs on your own server, and runs jsPsych, lab.js, and OpenSesame studies. JATOS is the established choice when the study is the jsPsych timeline alone. Potato fits when the timeline is one part of a session that also needs consent and screening pages, balanced assignment to conditions, ratings or questionnaires, and the return to Prolific. Potato sets up each of these parts in a YAML config rather than in JavaScript. Behavioral Experiment Tools Compared sets the tools side by side.

The study

The session runs in this order:

Consent. A page describes the study and asks for agreement.
Screener. A participant who reports difficulty telling red, green, or blue apart is screened out and sent back to Prolific with a screen-out code.
Stroop task. A jsPsych timeline shows each color word in each ink color twice, 18 trials in all. Participants in the immediate condition see "Correct", "Incorrect", or "Too slow" after each response, and participants in the none condition see no feedback.
Task ratings. Two five-point ratings ask how difficult the task was and how well the participant did.
Debrief. A page explains the Stroop effect and the feedback manipulation and offers a comment box.

Feedback is a between-subjects factor, and each participant keeps one condition for the whole session. Potato assigns the condition on arrival and passes the condition to the timeline.

Installing Potato and laying out the project

Install the release this tutorial was checked against:

bash

pip install potato-annotation==2.13.1

The project folder contains the config, three survey pages, the timeline, and one data file:

text

stroop-study/
├── config.yaml
├── data/task.jsonl
├── surveys/consent.json
├── surveys/screener.json
├── surveys/debrief.json
└── tasks/stroop.js

The config

The config below is complete and passes python -m potato.validate_cli config.yaml --strict on 2.12.1 and on 2.13.1. The validator also checks that tasks/stroop.js exists.

yaml

annotation_task_name: Stroop feedback study
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/task.jsonl
item_properties:
  id_key: id
  text_key: text
 
login:
  type: url_direct
  url_argument: PROLIFIC_PID
user_config:
  allow_all_users: true
  users: []
 
crowdsourcing:
  provider: prolific
  prolific:
    completion:
      code: C1STROOP
      screened_out_code: C1SCREEN
 
conditions:
  factors:
    feedback: [none, immediate]
  assignment: balanced
  balance_on: completed
 
phases:
  order: [consent, screener, stroop, annotation, debrief]
  consent:
    type: consent
    file: surveys/consent.json
  screener:
    type: prestudy
    file: surveys/screener.json
    exit_when:
      show_when:
        - {schema: color_vision, operator: equals, value: "Yes"}
  stroop:
    type: jspsych
    timeline: tasks/stroop.js
    plugins: [instructions, html-keyboard-response]
    save: incremental
  annotation:
    type: annotation
  debrief:
    type: poststudy
    file: surveys/debrief.json
 
annotation_schemes:
  - annotation_type: likert
    name: difficulty
    description: The color-naming task was difficult.
    size: 5
    min_label: Strongly disagree
    max_label: Strongly agree
    label_requirement:
      required: true
  - annotation_type: likert
    name: performance
    description: I did well on the color-naming task.
    size: 5
    min_label: Strongly disagree
    max_label: Strongly agree
    label_requirement:
      required: true
 
export_include_phase_data: true

login.type: url_direct logs each participant in with the PROLIFIC_PID from the study link, without a login page. balance_on: completed balances the two feedback conditions on participants who finished without being screened out. Participants who quit or fail the screener do not count toward either condition. From Potato 2.13.1, participants still working count toward their condition for in_progress_minutes after their last activity, 60 by default, so people who start at the same time are spread across both conditions. save: incremental stores each trial as the participant finishes it. With the default save: on_finish, closing the tab halfway through leaves no trials behind. export_include_phase_data: true puts the consent, screener, and debrief answers in the CSV export.

Every Potato study has an annotation phase with at least one item, and from 2.13.1 potato validate reports an error for a config without one. The Stroop study uses the phase for the two task ratings on a single item. data/task.jsonl has one line:

json

{"id": "stroop_task", "text": "The color-naming task you just finished"}

Prolific asks researchers to send participants who do not consent down a return path, and asks that such participants never be rejected (Prolific's guide to screening within a study). In Potato 2.13.1, every exit_when rule gives the same screen-out code. The consent page therefore has no exit_when. Participants who do not agree are asked to return the study on Prolific, and the screen-out code goes only to participants the screener excludes. surveys/consent.json reads:

json

[
  {
    "name": "study_information",
    "annotation_type": "pure_display",
    "description": "About this study",
    "labels": [
      "You will press keys for the ink color of color words, then answer a few questions. The study takes about 10 minutes.",
      "Your responses are stored on a university server under your Prolific ID.",
      "If you do not want to take part, close this page and return the study on Prolific."
    ]
  },
  {
    "name": "consent",
    "annotation_type": "radio",
    "description": "Do you agree to take part in this study?",
    "labels": ["I agree to take part"],
    "label_requirement": {"required_label": ["I agree to take part"]}
  }
]

On a survey page, the server checks required_label, and from 2.13.1 the server checks required: true as well. In 2.12.1, required: true was checked only in the browser. A test request on 2.12.1 that skipped the consent question and posted the page directly got a 400 response and stayed on consent. The screener in surveys/screener.json lists both options under required_label. The server then requires an answer without requiring a particular one:

json

[
  {
    "name": "color_vision",
    "annotation_type": "radio",
    "description": "Do you have difficulty telling red, green, or blue apart?",
    "labels": ["Yes", "No"],
    "label_requirement": {"required_label": ["Yes", "No"]}
  }
]

A participant who answers "Yes" goes straight to the end of the session with C1SCREEN. The screener has its own page before the task. Potato checks an exit_when on a jsPsych page only after the timeline ends, when the participant has already done the task. Prolific also recommends filtering with its own prescreeners where possible, before participants reach the study.

The Stroop timeline

The timeline file assigns a function to window.potatoTimeline. Potato creates the jsPsych instance, calls the function with the instance and a potato object, and saves the data when the timeline ends. The timeline below is adapted from the examples/advanced/jspsych-stroop example that ships with Potato:

javascript

window.potatoTimeline = function (jsPsych, potato) {
    var inks = [
        {name: 'RED', color: '#c62828', key: 'r'},
        {name: 'GREEN', color: '#2e7d32', key: 'g'},
        {name: 'BLUE', color: '#1565c0', key: 'b'}
    ];
 
    // Every word in every ink: 3 congruent and 6 incongruent pairs.
    var stimuli = [];
    inks.forEach(function (word) {
        inks.forEach(function (ink) {
            stimuli.push({
                word: word.name,
                ink: ink.name,
                color: ink.color,
                correct_response: ink.key,
                congruent: word.name === ink.name
            });
        });
    });
 
    var instructions = {
        type: jsPsychInstructions,
        pages: [
            '<p>A color word will appear in colored ink.</p>' +
            '<p>Press the key for the <strong>ink color</strong>, not the word:</p>' +
            '<p><strong>R</strong> = red, <strong>G</strong> = green, <strong>B</strong> = blue</p>',
            '<p>Answer as quickly and accurately as you can.</p>'
        ],
        show_clickable_nav: true
    };
 
    var fixation = {
        type: jsPsychHtmlKeyboardResponse,
        stimulus: '<div style="font-size:48px;">+</div>',
        choices: 'NO_KEYS',
        trial_duration: function () {
            return jsPsych.randomization.sampleWithoutReplacement([400, 500, 600], 1)[0];
        },
        data: {task: 'fixation'}
    };
 
    var trial = {
        type: jsPsychHtmlKeyboardResponse,
        stimulus: function () {
            return '<div style="font-size:64px;font-weight:700;color:' +
                jsPsych.evaluateTimelineVariable('color') + ';">' +
                jsPsych.evaluateTimelineVariable('word') + '</div>';
        },
        choices: ['r', 'g', 'b'],
        trial_duration: 2500,
        data: {
            task: 'stroop',
            word: jsPsych.timelineVariable('word'),
            ink: jsPsych.timelineVariable('ink'),
            congruent: jsPsych.timelineVariable('congruent'),
            correct_response: jsPsych.timelineVariable('correct_response')
        },
        on_finish: function (data) {
            data.correct = jsPsych.pluginAPI.compareKeys(data.response, data.correct_response);
        }
    };
 
    // Runs only for participants in the feedback: immediate condition.
    var feedback = {
        timeline: [{
            type: jsPsychHtmlKeyboardResponse,
            stimulus: function () {
                var last = jsPsych.data.get().last(1).values()[0];
                if (last.response === null) { return '<p>Too slow</p>'; }
                return last.correct ? '<p>Correct</p>' : '<p>Incorrect</p>';
            },
            choices: 'NO_KEYS',
            trial_duration: 600,
            data: {task: 'feedback'}
        }],
        conditional_function: function () {
            return potato.conditions.feedback === 'immediate';
        }
    };
 
    var block = {
        timeline: [fixation, trial, feedback],
        timeline_variables: stimuli,
        repetitions: 2,
        randomize_order: true
    };
 
    return [instructions, block];
};

potato.conditions.feedback is the level Potato assigned to the participant, and the conditional_function shows the feedback screen in the immediate condition only. The file does not call initJsPsych and sets no experiment-level on_finish, because Potato does both. Trial-level on_finish works as in any jsPsych study.

Potato 2.13.1 ships jsPsych 8.3.0 and 15 of its plugins inside the package, and the page loads both from the Potato server rather than from a CDN. The plugins list in the config names the two plugins the timeline uses. A plugin Potato does not ship, such as one from jspsych-contrib, goes in extra_scripts as a browser build. The 18 trials give each participant 6 congruent trials, and a real study raises repetitions.

The ratings and the debrief

The two ratings run in the annotation phase, after the task. Both ratings set label_requirement: required: true. The server checks the requirement on annotation items, and a participant cannot reach the debrief without answering both. The debrief in surveys/debrief.json explains the manipulation and leaves the comment optional:

json

[
  {
    "name": "debriefing",
    "annotation_type": "pure_display",
    "description": "What this study measured",
    "labels": [
      "Naming the ink color takes longer when the word names a different color. This slowdown is the Stroop effect. Half of the participants saw feedback after each response, and the study compares the Stroop effect between the two groups."
    ]
  },
  {
    "name": "comments",
    "annotation_type": "text",
    "description": "Anything else you want to tell us? (optional)"
  }
]

Testing the study on your machine

Start the server from the project folder:

bash

potato start config.yaml -p 8000

Open http://localhost:8000/?PROLIFIC_PID=TEST1&STUDY_ID=s1&SESSION_ID=x1 and use a new PROLIFIC_PID for each test participant. Run through the study once in full, once answering "Yes" on the screener, and once reloading the page in the middle of the Stroop block. A participant who finishes sees C1STROOP with a "Return to Prolific" button, and a screened-out participant sees C1SCREEN. In 2.12.1, the screened-out participant's page shows the same "You have completed the annotation task" text as a finished participant's page, and the code is the only difference.

The jsPsych page has no Next button. Until the timeline finishes and the server has stored the run, the server answers any request to move on with an error, and a direct request to skip the page returned a 400 in testing. A reload starts the timeline over, because jsPsych cannot resume partway through. With save: incremental, the trials from before the reload stay in the export as a run with status partial, and the new run gets attempt number 2.

Test participants count toward the conditions and appear in the export. Delete annotation_output/ and project.sqlite before launch.

Putting the study on a server

Prolific participants reach the study only through a public address. Production Setup covers running Potato behind nginx or Apache with TLS on a university server, and Deploying Potato on a VPS covers a rented server. The trial data stays in project.sqlite in the project folder, and the ratings and survey answers stay in annotation_output/, both on the machine that runs the server.

On Prolific, choose to record Prolific IDs with URL parameters and set the study URL to your server's address with the three parameters:

text

https://your-server.example/?PROLIFIC_PID={{%PROLIFIC_PID%}}&STUDY_ID={{%STUDY_ID%}}&SESSION_ID={{%SESSION_ID%}}

Add two completion paths on Prolific, C1STROOP for participants who finished and C1SCREEN with Prolific's "Screened out" label. Prolific pays screened-out participants the fixed screen-out reward you set, and their submissions do not count toward your places. Participants who do not consent return the study on Prolific and get no code. Crowdsourcing Integration has the Prolific API settings for tracking submissions.

Exporting the trial data

Potato stores each jsPsych run in project.sqlite (table runtime_runs), not in the participants' user_state.json files. The jspsych export format writes the runs and trials as CSV:

bash

python -m potato.export --config config.yaml --format jspsych --output export/

A test run on Potato 2.12.1 with seven simulated participants wrote these files:

File	One row per	Contents
`jspsych_runs.csv`	run of the timeline	Participant, page, attempt, status (`complete` or `partial`), start and finish times, trial count, and `condition_feedback`
`jspsych_trials.csv`	jsPsych trial	`run_status`, `attempt`, `condition_feedback`, and every column jsPsych recorded, including `rt`, `response`, `plugin_version`, and the timeline's own `task`, `word`, `ink`, `congruent`, and `correct`
`jspsych_raw/<participant>/stroop-<run>.json`	run	The run's data as jsPsych produced it

A fourth file, jspsych_interaction.csv, appears when jsPsych recorded blur, focus, or fullscreen events. The rows below are the first Stroop trials of one participant in the feedback condition, with a subset of the columns:

text

user_id,attempt,run_status,trial_index,rt,response,condition_feedback,plugin_version,task,word,ink,congruent,correct
5f1a02,1,complete,2,604,r,immediate,2.2.0,stroop,BLUE,RED,False,True
5f1a02,1,complete,5,538,b,immediate,2.2.0,stroop,RED,RED,True,False
5f1a02,1,complete,8,545,g,immediate,2.2.0,stroop,BLUE,GREEN,False,True

rt is in milliseconds. On a trial that timed out, rt and response are empty and correct is False. jsPsych records the plugin's version on every trial, here 2.2.0 for html-keyboard-response and 2.1.0 for instructions. The jsPsych core version is not in the data, and the methods section should name both versions, such as "jsPsych 8.3.0, served by Potato 2.13.1". The CSV file has True and False for true and false values, and the raw JSON keeps JSON booleans.

The CSV export writes the rest of the study:

bash

python -m potato.export --config config.yaml --format csv --output results/

results/participants.csv has one row per participant, with condition_feedback, screened_out, screened_out_page, and a runtime_runs column that gives the attempt count and status of the Stroop run. results/annotations.csv has the two ratings, with one column per scale point (difficulty.1 to difficulty.5), and results/phase_responses.csv has the consent, screener, and debrief answers. For a data repository, --format psych_ds writes all five tables under data/ with Psych-DS file names and a dataset_description.json that lists every column.

Analyzing reaction times in R

The R code below reads the trials table, keeps the Stroop trials from complete runs, and computes each participant's Stroop effect as the mean correct reaction time on incongruent trials minus the mean on congruent trials:

trials <- read.csv("export/jspsych_trials.csv")
stroop <- subset(trials, task == "stroop" & run_status == "complete")
stroop$congruent <- as.logical(stroop$congruent)
stroop$correct <- as.logical(stroop$correct)
 
accuracy <- aggregate(correct ~ user_id + condition_feedback, data = stroop, FUN = mean)
 
fast <- subset(stroop, correct & rt >= 200)
cells <- aggregate(rt ~ user_id + condition_feedback + congruent, data = fast, FUN = mean)
wide <- reshape(cells, idvar = c("user_id", "condition_feedback"),
                timevar = "congruent", direction = "wide")
wide$stroop_effect <- wide$rt.FALSE - wide$rt.TRUE
 
aggregate(stroop_effect ~ condition_feedback, data = wide, FUN = mean)
t.test(stroop_effect ~ condition_feedback, data = wide)

as.logical reads the True and False strings and leaves a column read.csv has already converted unchanged. The 200 ms floor drops anticipatory key presses, and a preregistered cutoff replaces the floor where one exists. The accuracy table counts a timeout as an error, because a timed-out trial has correct set to false.

A participant whose complete run has attempt 2 reloaded the page partway through and saw some trials twice. subset(stroop, attempt == 1) keeps first attempts only, and the choice belongs in the analysis plan. For a trial-level model, lme4 fits the congruency effect, the feedback condition, and their interaction, with a random congruency slope for each participant:

library(lme4)
fit <- lmer(rt ~ congruent * condition_feedback + (1 + congruent | user_id), data = fast)
summary(fit)

The congruentTRUE:condition_feedbacknone coefficient estimates how much the Stroop effect differs between the two feedback conditions. To count screen-outs for the participant flow, read results/participants.csv and tabulate screened_out_page.

Where to go next

Coming from jsPsych, for the concept map from jsPsych to Potato config keys
Runtime Phases, for every key a type: jspsych page takes, including assets_dir for image and audio stimuli
Experiment Flow, for conditions, exit_when, and counterbalanced page order
Online Behavioral Experiments, for timed trials on Potato's own items and SONA recruitment
Psychology experiments in Potato, for the behavioral features in one place
Behavioral Experiment Tools Compared, for jsPsych, PsychoPy, Gorilla, lab.js, and Qualtrics

Hosting options for a jsPsych study

The study

Installing Potato and laying out the project

The config

Consent and the color-vision screener

The Stroop timeline

The ratings and the debrief

Testing the study on your machine

Putting the study on a server

Exporting the trial data

Analyzing reaction times in R

Where to go next