Runtime Phases: jsPsych Timelines, Headphone Checks and External Tasks

New in v2.12.0

A runtime phase is a page of a Potato study that runs a task instead of showing a page of questions. Potato serves the page, passes the task the participant's ID, conditions and chosen survey answers, stores what the task records, and moves the participant on when it ends. Consent, surveys, annotation, the hand-off to Prolific or SONA, and the export stay in Potato.

Every runtime phase:

• sits in the prestudy or poststudy slot, decided by its place in phases.order relative to annotation (or set with slot:), and takes show_when, exit_when and counterbalancing like any other phase page (Experiment Flow);
• runs under a Content-Security-Policy that allows scripts and network requests only to the Potato server;
• cannot be skipped, because until the task's data is stored the server refuses the phase form and the annotation page's Next;
• stores its data in the project database (project.sqlite, table runtime_runs) instead of user_state.json, and exports it with --format runtime, or --format jspsych for jsPsych pages only.

These keys apply to every runtime phase:

jsPsych pages

phases:
  order: [consent, stroop, annotation, debrief]
  stroop:
    type: jspsych
    timeline: tasks/stroop.js
    plugins: [instructions, html-keyboard-response]

Potato ships jsPsych 8.3.0 and 15 of its plugins, so a study that uses only those loads nothing from a CDN and runs on a server with no internet access:

html-keyboard-response, html-button-response, html-slider-response, image-keyboard-response, image-button-response, audio-keyboard-response, audio-button-response, video-keyboard-response, instructions, preload, fullscreen, survey-likert, survey-text, survey-multi-choice and call-function.

Other plugins, including ones from jspsych-contrib, go in extra_scripts as browser builds, which is the index.browser.min.js file of the npm package.

A jsPsych page's exit_when is checked when the timeline finishes, so the run's data is already stored for a participant it screens out. To exclude people before they do the task, put the screener on its own page in front of it.

The timeline file

window.potatoTimeline = function (jsPsych, potato) {
    var trial = {
        type: jsPsychHtmlKeyboardResponse,
        stimulus: '<p style="font-size:48px">RED</p>',
        choices: ['r', 'g', 'b']
    };
    return [trial];
};

window.potatoTimeline is a timeline array, or a function that returns one. Potato calls the function with the jsPsych instance it created and a potato object:

Do not call initJsPsych yourself, and do not set on_finish on the experiment, because Potato does both. Trial-level on_finish and the rest of the jsPsych API work as usual. Every trial gets potato_user, potato_page, potato_run_id, potato_attempt and one condition_<factor> column per factor.

Stimuli load through potato.asset(), and the preload plugin preloads them as in any jsPsych experiment:

{
    type: jsPsychImageKeyboardResponse,
    stimulus: potato.asset('faces/f01.png')
}

Mouse tracking

extensions: [mouse-tracking] loads @jspsych/extension-mouse-tracking 1.2.0 and registers it. A trial turns it on in the usual way, and each such trial gets mouse_tracking_data, a list of {x, y, t, event} samples, and mouse_tracking_targets:

{
    type: jsPsychHtmlButtonResponse,
    stimulus: '<p id="stim">Which is bigger?</p>',
    choices: ['3', '8'],
    extensions: [{type: jsPsychExtensionMouseTracking, params: {targets: ['#stim']}}]
}

The WebGazer eye-tracking extension is not shipped. WebGazer needs 'unsafe-eval', which the page's Content-Security-Policy does not allow.

When the timeline ends

The page sends the data to the server, shows "Saving your responses", and goes to the next page once the server has stored it. If the save fails, it retries and then shows a Try again button, and the participant is not moved on without their data.

A reload starts the timeline over, because jsPsych cannot resume a timeline midway. With the default save: on_finish, trials done before a reload are lost, and only the page's page_loads count in participants.csv shows that it happened. With save: incremental, a run that has sent data is kept as incomplete and a reload starts a new attempt, so an analysis can exclude second attempts or keep them. Opening the task in a second tab counts as a reload, and the first tab's run can no longer finish.

Headphone check

A headphone_check page checks that a listening-study participant is wearing headphones and not listening on loudspeakers.

phases:
  order: [consent, headphones, annotation]
  headphones:
    type: headphone_check
    test: huggins
    trials: 6
    pass_threshold: 5

The page asks the participant to put on headphones and set a comfortable volume, then runs the trials. Each trial plays three sounds once, and the participant picks one.

• huggins (Milne et al., 2021) plays three bursts of noise. In one, a narrow band around 600 Hz is phase-inverted in one ear, which is heard as a faint tone over headphones and not over loudspeakers. The participant answers "Which sound had a faint tone hidden in the noise?"
• antiphase (Woods et al., 2017) plays three 200 Hz tones. One is 6 dB quieter, and one is in antiphase between the ears, which partly cancels over loudspeakers and sounds quietest there. The participant answers "Which sound was the quietest?"

The server makes the sounds when the page asks for them, so no audio files ship with Potato. It also picks which of the three sounds is the answer, and sends each trial's three sounds as one WAV that does not say which it is. The browser sends only the choice. A target or "correct" value in what it posts is ignored, as are a second answer to the same trial and an answer to a sound that was never fetched, so made-up answers score at chance.

The page answers two questions that later pages can test: <page>_passed (yes or no) and <page>_correct (a number, or <page>_correct_huggins and <page>_correct_antiphase for test: both). With on_fail: continue, test the answer yourself:

phases:
  speakers_note:
    type: prestudy
    file: surveys/speakers.json
    show_when:
      - {schema: headphones_passed, operator: equals, value: "no"}

Passing does not prove the participant wore headphones. Someone on loudspeakers can guess right one time in three on each trial, and a participant who writes code to analyse the downloaded sound can find the answer, as with any headphone screen whose sounds reach the browser.

External tasks

A type: external page sends the participant to a task on another server and takes them back when it ends. The task runs on the other server, and Potato hands over the participant and records that they came back.

phases:
  order: [consent, game, annotation]
  game:
    type: external
    url: "https://otree.example.org/room/public_goods/?participant_label={participant}&potato_return={return_url}"
    return_params: [payoff]
    description: The next part is a short game with three other participants.
    button_text: Start the game

The placeholders in url are URL-encoded: {participant} (the Potato user ID), {return_url} (the address that brings the participant back), {run_id} and {condition.<factor>}.

The other system sends the participant to {return_url} when they finish, adding any results as query parameters, as in <return_url>&payoff=4.50. The return must arrive in the same browser session and name the participant's current attempt, and an out-of-date link shows "This link is out of date" and changes nothing. The page then answers <page>_returned (yes) and <page>_<param> for each listed parameter, which later pages can test, and the parameters are stored with the run.

Signed returns

The return link is in the address the participant is sent to, so a participant can read it, change payoff=4.50 to payoff=99 and open it without playing. Without return_secret, the returned parameters are self-reported, and they should not decide a payment unless you check them against the other system's records.

With return_secret, the other system adds sig to the return link: the hex HMAC-SHA256, keyed with the secret, of the run ID followed by &<param>=<value> for each parameter in return_params, sorted by name. Each name and value is percent-encoded with every reserved character escaped, so a value containing & or = cannot pass for another parameter. In Python:

import hashlib, hmac
from urllib.parse import quote
message = run_id + "".join(f"&{quote(k, safe='')}={quote(params[k], safe='')}"
                           for k in sorted(params))
sig = hmac.new(secret.encode(), message.encode(), hashlib.sha256).hexdigest()
# redirect to f"{return_url}&{urlencode({**params, 'sig': sig})}"

A return without sig, or with one that does not match, gets status 409 and the page "This return could not be confirmed". The page is not marked done, so a correctly signed return still works afterwards. The secret has to stay on the other system's server. A hosted survey that builds its end redirect in the participant's browser cannot keep it private, so signing does not help there.

• oTree: send participants to a room URL with participant_label={participant}. The app has to carry the return URL itself: read the potato_return parameter when the participant arrives, keep it in participant.vars, and redirect to it from the last page. How the first page reads URL parameters depends on the oTree version, so check its documentation.
• Empirica: pass {participant} as the participantKey URL parameter.
• A hosted survey or experiment (Qualtrics, Gorilla, Pavlovia): put {participant} in an embedded-data or URL parameter, and set the end redirect to the return URL.

Data and export

python -m potato.export --config config.yaml --format runtime --output ./export/

The export writes these files for each runtime the project uses, where <kind> is jspsych, headphone_check or external:

Runs that did not finish have status started or partial. A headphone-check trial records test, trial, index (its position in the run) and response (the sound chosen, 1 to 3), plus three values the server adds: target (the correct sound), correct, and scored, which is false for an answer the server did not count.

Security and timing

The timeline is code you wrote, and it runs on the Potato server's origin with the participant's session, the same trust as a custom HTML instructions page. Its Content-Security-Policy allows scripts, styles and media only from the Potato server and any csp.extra_sources, and network requests only to the Potato server, so a stimulus that shows participant text cannot load from or send to anything else. Escape such text before putting it in HTML all the same.

Every file the page names must be inside the task directory. The asset route serves stimulus file types only, never .js or .html from assets_dir, and sends each with a sandboxing CSP of its own, so an SVG opened directly runs no script. csp.extra_sources takes single https:// origins and refuses wildcards.

jsPsych's own timing applies on the page, because Potato's annotation scripts do not load there. jsPsych's timing accuracy documentation reports that response times measured in JavaScript tend to run 10-40 ms longer than in lab software, with similar variance.

Examples

python potato/flask_server.py start examples/advanced/jspsych-stroop/config.yaml -p 8000
python potato/flask_server.py start examples/audio/listening-test-p808/config.yaml -p 8000

The first is a Stroop task between a screener and a word-rating task, with feedback shown to half of the participants. The second is a headphone check followed by speech clips rated on the ACR scale.

Further Reading

• Experiment Flow - Conditions, show_when, exit_when and counterbalancing
• Coming from jsPsych - jsPsych concepts mapped to Potato config keys
• Listening Tests - MUSHRA, ACR and the other listening-test methods
• Timed Trials - Speeded responses on annotation items without a jsPsych timeline

For implementation details, see the source documentation on runtime phases and the jsPsych phase.