Build a Self-Paced Reading Experiment Without JavaScript

Psycholinguists who run self-paced reading in PCIbex or have looked for a jsPsych plugin can follow this tutorial to build a moving-window study in Potato 2.13.1 from a YAML config and three JSON survey files, with no JavaScript to write. The study shows 24 garden-path items from a published benchmark in two Latin-square lists with fillers, asks a comprehension question after every sentence, and sends participants back to Prolific with a completion code. The export has one row per word, and a short Python script turns the rows into residual reading times at the disambiguating verb. A maze version of the study and an import from PCIbex come at the end. The configs, commands, and exported rows on this page come from runs of Potato 2.12.1 with 12 simulated readers in headless Chrome.

Tools for self-paced reading

Self-paced reading measures how long a reader spends on each word of a sentence (Just, Carpenter & Woolley, 1982). The reader presses a key to move from one word to the next. In the moving-window version, the sentence first appears as dashes, each key press reveals the next word, and the word before is masked again. A slowdown at one word shows where the reader had trouble.

PCIbex is built around sentence-processing experiments. PCIbex's DashedSentence and Maze controllers run moving-window reading and the maze task from a PennController script, and PCIbex Farm hosts the experiment. jsPsych users write a timeline around a self-paced reading plugin from jspsych-contrib, such as plugin-self-paced-reading or plugin-spr. PCIbex is the better fit when a trial needs PennController elements Potato has no key for, such as newTimer or custom JavaScript in newFunction. Potato runs the reading from a presentation block under trial_timing, and the same YAML file sets up the rest of the session, from consent to the return to Prolific. Behavioral Experiment Tools Compared sets the tools side by side.

The study

The items are main-verb/reduced-relative (MV/RR) garden-path sentences. The ambiguous version reads "The suspect sent the file deserved further investigation given the new evidence." A reader takes sent as the main verb and has to reanalyze at deserved. The unambiguous version adds who was ("The suspect who was sent the file deserved further investigation given the new evidence.") and rules out the main-verb reading before the verb arrives. The prediction is longer reading times at deserved and the next one or two words in the ambiguous version.

The 24 MV/RR items, the comprehension questions and answers, and the position of each disambiguating word come from the Syntactic Ambiguity Processing (SAP) benchmark (Huang et al., 2024). The SAP repository is released under the MIT license and has the items in Items for all subsets.xlsx, on the sheet NPSNPZMVRR. The 12 fillers and 2 practice sentences were written for this tutorial.

Each participant reads every item once, in one version. List 1 has the odd-numbered items in the ambiguous version and the even-numbered items in the unambiguous version, and list 2 has the reverse. Each list therefore has 12 items per condition. Both lists add the same 12 fillers and open with the 2 practice sentences. Each participant reads 38 sentences, and a yes-or-no question follows every sentence.

Installing Potato and laying out the project

Install the release this tutorial was checked against:

pip install potato-annotation==2.13.1

The project folder has the config, two list files, and three survey pages:

spr-study/
├── config.yaml
├── data/list1.jsonl
├── data/list2.jsonl
├── surveys/consent.json
├── surveys/instructions.json
└── surveys/background.json

Building the two lists

The list files are JSON Lines, with one sentence per line. The script below reads the SAP spreadsheet, places each item in the two lists in opposite versions, shuffles each list once with a fixed seed, and writes the files. The script needs pandas and openpyxl and reads the fillers and practice sentences from fillers.jsonl and practice.jsonl. The two files use the same fields as the item lines:

import json
import random
 
import pandas as pd
 
sap = pd.read_excel("Items for all subsets.xlsx", sheet_name="NPSNPZMVRR")
mvrr = sap[sap.condition == "MVRR_UAMB"]
fillers = [json.loads(line) for line in open("fillers.jsonl")]
practice = [json.loads(line) for line in open("practice.jsonl")]
 
rng = random.Random(20261029)
for n, name in enumerate(["list1", "list2"]):
    rows = []
    for _, r in mvrr.iterrows():
        item = int(r["item"])
        ambiguous = (item % 2 == 1) == (n == 0)  # the Latin square
        sentence = r["ambiguous"] if ambiguous else r["Unnamed: 3"]
        verb = r["disambPositionAmb"] if ambiguous else r["disambPositionUnamb"]
        rows.append({
            "id": f"mvrr{item:02d}_{'amb' if ambiguous else 'unamb'}",
            "item": item,
            "condition": "ambiguous" if ambiguous else "unambiguous",
            "sentence": " ".join(sentence.split()),
            "question": r["Question"].strip(),
            "answer": r["Option1"] if r["Answer"] == 1 else r["Option0"],
            "critical_region": int(verb) - 1,  # SAP counts words from 1
        })
    rows += [{**f, "id": f"{f['id']}_{name}"} for f in fillers]
    while True:  # no two items of one condition in a row
        rng.shuffle(rows)
        if all(a["condition"] == "filler" or a["condition"] != b["condition"]
               for a, b in zip(rows, rows[1:])):
            break
    with open(f"data/{name}.jsonl", "w") as out:
        for row in [{**p, "id": f"{p['id']}_{name}"} for p in practice] + rows:
            out.write(json.dumps(row) + "\n")

One line of data/list1.jsonl reads:

{"id": "mvrr01_amb", "item": 1, "condition": "ambiguous", "sentence": "The suspect sent the file deserved further investigation given the new evidence.", "question": "Did the suspect send the file to someone?", "answer": "No", "critical_region": 5}

critical_region is the position of the disambiguating verb, counted from 0 as in Potato's export. The answer field is for the analysis, because Potato records the key the participant pressed and does not score the answer. Item IDs must be unique across both files. The fillers and practice sentences therefore take the list name as a suffix, as in filler01_list1.

Potato serves a list's items in file order. Every participant on list 1 reads the same pseudo-random order, with the practice sentences first. For a second order, add levels such as list1b to the list factor, each level with a separate file.

The config

The config below is complete and passes python -m potato.validate_cli config.yaml --strict on 2.12.1 and on 2.13.1:

annotation_task_name: Garden-path self-paced reading
task_dir: .
output_annotation_dir: annotation_output/
data_files: []
item_properties:
  id_key: id
  text_key: sentence
 
login:
  type: url_direct
  url_argument: PROLIFIC_PID
user_config:
  allow_all_users: true
  users: []
 
crowdsourcing:
  provider: prolific
  prolific:
    completion:
      code: C1READING
 
conditions:
  factors:
    list: [list1, list2]
 
assignment_strategy: batch
batch_assignment:
  auto_assign_annotators: true
  group_by_condition: list
  groups:
    - {name: list1, data_file: data/list1.jsonl}
    - {name: list2, data_file: data/list2.jsonl}
 
phases:
  order: [consent, instructions, annotation, background]
  consent:
    type: consent
    file: surveys/consent.json
  instructions:
    type: instructions
    file: surveys/instructions.json
  annotation:
    type: annotation
  background:
    type: poststudy
    file: surveys/background.json
 
trial_timing:
  enabled: true
  fixation_ms: 400
  iti_ms: 500
  presentation:
    mode: moving_window
    field: sentence
    advance_key: space
    question_field: question
  response:
    schema: comprehension
    keys: {f: "Yes", j: "No"}
 
annotation_schemes:
  - annotation_type: radio
    name: comprehension
    description: Answer the question
    labels: ["Yes", "No"]
    label_requirement:
      required: true
 
export_include_phase_data: true

The presentation block turns each item into a moving-window trial. field: sentence names the item field to show, and with no region_separator each word is one region. To time phrases instead of words, mark the phrase boundaries with | in the sentence and set region_separator: "|". question_field: question shows each sentence's question after the last word, and a sentence with an empty question ends without a question. The response block maps F to yes and J to no, and the comprehension scheme stores the answer.

fixation_ms: 400 shows a fixation cross before each sentence, and iti_ms: 500 leaves half a second between sentences. Neither the reading nor the question has a deadline. In 2.13.1, a trial_duration_ms applies to the question only, never to the reading.

The list factor and batch_assignment.group_by_condition: list give each participant one list, and data_files stays empty because Potato loads the items from the two group files. Assignment is balanced by default, and each new participant goes to the list with fewer participants. The config leaves balance_on at the default, assigned. Under balance_on: completed in Potato 2.12.1, a participant counted toward a list only after finishing, and twelve simulated readers who started within a second of each other split 8 and 4 between the lists, against 6 and 6 under the default. Potato 2.13.1 also counts participants still working within in_progress_minutes of their last activity, 60 by default, so completed replaces dropouts without that split.

login.type: url_direct reads the participant's ID from the PROLIFIC_PID in the study link. At the end, the crowdsourcing block shows the completion code C1READING with a "Return to Prolific" button. Run a Stroop Task on Your Own Server With jsPsych covers the Prolific study URL, a screen-out code for a screener, and moving the study to a public server. export_include_phase_data: true adds the survey answers to the export.

The survey pages

Each survey page is a JSON list of questions. The consent question lists the one answer under required_label. The server checks required_label on survey pages, and from 2.13.1 the server checks required: true as well. In 2.12.1, required: true was checked only in the browser. surveys/consent.json reads:

[
  {
    "name": "study_information",
    "annotation_type": "pure_display",
    "description": "About this study",
    "labels": [
      "You will read 38 sentences one word at a time and answer a yes-or-no question after each. The study takes about 15 minutes.",
      "Your key presses and answers are stored on a university server under your Prolific ID.",
      "If you do not want to take part, close this page and return the study on Prolific."
    ]
  },
  {
    "name": "consent",
    "annotation_type": "radio",
    "description": "Do you agree to take part in this study?",
    "labels": ["I agree to take part"],
    "label_requirement": {"required_label": ["I agree to take part"]}
  }
]

The instructions page in surveys/instructions.json names the keys before the practice sentences:

[
  {
    "name": "how_to_read",
    "annotation_type": "pure_display",
    "description": "How the reading works",
    "labels": [
      "Each sentence first appears as a row of dashes. Press the space bar to see the first word, and press it again for each next word. Each word disappears when the next one appears.",
      "Read at your normal pace. After the last word, a question appears. Press F for yes and J for no.",
      "The first two sentences are practice."
    ]
  }
]

The background page in surveys/background.json asks about native language after the reading. Listing both answers under required_label makes the server require an answer without requiring a particular one:

[
  {
    "name": "native_english",
    "annotation_type": "radio",
    "description": "Is English your native language?",
    "labels": ["Yes", "No"],
    "label_requirement": {"required_label": ["Yes", "No"]}
  },
  {
    "name": "comments",
    "annotation_type": "text",
    "description": "Anything else you want to tell us? (optional)"
  }
]

Testing on your machine

Start the server from the project folder:

potato start config.yaml -p 8000

Open http://localhost:8000/?PROLIFIC_PID=TEST1&STUDY_ID=s1&SESSION_ID=x1, with a new PROLIFIC_PID for each test participant. After the consent and instructions pages, each sentence opens with a fixation cross and then appears as dashes, with "Space = next word" underneath. The display uses a monospace font, and each mask has one dash per character of the hidden word. The line therefore does not shift as words appear. The first press of the space bar shows the first word. Each later press shows the next word and masks the previous word, and the press after the last word brings up the question with "F = Yes" and "J = No" below the question. Answering loads the next sentence.

Participants need a keyboard because the space bar advances the sentence. Potato hides the Back button on the reading pages because trial_timing.disable_back_navigation defaults to true. Reloading the page partway through a sentence starts the sentence over. The trial then has reloads set to 1 and the flag reloaded_mid_trial, and the trial's word times come from the second reading. Test participants count toward the lists and appear in the export. Delete annotation_output/ and project.sqlite before launch.

Exporting reading times

Export the study as CSV:

python -m potato.export --config config.yaml --format csv --output results/

The run with 12 simulated readers wrote these files:

The rows below are words 4 to 8 of one ambiguous item for one reader. The item is "The assistant manager assigned the training seemed unnecessarily demanding to new staff." The disambiguating verb is word 6, seemed:

user_id,instance_id,condition_list,region,text,rt_ms,correct,side,errors,trial_outcome,onset_epoch_ms
r01,mvrr12_amb,list2,4,the,295.9,,,,response,1793287302649
r01,mvrr12_amb,list2,5,training,458.5,,,,response,1793287302649
r01,mvrr12_amb,list2,6,seemed,553.2,,,,response,1793287302649
r01,mvrr12_amb,list2,7,unnecessarily,564.2,,,,response,1793287302649
r01,mvrr12_amb,list2,8,demanding,433.9,,,,response,1793287302649

rt_ms runs from the display frame on which the word appeared to the next press of the space bar. The correct, side, and errors columns stay empty outside the maze task. trial_outcome is response because the sentence ended with an answered question. regions.csv has the list condition but none of the item's own fields. Item fields such as condition and critical_region come from the list files through a join on instance_id.

In trials.csv, rt_ms runs from the question's onset to the answer. Each trial also records the frame interval the participant's browser measured and the browser's timer resolution, 8.3 ms and 0.1 ms in the headless test. The two columns show what each participant's machine could deliver.

Residual reading times in Python

Raw reading times grow with word length and differ between readers. A residual reading time removes both effects by fitting each reader's times on word length and keeping the residual (Ferreira & Clifton, 1986). The analysis script below drops the practice sentences, readers below 80% accuracy on the questions, the reloaded trial, and times outside 100 to 2,000 ms. Next, the script computes residuals per reader and averages the residuals by condition at the word before the verb, the verb, and the two words after the verb. The last step fits a mixed model on the verb and the two following words, with random intercepts for participants and items:

import numpy as np
import pandas as pd
import statsmodels.formula.api as smf
 
items = pd.concat([pd.read_json(f"data/{f}.jsonl", lines=True) for f in ("list1", "list2")])
items = items.rename(columns={"id": "instance_id"})
trials = pd.read_csv("results/trials.csv").merge(items, on="instance_id")
regions = pd.read_csv("results/regions.csv").merge(items, on="instance_id")
 
# Comprehension accuracy per participant, practice excluded.
trials = trials[trials.condition != "practice"]
trials["correct"] = trials.label == trials.answer
accuracy = trials.groupby("user_id").correct.mean()
keep = accuracy[accuracy >= 0.8].index
print(accuracy.round(2).to_dict())
 
# Drop practice, low-accuracy readers, reloaded trials, and extreme times.
flagged = trials[trials["flags"].fillna("").str.contains("reloaded_mid_trial")]
rt = regions[(regions.condition != "practice") & regions.user_id.isin(keep)]
rt = rt.merge(flagged[["user_id", "instance_id"]], how="left", indicator=True)
rt = rt[(rt._merge == "left_only") & rt.rt_ms.between(100, 2000)].copy()
 
# Residual reading time: each reader's times regressed on word length.
rt["length"] = rt.text.str.replace(r"[^A-Za-z]", "", regex=True).str.len()
 
 
def residuals(d):
    slope, intercept = np.polyfit(d.length, d.rt_ms, 1)
    return d.rt_ms - (intercept + slope * d.length)
 
 
rt["residual"] = pd.concat([residuals(d) for _, d in rt.groupby("user_id")])
 
# Positions relative to the disambiguating verb, critical items only.
crit = rt[rt.condition.isin(["ambiguous", "unambiguous"])].copy()
crit["position"] = (crit.region - crit.critical_region).astype(int)
window = crit[crit.position.between(-1, 2)]
window.to_csv("window.csv", index=False)
print(window.pivot_table(index="position", columns="condition",
                         values="residual", aggfunc="mean").round(1))
 
# Mixed model at the verb and the two words after it.
spill = window[window.position >= 0]
fit = smf.mixedlm("residual ~ condition", spill, groups="user_id",
                  vc_formula={"item": "0 + C(item)"}).fit()
print(fit.summary().tables[1])

Run from the project folder on the test export, the script printed:

{'r01': 0.92, 'r02': 0.83, 'r03': 0.89, 'r04': 0.97, 'r05': 0.94, 'r06': 0.67, 'r07': 0.92, 'r08': 0.89, 'r09': 0.89, 'r10': 0.94, 'r11': 0.81, 'r12': 0.94}
condition  ambiguous  unambiguous
position
-1               0.8          2.3
 0              80.7         -7.8
 1              56.4        -13.8
 2              18.5        -10.4
                            Coef. Std.Err.        z  P>|z|   [0.025   0.975]
Intercept                  51.865    2.559   20.268  0.000   46.850   56.881
condition[T.unambiguous]  -62.542    3.626  -17.249  0.000  -69.648  -55.435
item Var                    0.000    1.955

The run used Potato 2.12.1 and simulated readers. The headless-browser driver set each word's delay from the word's length plus random noise, and added 90, 60, and 25 ms at the verb and the next two words of every ambiguous item. The analysis recovered differences of 89, 70, and 29 ms between the conditions at those three positions. The path from key press to export to residual therefore kept the effect the driver inserted, and the differences say nothing about how people read these sentences. Reader r06 answered 67% of the questions correctly and was dropped. The item variance is zero because the simulated items have no item effects, and statsmodels warns that the estimate is on the boundary of the parameter space.

The script also writes the four positions around the verb to window.csv for an analysis in R. The lme4 model below adds random condition slopes for participants and items. R was not run for this page:

library(lme4)
window <- read.csv("window.csv")
spill <- subset(window, position >= 0)
fit <- lmer(residual ~ condition + (1 + condition | user_id) + (1 + condition | item),
            data = spill)
summary(fit)

A maze version

The maze task shows two words side by side at each position, and the reader picks the word that continues the sentence (Boyce, Futrell & Levy, 2020). For a maze version of the study, replace the trial_timing and annotation_schemes blocks of the config above with the blocks below:

trial_timing:
  enabled: true
  fixation_ms: 400
  iti_ms: 500
  presentation:
    mode: maze
    field: sentence
    distractor_field: distractors
    maze_keys: {left: e, right: i}
    on_error: redo
 
annotation_schemes:
  - annotation_type: pure_display
    name: maze_instructions
    description: How to read
    labels:
      - Press E for the left word or I for the right word, whichever continues the sentence.

Each item then needs a distractors field with one wrong word per word of the sentence. When the field has one word fewer than the sentence, Potato pairs the first word with x-x-x:

{"id": "mvrr01_amb", "item": 1, "condition": "ambiguous", "sentence": "The suspect sent the file deserved further investigation given the new evidence.", "distractors": "ran very of went ninety whose upon slept seemed of often.", "critical_region": 5}

The distractors above were written by hand for a test run. A-maze picks each distractor with a language model, from words of similar length and frequency that do not continue the sentence. The Maze repository has the generation code, and the generation runs before you build the list files.

The maze version asks no comprehension question and has no response block. On a timed page without a response block, Potato accepts only pure_display schemes, and the instruction is the page's one scheme. on_error: redo marks a wrong choice and waits for the right word, and on_error: stop ends the sentence at the first error instead. In a test run with two simulated readers, regions.csv filled correct and side for every word, and trial_outcome was completed because no question follows a maze sentence in this config. One reader pressed the wrong key once at deserved. The row for deserved has errors set to 1, and the row's rt_ms of 1,072.6 runs from the pair's onset to the correct press.

Importing a PCIbex experiment

potato import -f ibex converts an existing PCIbex experiment. Point the command at the folder with data_includes/ and chunk_includes/:

potato import -f ibex -i my_experiment/ -o my-study/

The test experiment had one Template("items.csv", ...) trial and a Group column with the levels A and B. The trial showed newController("DashedSentence", {s: row.Sentence}) and then asked a question with newText(row.Question) and newKey("FJ"). The importer wrote a moving-window config with a list factor and one data file per group, and the config passed validate_cli --strict. IMPORT_REPORT.md listed the Sequence ordering and the newHtml instructions screen as not translated. On Potato 2.12.1, the comprehension question was dropped without a line in the report, and question_field and a response block had to be added by hand, as in the config above. Run on the same experiment, the 2.13.1 importer kept the question as a comprehension scheme with question_field and a response block, and the report notes that the F and J keys became Yes and No. Coming from Ibex lists the PCIbex constructs the importer translates.

Where to go next

• Self-Paced Reading and the Maze Task, for the cumulative and centered modes, mask_char, and every flag the server sets
• Experiment Flow, for lists, conditions, and screen-outs
• Timed Trials, for the fixation, deadline, and timing record that reading trials share
• Run a Stroop Task on Your Own Server With jsPsych, for the Prolific setup and the move to a public server
• Online Behavioral Experiments, for screeners, SONA, and the other export formats
• Potato for linguistics research and Potato for psychology research, for the other features each field uses

References

• Boyce, V., Futrell, R., & Levy, R. P. (2020). Maze Made Easy: Better and easier measurement of incremental processing difficulty. Journal of Memory and Language, 111, 104082. https://doi.org/10.1016/j.jml.2019.104082
• Ferreira, F., & Clifton, C. (1986). The independence of syntactic processing. Journal of Memory and Language, 25(3), 348-368. https://doi.org/10.1016/0749-596X(86)90006-9
• Huang, K.-J., Arehalli, S., Kugemoto, M., Muxica, C., Prasad, G., Dillon, B., & Linzen, T. (2024). Large-scale benchmark yields no evidence that language model surprisal explains syntactic disambiguation difficulty. Journal of Memory and Language, 137, 104510. https://doi.org/10.1016/j.jml.2024.104510
• Just, M. A., Carpenter, P. A., & Woolley, J. D. (1982). Paradigms and processes in reading comprehension. Journal of Experimental Psychology: General, 111(2), 228-238. https://doi.org/10.1037/0096-3445.111.2.228