# Build a Self-Paced Reading Experiment Without JavaScript

Source: https://www.potatoannotator.com/blog/self-paced-reading-experiment

Psycholinguists who run self-paced reading in PCIbex or have looked for a jsPsych plugin can follow this tutorial to build a moving-window study in Potato 2.13.1 from a YAML config and three JSON survey files, with no JavaScript to write. The study shows 24 garden-path items from a published benchmark in two Latin-square lists with fillers, asks a comprehension question after every sentence, and sends participants back to Prolific with a completion code. The export has one row per word, and a short Python script turns the rows into residual reading times at the disambiguating verb. A maze version of the study and an import from PCIbex come at the end. The configs, commands, and exported rows on this page come from runs of Potato 2.12.1 with 12 simulated readers in headless Chrome.

## Tools for self-paced reading

Self-paced reading measures how long a reader spends on each word of a sentence ([Just, Carpenter & Woolley, 1982](https://doi.org/10.1037/0096-3445.111.2.228)). The reader presses a key to move from one word to the next. In the moving-window version, the sentence first appears as dashes, each key press reveals the next word, and the word before is masked again. A slowdown at one word shows where the reader had trouble.

[PCIbex](https://www.pcibex.net/) is built around sentence-processing experiments. PCIbex's `DashedSentence` and `Maze` controllers run moving-window reading and the maze task from a PennController script, and PCIbex Farm hosts the experiment. jsPsych users write a timeline around a self-paced reading plugin from [jspsych-contrib](https://github.com/jspsych/jspsych-contrib), such as `plugin-self-paced-reading` or `plugin-spr`. PCIbex is the better fit when a trial needs PennController elements Potato has no key for, such as `newTimer` or custom JavaScript in `newFunction`. Potato runs the reading from a `presentation` block under `trial_timing`, and the same YAML file sets up the rest of the session, from consent to the return to Prolific. [Behavioral Experiment Tools Compared](/docs/guides/behavioral-experiment-tools-compared) sets the tools side by side.

## The study

The items are main-verb/reduced-relative (MV/RR) [garden-path sentences](https://en.wikipedia.org/wiki/Garden-path_sentence). The ambiguous version reads "The suspect sent the file deserved further investigation given the new evidence." A reader takes *sent* as the main verb and has to reanalyze at *deserved*. The unambiguous version adds *who was* ("The suspect who was sent the file deserved further investigation given the new evidence.") and rules out the main-verb reading before the verb arrives. The prediction is longer reading times at *deserved* and the next one or two words in the ambiguous version.

The 24 MV/RR items, the comprehension questions and answers, and the position of each disambiguating word come from the Syntactic Ambiguity Processing (SAP) benchmark ([Huang et al., 2024](https://doi.org/10.1016/j.jml.2024.104510)). The [SAP repository](https://github.com/caplabnyu/sapbenchmark) is released under the MIT license and has the items in `Items for all subsets.xlsx`, on the sheet `NPSNPZMVRR`. The 12 fillers and 2 practice sentences were written for this tutorial.

Each participant reads every item once, in one version. List 1 has the odd-numbered items in the ambiguous version and the even-numbered items in the unambiguous version, and list 2 has the reverse. Each list therefore has 12 items per condition. Both lists add the same 12 fillers and open with the 2 practice sentences. Each participant reads 38 sentences, and a yes-or-no question follows every sentence.

## Installing Potato and laying out the project

Install the release this tutorial was checked against:

```bash
pip install potato-annotation==2.13.1
```

The project folder has the config, two list files, and three survey pages:

```text
spr-study/
├── config.yaml
├── data/list1.jsonl
├── data/list2.jsonl
├── surveys/consent.json
├── surveys/instructions.json
└── surveys/background.json
```

## Building the two lists

The list files are JSON Lines, with one sentence per line. The script below reads the SAP spreadsheet, places each item in the two lists in opposite versions, shuffles each list once with a fixed seed, and writes the files. The script needs pandas and openpyxl and reads the fillers and practice sentences from `fillers.jsonl` and `practice.jsonl`. The two files use the same fields as the item lines:

```python
import json
import random

import pandas as pd

sap = pd.read_excel("Items for all subsets.xlsx", sheet_name="NPSNPZMVRR")
mvrr = sap[sap.condition == "MVRR_UAMB"]
fillers = [json.loads(line) for line in open("fillers.jsonl")]
practice = [json.loads(line) for line in open("practice.jsonl")]

rng = random.Random(20261029)
for n, name in enumerate(["list1", "list2"]):
    rows = []
    for _, r in mvrr.iterrows():
        item = int(r["item"])
        ambiguous = (item % 2 == 1) == (n == 0)  # the Latin square
        sentence = r["ambiguous"] if ambiguous else r["Unnamed: 3"]
        verb = r["disambPositionAmb"] if ambiguous else r["disambPositionUnamb"]
        rows.append({
            "id": f"mvrr{item:02d}_{'amb' if ambiguous else 'unamb'}",
            "item": item,
            "condition": "ambiguous" if ambiguous else "unambiguous",
            "sentence": " ".join(sentence.split()),
            "question": r["Question"].strip(),
            "answer": r["Option1"] if r["Answer"] == 1 else r["Option0"],
            "critical_region": int(verb) - 1,  # SAP counts words from 1
        })
    rows += [{**f, "id": f"{f['id']}_{name}"} for f in fillers]
    while True:  # no two items of one condition in a row
        rng.shuffle(rows)
        if all(a["condition"] == "filler" or a["condition"] != b["condition"]
               for a, b in zip(rows, rows[1:])):
            break
    with open(f"data/{name}.jsonl", "w") as out:
        for row in [{**p, "id": f"{p['id']}_{name}"} for p in practice] + rows:
            out.write(json.dumps(row) + "\n")
```

One line of `data/list1.jsonl` reads:

```json
{"id": "mvrr01_amb", "item": 1, "condition": "ambiguous", "sentence": "The suspect sent the file deserved further investigation given the new evidence.", "question": "Did the suspect send the file to someone?", "answer": "No", "critical_region": 5}
```

`critical_region` is the position of the disambiguating verb, counted from 0 as in Potato's export. The `answer` field is for the analysis, because Potato records the key the participant pressed and does not score the answer. Item IDs must be unique across both files. The fillers and practice sentences therefore take the list name as a suffix, as in `filler01_list1`.

Potato serves a list's items in file order. Every participant on list 1 reads the same pseudo-random order, with the practice sentences first. For a second order, add levels such as `list1b` to the `list` factor, each level with a separate file.

## The config

The config below is complete and passes `python -m potato.validate_cli config.yaml --strict` on 2.12.1 and on 2.13.1:

```yaml
annotation_task_name: Garden-path self-paced reading
task_dir: .
output_annotation_dir: annotation_output/
data_files: []
item_properties:
  id_key: id
  text_key: sentence

login:
  type: url_direct
  url_argument: PROLIFIC_PID
user_config:
  allow_all_users: true
  users: []

crowdsourcing:
  provider: prolific
  prolific:
    completion:
      code: C1READING

conditions:
  factors:
    list: [list1, list2]

assignment_strategy: batch
batch_assignment:
  auto_assign_annotators: true
  group_by_condition: list
  groups:
    - {name: list1, data_file: data/list1.jsonl}
    - {name: list2, data_file: data/list2.jsonl}

phases:
  order: [consent, instructions, annotation, background]
  consent:
    type: consent
    file: surveys/consent.json
  instructions:
    type: instructions
    file: surveys/instructions.json
  annotation:
    type: annotation
  background:
    type: poststudy
    file: surveys/background.json

trial_timing:
  enabled: true
  fixation_ms: 400
  iti_ms: 500
  presentation:
    mode: moving_window
    field: sentence
    advance_key: space
    question_field: question
  response:
    schema: comprehension
    keys: {f: "Yes", j: "No"}

annotation_schemes:
  - annotation_type: radio
    name: comprehension
    description: Answer the question
    labels: ["Yes", "No"]
    label_requirement:
      required: true

export_include_phase_data: true
```

The `presentation` block turns each item into a moving-window trial. `field: sentence` names the item field to show, and with no `region_separator` each word is one region. To time phrases instead of words, mark the phrase boundaries with `|` in the sentence and set `region_separator: "|"`. `question_field: question` shows each sentence's question after the last word, and a sentence with an empty `question` ends without a question. The `response` block maps F to yes and J to no, and the `comprehension` scheme stores the answer.

`fixation_ms: 400` shows a fixation cross before each sentence, and `iti_ms: 500` leaves half a second between sentences. Neither the reading nor the question has a deadline. In 2.13.1, a `trial_duration_ms` applies to the question only, never to the reading.

The `list` factor and `batch_assignment.group_by_condition: list` give each participant one list, and `data_files` stays empty because Potato loads the items from the two group files. Assignment is `balanced` by default, and each new participant goes to the list with fewer participants. The config leaves `balance_on` at the default, `assigned`. Under `balance_on: completed` in Potato 2.12.1, a participant counted toward a list only after finishing, and twelve simulated readers who started within a second of each other split 8 and 4 between the lists, against 6 and 6 under the default. Potato 2.13.1 also counts participants still working within `in_progress_minutes` of their last activity, 60 by default, so `completed` replaces dropouts without that split.

`login.type: url_direct` reads the participant's ID from the `PROLIFIC_PID` in the study link. At the end, the `crowdsourcing` block shows the completion code `C1READING` with a "Return to Prolific" button. [Run a Stroop Task on Your Own Server With jsPsych](/blog/stroop-task-jspsych-own-server) covers the Prolific study URL, a screen-out code for a screener, and moving the study to a public server. `export_include_phase_data: true` adds the survey answers to the export.

## The survey pages

Each survey page is a JSON list of questions. The consent question lists the one answer under `required_label`. The server checks `required_label` on survey pages, and from 2.13.1 the server checks `required: true` as well. In 2.12.1, `required: true` was checked only in the browser. `surveys/consent.json` reads:

```json
[
  {
    "name": "study_information",
    "annotation_type": "pure_display",
    "description": "About this study",
    "labels": [
      "You will read 38 sentences one word at a time and answer a yes-or-no question after each. The study takes about 15 minutes.",
      "Your key presses and answers are stored on a university server under your Prolific ID.",
      "If you do not want to take part, close this page and return the study on Prolific."
    ]
  },
  {
    "name": "consent",
    "annotation_type": "radio",
    "description": "Do you agree to take part in this study?",
    "labels": ["I agree to take part"],
    "label_requirement": {"required_label": ["I agree to take part"]}
  }
]
```

The instructions page in `surveys/instructions.json` names the keys before the practice sentences:

```json
[
  {
    "name": "how_to_read",
    "annotation_type": "pure_display",
    "description": "How the reading works",
    "labels": [
      "Each sentence first appears as a row of dashes. Press the space bar to see the first word, and press it again for each next word. Each word disappears when the next one appears.",
      "Read at your normal pace. After the last word, a question appears. Press F for yes and J for no.",
      "The first two sentences are practice."
    ]
  }
]
```

The background page in `surveys/background.json` asks about native language after the reading. Listing both answers under `required_label` makes the server require an answer without requiring a particular one:

```json
[
  {
    "name": "native_english",
    "annotation_type": "radio",
    "description": "Is English your native language?",
    "labels": ["Yes", "No"],
    "label_requirement": {"required_label": ["Yes", "No"]}
  },
  {
    "name": "comments",
    "annotation_type": "text",
    "description": "Anything else you want to tell us? (optional)"
  }
]
```

## Testing on your machine

Start the server from the project folder:

```bash
potato start config.yaml -p 8000
```

Open `http://localhost:8000/?PROLIFIC_PID=TEST1&STUDY_ID=s1&SESSION_ID=x1`, with a new `PROLIFIC_PID` for each test participant. After the consent and instructions pages, each sentence opens with a fixation cross and then appears as dashes, with "Space = next word" underneath. The display uses a monospace font, and each mask has one dash per character of the hidden word. The line therefore does not shift as words appear. The first press of the space bar shows the first word. Each later press shows the next word and masks the previous word, and the press after the last word brings up the question with "F = Yes" and "J = No" below the question. Answering loads the next sentence.

Participants need a keyboard because the space bar advances the sentence. Potato hides the Back button on the reading pages because `trial_timing.disable_back_navigation` defaults to true. Reloading the page partway through a sentence starts the sentence over. The trial then has `reloads` set to 1 and the flag `reloaded_mid_trial`, and the trial's word times come from the second reading. Test participants count toward the lists and appear in the export. Delete `annotation_output/` and `project.sqlite` before launch.

## Exporting reading times

Export the study as CSV:

```bash
python -m potato.export --config config.yaml --format csv --output results/
```

The run with 12 simulated readers wrote these files:

| File | One row per | Contents |
|---|---|---|
| `regions.csv` | word shown | Participant, item, list, word position from 0, the word, `rt_ms`, and the trial's outcome and onset time (6,132 rows) |
| `trials.csv` | sentence | The answer's `key` and `label`, the time to answer, intended and measured fixation and inter-trial times, `frame_ms`, `timer_resolution_ms`, `reloads`, and the server's flags (456 rows) |
| `participants.csv` | participant | `condition_list` and any screen-out |
| `annotations.csv` | participant and sentence | The comprehension answer, one column per label |
| `phase_responses.csv` | survey answer | Consent, native language, and the comment |

The rows below are words 4 to 8 of one ambiguous item for one reader. The item is "The assistant manager assigned the training seemed unnecessarily demanding to new staff." The disambiguating verb is word 6, *seemed*:

```text
user_id,instance_id,condition_list,region,text,rt_ms,correct,side,errors,trial_outcome,onset_epoch_ms
r01,mvrr12_amb,list2,4,the,295.9,,,,response,1793287302649
r01,mvrr12_amb,list2,5,training,458.5,,,,response,1793287302649
r01,mvrr12_amb,list2,6,seemed,553.2,,,,response,1793287302649
r01,mvrr12_amb,list2,7,unnecessarily,564.2,,,,response,1793287302649
r01,mvrr12_amb,list2,8,demanding,433.9,,,,response,1793287302649
```

`rt_ms` runs from the display frame on which the word appeared to the next press of the space bar. The `correct`, `side`, and `errors` columns stay empty outside the maze task. `trial_outcome` is `response` because the sentence ended with an answered question. `regions.csv` has the `list` condition but none of the item's own fields. Item fields such as `condition` and `critical_region` come from the list files through a join on `instance_id`.

In `trials.csv`, `rt_ms` runs from the question's onset to the answer. Each trial also records the frame interval the participant's browser measured and the browser's timer resolution, 8.3 ms and 0.1 ms in the headless test. The two columns show what each participant's machine could deliver.

## Residual reading times in Python

Raw reading times grow with word length and differ between readers. A residual reading time removes both effects by fitting each reader's times on word length and keeping the residual ([Ferreira & Clifton, 1986](https://doi.org/10.1016/0749-596X(86)90006-9)). The analysis script below drops the practice sentences, readers below 80% accuracy on the questions, the reloaded trial, and times outside 100 to 2,000 ms. Next, the script computes residuals per reader and averages the residuals by condition at the word before the verb, the verb, and the two words after the verb. The last step fits a mixed model on the verb and the two following words, with random intercepts for participants and items:

```python
import numpy as np
import pandas as pd
import statsmodels.formula.api as smf

items = pd.concat([pd.read_json(f"data/{f}.jsonl", lines=True) for f in ("list1", "list2")])
items = items.rename(columns={"id": "instance_id"})
trials = pd.read_csv("results/trials.csv").merge(items, on="instance_id")
regions = pd.read_csv("results/regions.csv").merge(items, on="instance_id")

# Comprehension accuracy per participant, practice excluded.
trials = trials[trials.condition != "practice"]
trials["correct"] = trials.label == trials.answer
accuracy = trials.groupby("user_id").correct.mean()
keep = accuracy[accuracy >= 0.8].index
print(accuracy.round(2).to_dict())

# Drop practice, low-accuracy readers, reloaded trials, and extreme times.
flagged = trials[trials["flags"].fillna("").str.contains("reloaded_mid_trial")]
rt = regions[(regions.condition != "practice") & regions.user_id.isin(keep)]
rt = rt.merge(flagged[["user_id", "instance_id"]], how="left", indicator=True)
rt = rt[(rt._merge == "left_only") & rt.rt_ms.between(100, 2000)].copy()

# Residual reading time: each reader's times regressed on word length.
rt["length"] = rt.text.str.replace(r"[^A-Za-z]", "", regex=True).str.len()


def residuals(d):
    slope, intercept = np.polyfit(d.length, d.rt_ms, 1)
    return d.rt_ms - (intercept + slope * d.length)


rt["residual"] = pd.concat([residuals(d) for _, d in rt.groupby("user_id")])

# Positions relative to the disambiguating verb, critical items only.
crit = rt[rt.condition.isin(["ambiguous", "unambiguous"])].copy()
crit["position"] = (crit.region - crit.critical_region).astype(int)
window = crit[crit.position.between(-1, 2)]
window.to_csv("window.csv", index=False)
print(window.pivot_table(index="position", columns="condition",
                         values="residual", aggfunc="mean").round(1))

# Mixed model at the verb and the two words after it.
spill = window[window.position >= 0]
fit = smf.mixedlm("residual ~ condition", spill, groups="user_id",
                  vc_formula={"item": "0 + C(item)"}).fit()
print(fit.summary().tables[1])
```

Run from the project folder on the test export, the script printed:

```text
{'r01': 0.92, 'r02': 0.83, 'r03': 0.89, 'r04': 0.97, 'r05': 0.94, 'r06': 0.67, 'r07': 0.92, 'r08': 0.89, 'r09': 0.89, 'r10': 0.94, 'r11': 0.81, 'r12': 0.94}
condition  ambiguous  unambiguous
position
-1               0.8          2.3
 0              80.7         -7.8
 1              56.4        -13.8
 2              18.5        -10.4
                            Coef. Std.Err.        z  P>|z|   [0.025   0.975]
Intercept                  51.865    2.559   20.268  0.000   46.850   56.881
condition[T.unambiguous]  -62.542    3.626  -17.249  0.000  -69.648  -55.435
item Var                    0.000    1.955
```

The run used Potato 2.12.1 and simulated readers. The headless-browser driver set each word's delay from the word's length plus random noise, and added 90, 60, and 25 ms at the verb and the next two words of every ambiguous item. The analysis recovered differences of 89, 70, and 29 ms between the conditions at those three positions. The path from key press to export to residual therefore kept the effect the driver inserted, and the differences say nothing about how people read these sentences. Reader r06 answered 67% of the questions correctly and was dropped. The item variance is zero because the simulated items have no item effects, and statsmodels warns that the estimate is on the boundary of the parameter space.

The script also writes the four positions around the verb to `window.csv` for an analysis in R. The [lme4](https://cran.r-project.org/package=lme4) model below adds random condition slopes for participants and items. R was not run for this page:

```r
library(lme4)
window <- read.csv("window.csv")
spill <- subset(window, position >= 0)
fit <- lmer(residual ~ condition + (1 + condition | user_id) + (1 + condition | item),
            data = spill)
summary(fit)
```

## A maze version

The maze task shows two words side by side at each position, and the reader picks the word that continues the sentence ([Boyce, Futrell & Levy, 2020](https://doi.org/10.1016/j.jml.2019.104082)). For a maze version of the study, replace the `trial_timing` and `annotation_schemes` blocks of the config above with the blocks below:

```yaml
trial_timing:
  enabled: true
  fixation_ms: 400
  iti_ms: 500
  presentation:
    mode: maze
    field: sentence
    distractor_field: distractors
    maze_keys: {left: e, right: i}
    on_error: redo

annotation_schemes:
  - annotation_type: pure_display
    name: maze_instructions
    description: How to read
    labels:
      - Press E for the left word or I for the right word, whichever continues the sentence.
```

Each item then needs a `distractors` field with one wrong word per word of the sentence. When the field has one word fewer than the sentence, Potato pairs the first word with `x-x-x`:

```json
{"id": "mvrr01_amb", "item": 1, "condition": "ambiguous", "sentence": "The suspect sent the file deserved further investigation given the new evidence.", "distractors": "ran very of went ninety whose upon slept seemed of often.", "critical_region": 5}
```

The distractors above were written by hand for a test run. A-maze picks each distractor with a language model, from words of similar length and frequency that do not continue the sentence. The [Maze repository](https://github.com/vboyce/Maze) has the generation code, and the generation runs before you build the list files.

The maze version asks no comprehension question and has no `response` block. On a timed page without a `response` block, Potato accepts only `pure_display` schemes, and the instruction is the page's one scheme. `on_error: redo` marks a wrong choice and waits for the right word, and `on_error: stop` ends the sentence at the first error instead. In a test run with two simulated readers, `regions.csv` filled `correct` and `side` for every word, and `trial_outcome` was `completed` because no question follows a maze sentence in this config. One reader pressed the wrong key once at *deserved*. The row for *deserved* has `errors` set to 1, and the row's `rt_ms` of 1,072.6 runs from the pair's onset to the correct press.

## Importing a PCIbex experiment

`potato import -f ibex` converts an existing PCIbex experiment. Point the command at the folder with `data_includes/` and `chunk_includes/`:

```bash
potato import -f ibex -i my_experiment/ -o my-study/
```

The test experiment had one `Template("items.csv", ...)` trial and a `Group` column with the levels A and B. The trial showed `newController("DashedSentence", {s: row.Sentence})` and then asked a question with `newText(row.Question)` and `newKey("FJ")`. The importer wrote a moving-window config with a `list` factor and one data file per group, and the config passed `validate_cli --strict`. `IMPORT_REPORT.md` listed the `Sequence` ordering and the `newHtml` instructions screen as not translated. On Potato 2.12.1, the comprehension question was dropped without a line in the report, and `question_field` and a `response` block had to be added by hand, as in the config above. Run on the same experiment, the 2.13.1 importer kept the question as a `comprehension` scheme with `question_field` and a `response` block, and the report notes that the F and J keys became Yes and No. [Coming from Ibex](/docs/guides/coming-from-ibex) lists the PCIbex constructs the importer translates.

## Where to go next

- [Self-Paced Reading and the Maze Task](/docs/experiments/self-paced-reading), for the cumulative and centered modes, `mask_char`, and every flag the server sets
- [Experiment Flow](/docs/experiments/experiment-flow#different-items-for-each-cell), for lists, conditions, and screen-outs
- [Timed Trials](/docs/experiments/timed-trials), for the fixation, deadline, and timing record that reading trials share
- [Run a Stroop Task on Your Own Server With jsPsych](/blog/stroop-task-jspsych-own-server), for the Prolific setup and the move to a public server
- [Online Behavioral Experiments](/docs/guides/online-behavioral-experiments), for screeners, SONA, and the other export formats
- [Potato for linguistics research](/use-cases/linguistics) and [Potato for psychology research](/use-cases/psychology), for the other features each field uses

## References

- Boyce, V., Futrell, R., & Levy, R. P. (2020). Maze Made Easy: Better and easier measurement of incremental processing difficulty. *Journal of Memory and Language*, 111, 104082. https://doi.org/10.1016/j.jml.2019.104082
- Ferreira, F., & Clifton, C. (1986). The independence of syntactic processing. *Journal of Memory and Language*, 25(3), 348-368. https://doi.org/10.1016/0749-596X(86)90006-9
- Huang, K.-J., Arehalli, S., Kugemoto, M., Muxica, C., Prasad, G., Dillon, B., & Linzen, T. (2024). Large-scale benchmark yields no evidence that language model surprisal explains syntactic disambiguation difficulty. *Journal of Memory and Language*, 137, 104510. https://doi.org/10.1016/j.jml.2024.104510
- Just, M. A., Carpenter, P. A., & Woolley, J. D. (1982). Paradigms and processes in reading comprehension. *Journal of Experimental Psychology: General*, 111(2), 228-238. https://doi.org/10.1037/0096-3445.111.2.228