Experiment Flow: Conditions, Skips, Screen-outs and Counterbalancing

Assign participants to between-subjects conditions, skip or screen out with rules, counterbalance page order by Latin square, and export every assignment.

New in v2.12.0

Potato 2.12 adds five settings for the design side of an experiment. Each one is a few lines of YAML, and everything a participant was assigned or shown is recorded, so the CSV and TSV exports can write it to participants.csv.

Setting	What it does
`conditions`	Assigns each participant to one cell of a between-subjects design
`show_when` on a phase page	Skips the page unless a rule holds
`exit_when` on a phase page	Ends the study for the participant, as a screener does
`phases.counterbalance`	Orders a group of pages per participant, at random or by Latin square
`randomize_questions` on a survey page	Orders that page's questions per participant

The pages these settings act on are the phase pages described in Multi-Phase Workflows: consent, prestudy, instructions, training, annotation and poststudy, plus the runtime pages that run a jsPsych timeline, a headphone check or an external task.

Conditions

yaml

conditions:
  factors:
    feedback: [none, immediate]
    font: [serif, sans]
  assignment: balanced        # balanced | random | round_robin
  balance_on: assigned        # assigned | completed
  max_per_cell: 40            # optional

Two factors with two levels each give four cells. A participant is assigned a cell the first time the server sees them, and keeps it after a reload or a server restart.

Key	Default	Meaning
`factors`	required	Factor name to a list of at least two levels
`assignment`	`balanced`	`balanced` picks the cell with the fewest participants, breaking ties at random per participant. `random` draws a cell, seeded per participant. `round_robin` cycles through the cells in arrival order
`balance_on`	`assigned`	`completed` counts only participants who reached the end without being screened out, so a dropout is replaced by the next arrival
`max_per_cell`	none	A full cell is skipped. When every cell is full, the least-filled one is used and a warning is logged
`cells`	full factorial	An explicit list of cells, a subset of the factorial, each with an optional `weight`

Weighted cells give one condition a larger share of participants:

yaml

conditions:
  factors:
    prime: [none, positive, negative]
  cells:
    - {prime: none, weight: 2}
    - {prime: positive}
    - {prime: negative}

Quote level names that YAML reads as booleans ("yes", "no", "on", "off"). The validator and the server both report an unquoted one as an error.

Testing a condition in rules

Rules address a factor as condition.<factor> in page show_when, page exit_when and question display_logic. The value compared with a factor must be one of its levels, because a typo would otherwise hide the page or question from everyone, and the validator reports one that is not.

yaml

- name: feedback_rating
  annotation_type: likert
  description: How useful was the feedback?
  size: 5
  display_logic:
    show_when:
      - {schema: condition.feedback, operator: equals, value: immediate}

A question hidden by its condition is not required, either on the page or on the server.

Different items for each cell

To give each cell its own items, name batch groups after the levels of a factor. This is how a Latin-square list design assigns one list per participant:

yaml

conditions:
  factors:
    list: [list_a, list_b]
assignment_strategy: batch
batch_assignment:
  auto_assign_annotators: true
  group_by_condition: list
  groups:
    - {name: list_a, data_file: data/list_a.jsonl}
    - {name: list_b, data_file: data/list_b.jsonl}

Item IDs must be unique across all data files, including the files of different groups. The server will not start when two lists share an ID, so a stimulus that appears in both lists needs a distinct ID in each, such as h1_a and h1_b.

A jsPsych timeline receives the participant's conditions as potato.conditions, and its trial data gets one condition_<factor> column per factor. See Runtime Phases.

Wording per condition

{condition.<factor>} anywhere on a page, in item text, question wording or option labels, is replaced by the participant's level of that factor. An endorsement experiment or a factorial vignette varies its text this way:

yaml

conditions:
  factors:
    endorser: ["The city council", "A neighbourhood association"]

json

{"id": "p1", "text": "{condition.endorser} proposes turning the car park into a park."}

The replacement is HTML-escaped. A placeholder for a factor the participant has no level of stays on the page as written, so a typo shows up in testing. An element with data-condition-only="factor=level" is removed for everyone outside that level, which is how a list experiment shows the sensitive statement to the treatment group only.

Exports write the item text as it is in the data file, with the placeholder. Span offsets count the characters of the filled text, as the participant saw it, so a span after a placeholder does not line up with the exported text. Either shift it by the difference in length between the placeholder and the participant's level (their conditions are in the export), or keep placeholders out of text that is annotated with spans.

Skipping pages with `show_when`

yaml

phases:
  order: [screener, smoking_followup, general, annotation]
  screener:
    type: prestudy
    file: surveys/screener.json
  smoking_followup:
    type: prestudy
    file: surveys/smoking.json
    show_when:
      - {schema: smoker, operator: equals, value: "Yes"}
  general:
    type: prestudy
    file: surveys/general.json

The grammar is the one question-level conditional questions use: a list of conditions with schema, operator and value, combined with logic: all (the default) or logic: any, with the same 19 operators and nested all, any and not groups. A rule can test any question asked on an earlier page, in any phase, and any condition.<factor>.

The rule is checked when the participant leaves the page before it. A hidden page is passed over in both directions: Next skips it, and Previous does not land on it. Each skip is recorded.

The server refuses to start with a rule that names a question no page asks, or one asked on the same page or a later one, since that question could only ever be unanswered. The annotation phase itself cannot be skipped.

Screening out with `exit_when`

yaml

phases:
  screener:
    type: prestudy
    file: surveys/screener.json
    exit_when:
      show_when:
        - {schema: age, operator: lt, value: 18}
      message: This study is for adults only. Thank you for your time.

The rule is checked when the participant submits the page, so it can test questions on that page. When it holds, the participant goes to the end of the study:

Without a crowdsourcing provider, they see message.
With one (Prolific, SONA, CloudResearch Connect and the others), they get the platform's screen-out outcome and never the completion code.

exit_when works on consent, prestudy, instructions and poststudy pages. A screened-out participant stays at the end of the study. Previous does not take them back, even with allow_phase_back_navigation, and the server stores nothing more they send. Answers they gave before the screen-out remain in user_state.json, and the screen-out is written to participants.csv, not annotations.csv.

Counterbalanced page order

yaml

phases:
  order: [intro, block_a, block_b, block_c, annotation]
  # ...block definitions...
  counterbalance:
    - pages: [block_a, block_b, block_c]
      method: latin_square        # or random

The pages keep the slots they hold in order, and each participant gets their own arrangement of them.

random draws a permutation seeded on the participant, so it is the same after a reload or a restart.
latin_square takes rows of a balanced (Williams) Latin square in arrival order. With conditions configured, arrival order is counted within each cell, so every condition works through the whole square and page order is balanced inside each condition instead of being confounded with it. For an odd number of pages the square has twice as many rows. The counters are rebuilt from saved states when the server restarts.

All pages in a group must have the same phase type. Pages of different types are never next to each other in a study, so they could not trade places.

Question order on a survey page

yaml

phases:
  big_five:
    type: poststudy
    instrument: tipi
    randomize_questions:
      pin_last: [comments]     # or: randomize_questions: true

Each participant sees the page's questions in their own order, seeded on the participant and the page and recorded the first time the page is shown. pin_first and pin_last keep named questions at either end. Pages laid out with layout.groups keep their order. The order of options within a question is a separate setting, randomize_order on the scheme.

The flow record

user_state.json holds a flow_record for each participant:

Key	Content
`conditions`, `condition_cell`, `condition_ordinal`	The assigned cell and the study-wide arrival number
`cell_ordinal`	Arrival number within the cell, which picks the Latin-square row when conditions are configured
`latin_ordinal`	Arrival number for Latin-square rows when no conditions are configured
`page_orders`	Each counterbalanced group's order
`question_orders`	Each randomized page's question order
`skipped_pages`	Pages passed over by `show_when`
`screened_out`	The page and time of a screen-out
`runtimes`	Kind, status, attempts, page loads, trial count and run ID of each runtime page
`runtime_answers`	What runtime pages derived from their runs, such as a headphone check's `<page>_passed`

The CSV and TSV exports write one row per participant to participants.csv (or .tsv). With a feedback factor, its header is:

text

user_id,condition_feedback,condition_ordinal,cell_ordinal,latin_ordinal,final_phase,screened_out,screened_out_page,skipped_pages,page_orders,question_orders,runtime_runs,runtime_answers

skipped_pages is separated by |, and the last four columns hold JSON. The MySQL storage backend does not store the flow record, and a server using it logs a warning when these features are configured.

Example

examples/advanced/jspsych-stroop/ in the Potato repository uses conditions, exit_when, a jsPsych page and randomize_questions. Start it from the repository root:

bash

python potato/flask_server.py start examples/advanced/jspsych-stroop/config.yaml -p 8000

Experiment Flow: Conditions, Skips, Screen-outs and Counterbalancing

Conditions

Testing a condition in rules

Different items for each cell

Wording per condition

Skipping pages with show_when

Screening out with exit_when

Counterbalanced page order

Question order on a survey page

The flow record

Example

Further Reading

Skipping pages with `show_when`

Screening out with `exit_when`