# Experiment Flow: Conditions, Skips, Screen-outs and Counterbalancing Source: https://www.potatoannotator.com/docs/experiments/experiment-flow *New in v2.12.0* Potato 2.12 adds five settings for the design side of an experiment. Each one is a few lines of YAML, and everything a participant was assigned or shown is recorded, so the CSV and TSV exports can write it to `participants.csv`. | Setting | What it does | |---|---| | `conditions` | Assigns each participant to one cell of a between-subjects design | | `show_when` on a phase page | Skips the page unless a rule holds | | `exit_when` on a phase page | Ends the study for the participant, as a screener does | | `phases.counterbalance` | Orders a group of pages per participant, at random or by Latin square | | `randomize_questions` on a survey page | Orders that page's questions per participant | The pages these settings act on are the phase pages described in [Multi-Phase Workflows](/docs/features/surveyflow): consent, prestudy, instructions, training, annotation and poststudy, plus the [runtime pages](/docs/experiments/runtime-phases) that run a jsPsych timeline, a headphone check or an external task. ## Conditions ```yaml conditions: factors: feedback: [none, immediate] font: [serif, sans] assignment: balanced # balanced | random | round_robin balance_on: assigned # assigned | completed max_per_cell: 40 # optional ``` Two factors with two levels each give four cells. A participant is assigned a cell the first time the server sees them, and keeps it after a reload or a server restart. | Key | Default | Meaning | |---|---|---| | `factors` | required | Factor name to a list of at least two levels | | `assignment` | `balanced` | `balanced` picks the cell with the fewest participants, breaking ties at random per participant. `random` draws a cell, seeded per participant. `round_robin` cycles through the cells in arrival order | | `balance_on` | `assigned` | `completed` counts only participants who reached the end without being screened out, so a dropout is replaced by the next arrival | | `max_per_cell` | none | A full cell is skipped. When every cell is full, the least-filled one is used and a warning is logged | | `cells` | full factorial | An explicit list of cells, a subset of the factorial, each with an optional `weight` | Weighted cells give one condition a larger share of participants: ```yaml conditions: factors: prime: [none, positive, negative] cells: - {prime: none, weight: 2} - {prime: positive} - {prime: negative} ``` Quote level names that YAML reads as booleans (`"yes"`, `"no"`, `"on"`, `"off"`). The validator and the server both report an unquoted one as an error. ### Testing a condition in rules Rules address a factor as `condition.` in page `show_when`, page `exit_when` and question `display_logic`. The value compared with a factor must be one of its levels, because a typo would otherwise hide the page or question from everyone, and the validator reports one that is not. ```yaml - name: feedback_rating annotation_type: likert description: How useful was the feedback? size: 5 display_logic: show_when: - {schema: condition.feedback, operator: equals, value: immediate} ``` A question hidden by its condition is not required, either on the page or on the server. ### Different items for each cell To give each cell its own items, name batch groups after the levels of a factor. This is how a Latin-square list design assigns one list per participant: ```yaml conditions: factors: list: [list_a, list_b] assignment_strategy: batch batch_assignment: auto_assign_annotators: true group_by_condition: list groups: - {name: list_a, data_file: data/list_a.jsonl} - {name: list_b, data_file: data/list_b.jsonl} ``` Item IDs must be unique across all data files, including the files of different groups. The server will not start when two lists share an ID, so a stimulus that appears in both lists needs a distinct ID in each, such as `h1_a` and `h1_b`. A jsPsych timeline receives the participant's conditions as `potato.conditions`, and its trial data gets one `condition_` column per factor. See [Runtime Phases](/docs/experiments/runtime-phases). ## Wording per condition `{condition.}` anywhere on a page, in item text, question wording or option labels, is replaced by the participant's level of that factor. An endorsement experiment or a factorial vignette varies its text this way: ```yaml conditions: factors: endorser: ["The city council", "A neighbourhood association"] ``` ```json {"id": "p1", "text": "{condition.endorser} proposes turning the car park into a park."} ``` The replacement is HTML-escaped. A placeholder for a factor the participant has no level of stays on the page as written, so a typo shows up in testing. An element with `data-condition-only="factor=level"` is removed for everyone outside that level, which is how a [list experiment](/docs/annotation-types/sensitive-questions) shows the sensitive statement to the treatment group only. Exports write the item text as it is in the data file, with the placeholder. Span offsets count the characters of the filled text, as the participant saw it, so a span after a placeholder does not line up with the exported text. Either shift it by the difference in length between the placeholder and the participant's level (their conditions are in the export), or keep placeholders out of text that is annotated with spans. ## Skipping pages with `show_when` ```yaml phases: order: [screener, smoking_followup, general, annotation] screener: type: prestudy file: surveys/screener.json smoking_followup: type: prestudy file: surveys/smoking.json show_when: - {schema: smoker, operator: equals, value: "Yes"} general: type: prestudy file: surveys/general.json ``` The grammar is the one question-level [conditional questions](/docs/features/surveyflow#conditional-questions) use: a list of conditions with `schema`, `operator` and `value`, combined with `logic: all` (the default) or `logic: any`, with the same 19 operators and nested `all`, `any` and `not` groups. A rule can test any question asked on an earlier page, in any phase, and any `condition.`. The rule is checked when the participant leaves the page before it. A hidden page is passed over in both directions: Next skips it, and Previous does not land on it. Each skip is recorded. The server refuses to start with a rule that names a question no page asks, or one asked on the same page or a later one, since that question could only ever be unanswered. The annotation phase itself cannot be skipped. ## Screening out with `exit_when` ```yaml phases: screener: type: prestudy file: surveys/screener.json exit_when: show_when: - {schema: age, operator: lt, value: 18} message: This study is for adults only. Thank you for your time. ``` The rule is checked when the participant submits the page, so it can test questions on that page. When it holds, the participant goes to the end of the study: - Without a crowdsourcing provider, they see `message`. - With one (Prolific, SONA, CloudResearch Connect and the others), they get the platform's screen-out outcome and never the completion code. `exit_when` works on consent, prestudy, instructions and poststudy pages. A screened-out participant stays at the end of the study. Previous does not take them back, even with `allow_phase_back_navigation`, and the server stores nothing more they send. Answers they gave before the screen-out remain in `user_state.json`, and the screen-out is written to `participants.csv`, not `annotations.csv`. ## Counterbalanced page order ```yaml phases: order: [intro, block_a, block_b, block_c, annotation] # ...block definitions... counterbalance: - pages: [block_a, block_b, block_c] method: latin_square # or random ``` The pages keep the slots they hold in `order`, and each participant gets their own arrangement of them. - **`random`** draws a permutation seeded on the participant, so it is the same after a reload or a restart. - **`latin_square`** takes rows of a balanced (Williams) [Latin square](https://en.wikipedia.org/wiki/Latin_square) in arrival order. With `conditions` configured, arrival order is counted within each cell, so every condition works through the whole square and page order is balanced inside each condition instead of being confounded with it. For an odd number of pages the square has twice as many rows. The counters are rebuilt from saved states when the server restarts. All pages in a group must have the same phase type. Pages of different types are never next to each other in a study, so they could not trade places. ## Question order on a survey page ```yaml phases: big_five: type: poststudy instrument: tipi randomize_questions: pin_last: [comments] # or: randomize_questions: true ``` Each participant sees the page's questions in their own order, seeded on the participant and the page and recorded the first time the page is shown. `pin_first` and `pin_last` keep named questions at either end. Pages laid out with `layout.groups` keep their order. The order of options within a question is a separate setting, `randomize_order` on the scheme. ## The flow record `user_state.json` holds a `flow_record` for each participant: | Key | Content | |---|---| | `conditions`, `condition_cell`, `condition_ordinal` | The assigned cell and the study-wide arrival number | | `cell_ordinal` | Arrival number within the cell, which picks the Latin-square row when conditions are configured | | `latin_ordinal` | Arrival number for Latin-square rows when no conditions are configured | | `page_orders` | Each counterbalanced group's order | | `question_orders` | Each randomized page's question order | | `skipped_pages` | Pages passed over by `show_when` | | `screened_out` | The page and time of a screen-out | | `runtimes` | Kind, status, attempts, page loads, trial count and run ID of each runtime page | | `runtime_answers` | What runtime pages derived from their runs, such as a headphone check's `_passed` | The CSV and TSV exports write one row per participant to `participants.csv` (or `.tsv`). With a `feedback` factor, its header is: ```text user_id,condition_feedback,condition_ordinal,cell_ordinal,latin_ordinal,final_phase,screened_out,screened_out_page,skipped_pages,page_orders,question_orders,runtime_runs,runtime_answers ``` `skipped_pages` is separated by `|`, and the last four columns hold JSON. The MySQL storage backend does not store the flow record, and a server using it logs a warning when these features are configured. ## Example `examples/advanced/jspsych-stroop/` in the Potato repository uses `conditions`, `exit_when`, a jsPsych page and `randomize_questions`. Start it from the repository root: ```bash python potato/flask_server.py start examples/advanced/jspsych-stroop/config.yaml -p 8000 ``` ## Further Reading - [Runtime Phases](/docs/experiments/runtime-phases) - jsPsych timelines, headphone checks and external tasks as pages of a study - [Timed Trials](/docs/experiments/timed-trials) - Fixation, stimulus duration, deadlines and reaction times on annotation items - [Multi-Phase Workflows](/docs/features/surveyflow) - Consent, screeners, instructions, training and surveys - [Coming from jsPsych](/docs/guides/coming-from-jspsych) - jsPsych concepts mapped to Potato config keys - [Online Behavioral Experiments](/docs/guides/online-behavioral-experiments) - Running a psychology or linguistics study in Potato from start to finish For implementation details, see the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/workflow/experiment_flow.md).