Experience Sampling, Follow-Up Waves, and Delphi Rounds

New in v2.13.0

With a schedule: block, Potato opens short survey sessions after a participant's first visit, and a notifications: block sends a message by email, SMS, or Prolific message when each session opens. Survey researchers use follow-up waves for longitudinal panels, psychologists and health researchers use experience sampling for diary and mood studies, and expert panels use Delphi rounds to reach consensus on criteria or guidelines. For every session, Potato records the answers, the time the prompt went out, the time the participant first touched the form, and whether the session was answered, expired, or missed.

The experience sampling method (ESM) asks participants the same short questions several times a day about the moment of each prompt. Health researchers call the same design ecological momentary assessment (EMA). In the Delphi method, a panel of experts rates the same items over several rounds and sees the panel's anonymous answers from the previous round before rating again.

• Time zones. Each participant picks a time zone at enrollment, and the enrollment page fills in the browser's zone. Clock times such as "09:00" are placed in the participant's zone on each local date.
• Channels. Messages go out by email through any SMTP server, by SMS through Twilio, or to the participant's Prolific inbox. Credentials are read from environment variables only.
• Contact details. Potato asks for an email address or phone number only after a yes to a consent question, and keeps addresses outside the output directory and out of every export.

Enrollment and the session plan

Each participant goes through these steps once:

1. The participant completes the regular flow of consent, instructions, annotation, and surveys.
2. At the end of the flow, an enrollment page asks for a time zone, and asks for an email address or mobile number only after a yes to the consent question. Participants who were screened out, failed training, or were blocked by attention checks are not enrolled.
3. Potato computes every session and every reminder for the participant at that moment and stores the plan.
4. A background loop sends each message at the message's due time, with a resume link that signs the participant in and opens the session.
5. A participant who returns with the study link goes to the open session, or to a status page that shows when the next session opens in the participant's own time zone.

The plan and the time zone are fixed at enrollment. Editing waves later changes the plan only for participants who enroll afterward. A participant who declines contact is still enrolled, gets no messages, and finds open sessions by returning to the study link.

Configuration

The config below runs a seven-day experience-sampling study with five prompts a day at random times between 09:00 and 21:00. The regular flow is a consent page and one intake item, and finishing the intake item enrolls the participant:

annotation_task_name: "Daily mood study"
task_dir: .
output_annotation_dir: annotation_output
data_files: [data/intake.jsonl]
item_properties: {id_key: id, text_key: text}
user_config:
  allow_all_users: true
 
phases:
  order: [consent, annotation]
  consent:
    type: consent
    file: surveys/consent.json
  annotation:
    type: annotation
 
annotation_schemes:            # the intake questions, answered once
  - annotation_type: likert
    name: usual_mood
    description: "In general, how would you describe your mood?"
    size: 7
    min_label: "Very low"
    max_label: "Very good"
 
schedule:
  timezone: "America/Detroit"  # used when the browser reports no time zone
  seed: "daily-mood-2026"
  experience_sampling:
    title: "How are you right now?"
    days: 7
    start_day: 1               # first prompts on the day after enrollment
    daily_window: "09:00-21:00"
    signal: {per_day: 5, min_gap: 90m}
    expires_after: 30m
    questions:
      - {name: mood, annotation_type: likert, size: 7, required: true,
         description: "How do you feel right now?",
         min_label: "Very bad", max_label: "Very good"}
      - {name: company, annotation_type: radio, required: true,
         description: "Who are you with?",
         labels: ["Alone", "Family", "Friends", "Colleagues", "Strangers"]}
    reminders:
      - at: open
      - after_open: 15m
 
notifications:
  dry_run: true                # pilot: render and log every message, send none
  base_url: "https://mood-study.example.org"
  channels:
    - type: log
    - type: twilio
      account_sid_env: TWILIO_ACCOUNT_SID
      auth_token_env: TWILIO_AUTH_TOKEN
      from_number: "+12025550143"
  contact:
    consent: {question: contact_consent, value: "Yes"}

The two reminders send the prompt when the prompt opens and a second message 15 minutes later to participants who have not answered. With dry_run: true, Potato renders and addresses every Twilio message and lists each message on the admin page without delivering any. Set dry_run: false once the pilot is done. Random prompt times are drawn from seed, the participant ID, and the date. Each participant gets an independent draw, and the draw can be recomputed later.

The consent page in surveys/consent.json asks the question that notifications.contact.consent names:

[
  {"name": "study_consent", "annotation_type": "radio",
   "description": "I agree to answer short surveys five times a day for seven days.",
   "labels": ["I agree", "I do not agree"],
   "label_requirement": {"required_label": ["I agree"]}},
  {"name": "contact_consent", "annotation_type": "radio",
   "description": "May we text you when each survey opens? Your number is kept apart from your answers and deleted when the study ends.",
   "labels": ["Yes", "No"],
   "label_requirement": {"required": true}}
]

At startup, Potato checks that contact_consent is asked on a phase page and that Yes is one of the question's options. Quote "Yes" in the YAML because YAML 1.1 reads a bare Yes as a boolean and a true or false consent value fails validation. The value may also be a list of answers that allow contact.

Experience sampling options

A study uses exactly one of signal (signal-contingent, at random times) or interval (interval-contingent, at fixed times). The interval block takes exactly one of every or times. Durations are written as 30m, 48h, 7d, or 2w, or as a number of seconds.

A plan that cannot fit in the daily window fails validation. Five prompts 90 minutes apart need four gaps of 90 minutes plus the 30 minutes the last prompt stays open, or 6 hours 30 minutes of the window. Nine prompts at the same spacing need 12 hours 30 minutes and fail validation in a 12-hour window. An interval.every longer than the window and fixed times closer together than expires_after fail the same way.

For each day, Potato places the daily window on that local date in the participant's time zone. The window therefore runs from 09:00 to 21:00 local time on a day when the clocks change. Random times are drawn uniformly over every arrangement that keeps the prompts at least min_gap apart and closes the last prompt inside the window. The status page shows the day's window and never the time of the next random prompt. Prompts whose time has passed at enrollment are left out.

Follow-up waves

A longitudinal panel uses waves. The partial block below replaces schedule in the config above and adds a one-week and a one-month follow-up:

schedule:
  timezone: "America/Detroit"
  waves:
    - name: week1
      title: "One-week follow-up"
      offset: 7d
      at: "09:00"              # local clock time on day 7
      window: 48h
      questions:
        - {name: changed_view, annotation_type: radio, required: true,
           description: "Since last week, has your view changed?",
           labels: ["No", "A little", "A lot"]}
      reminders:
        - before_open: 12h
        - at: open
        - before_close: 12h
      completion_code: "FOLLOWUP-DONE"
    - name: month1
      offset: 30d
      at: "09:00"
      window: 72h
      questions_file: surveys/followup.json

With at, each wave opens at 09:00 local time on the given day even when the clocks change in between, and offset must then be a whole number of days. Without at, offset: 7d means seven days of elapsed time, and a daylight-saving change moves the local opening time by an hour. Quote clock times such as "09:00". YAML 1.1 reads an unquoted 10:30 as the number 630, and Potato reads 630 as minutes after midnight.

Delphi rounds

A Delphi study uses delphi. The partial block below replaces schedule in the experience-sampling config, and data/criteria.json holds a list of {"id", "text"} items:

schedule:
  timezone: "UTC"
  delphi:
    title: "Guideline criteria"
    items_file: data/criteria.json
    scale: {min: 1, max: 9, min_label: "Not important", max_label: "Essential"}
    comments: true
    max_rounds: 3
    round_length: 72h
    gap: 12h
    start: "2026-11-16T09:00:00-05:00"
    min_panel: 3
    stop:
      iqr_at_most: 1
      changed_share_below: 0.15
    reminders:
      - at: open
      - before_close: 24h

Rounds are panel-wide. Round 1 opens at start, or at the first server start with the config when start is not set, and Potato keeps that time across restarts. Each later round opens gap after the previous round closes. A panelist who enrolls during round 2 joins round 2 and sees the round-1 feedback without a rating of the panelist's own. A panelist who stops messages still gets each round, without reminders.

From round 2, each item shows the following about the previous round:

• The median, and the middle half of ratings (Q1 to Q3) as a shaded band on the scale, with the panelist's own rating marked.
• The number of ratings.
• The other panelists' comments without names, in an order fixed per panelist. A panelist's own comment is not repeated back.

An item rated by fewer than min_panel panelists shows no summary and no comments. With the threshold, a small panel's ratings stay untraceable to one person. The current round's rating is never pre-filled.

When a round closes at the end of the round's window, or when an admin closes the round early on /admin/schedule, Potato computes each item's n, median, mean, Q1, Q3, and interquartile range (IQR). Quartiles use linear interpolation between order statistics, the method R calls type 7 and NumPy's default. From round 2, Potato also computes median_change and changed_share. The median_change value is the absolute change in the median, and changed_share is the share of panelists who rated the item in both rounds and changed their rating. An item is settled when the item meets at least one configured criterion:

The study stops after the first round in which every item is settled, or after max_rounds. Without a stop block, the study runs all max_rounds rounds. The stop reason appears on the panelists' status page and on the admin page.

Session questions

Wave and prompt questions are written like annotation schemes, inline under questions or in a JSON or YAML questions_file read relative to the config file. Each question takes name, annotation_type, description, and either required: true or label_requirement: {required: true}. The six question types take these keys:

A key outside the type's list fails validation. Unlike the slider annotation scheme, the session slider takes min_label and max_label. A slider counts as answered only after the participant moves the slider, and the server refuses answers outside a question's options or range. Potato stores session answers per session in <output_annotation_dir>/scheduling/schedule.sqlite and not in user_state.json. In the config above, one participant can answer each prompt question 35 times, once per prompt.

Reminders

Each wave, the experience-sampling block, and the Delphi block take a reminders list built from four rules:

Potato skips a reminder when the session is already answered or closed, and skips a before_open reminder once the session has opened. Reminders that fell due before the participant enrolled are not sent. An after_open or before_close offset must be shorter than the window. Experience sampling does not accept before_open, because the message would announce a random prompt in advance.

Notifications

The partial block below replaces notifications in the config above and sends messages by email and to Prolific inboxes, with a custom text for prompts:

notifications:
  base_url: "https://mood-study.example.org"
  channels:
    - type: log
    - type: smtp
      host: smtp.example.org
      port: 587
      security: starttls        # starttls, ssl, or none
      username_env: POTATO_SMTP_USER
      password_env: POTATO_SMTP_PASSWORD
      from: "Mood study <mood-study@example.org>"
    - type: prolific
      token_env: PROLIFIC_API_TOKEN
  templates:
    prompt:
      subject: "{study}: a short survey is open"
      body: "Please answer within {time_left}: {link}\n\nStop messages: {stop_link}"
  contact:
    consent: {question: contact_consent, value: "Yes"}
    delete_after_study: true

Each channel type may appear once:

The Prolific channel reaches only participants who arrived from Prolific. Potato reads each variable that a *_env key names when a message is sent. Potato stops at startup when the YAML holds a credential under a key such as password, token, or account_sid. A missing variable is logged at startup, and each message on that channel then fails with the variable's name. Set the variables in the server's environment before starting Potato:

export POTATO_SMTP_USER="mood-study@example.org"
export POTATO_SMTP_PASSWORD="<your SMTP password>"
export PROLIFIC_API_TOKEN="<your Prolific API token>"

Template kinds are before_open, open, reminder (for after_open and before_close), prompt (the opening message of an experience-sampling prompt), and delphi_round. Each kind takes a subject and a body. The placeholders are {study}, {session}, {opens}, {closes}, {time_left}, {link}, and {stop_link}. Potato shows {opens} and {closes} in the participant's time zone, and stops at startup when a template names an unknown placeholder. SMS uses the body only, up to Twilio's 1,600-character limit, and a Prolific message joins the subject and the body.

Potato claims each message in the database before sending. Potato therefore sends no message twice, even after a restart or from a second server process. A message interrupted mid-send is marked failed and not resent. After a transient error, Potato tries again 5 and then 10 minutes later while the session is still open, for at most three attempts in all. The transient errors are an SMTP connection error and a Twilio or Prolific 429 or 5xx response. Other errors fail at once.

Schedule-wide settings

The background loop sends due messages, opens and closes Delphi rounds, and deletes finished participants' contact details. With scheduler: false, nothing is sent until an admin presses Send due messages now on /admin/schedule or posts to /admin/api/schedule/tick. A notifications block without a schedule block fails validation.

Resume links and contact details

A resume link (/s/<token>) signs the participant in, opens one session, and expires when the session closes. Links are signed with HMAC-SHA256 under POTATO_SCHEDULE_LINK_KEY, or under a random key generated once in <task_dir>/.potato/scheduling/link.key when the variable is not set. Set the variable when a server may restart on a fresh disk. When a participant stops messages, Potato invalidates every link sent to the participant before.

Email addresses and phone numbers are kept in <task_dir>/.potato/scheduling/contacts.sqlite with file mode 0600, outside the annotation output directory. No exporter, data archive, or deploy bundle reads the contact store, and the notification log records the channel and never the address. A host that wipes the disk at each restart loses the addresses, and participants on that host get no further messages.

Participants can stop messages from the status page or from the stop link in every message. The stop link opens a confirmation page, and only the page's button deletes the details. A mail scanner that opens the link therefore unsubscribes nobody. With delete_after_study: true, details are deleted once the participant has no open or upcoming sessions, and a Delphi study keeps the details until the panel finishes.

Missed and expired sessions

A session closes at the end of the session's window, and the server refuses answers submitted after the close. Potato records a closed session in one of three states:

• answered means the participant submitted the session before the close.
• expired means the participant opened the session or started answering but did not submit before the close.
• missed means the participant never opened the session while the session was open.

Potato does not reschedule a closed session, and pending reminders for the session are skipped. A participant who opens an old link sees when the session closed and when the next session opens, unless the next session is a random prompt. Compliance is answered sessions divided by closed sessions, per participant and per session kind.

The admin page

/admin/schedule needs the admin API key or an RBAC admin role. The page shows each participant's time zone, consent, compliance, median latency, and next session, and lists every message by status, with the full text for log and dry_run messages. Delphi rounds appear with item statistics. Buttons on the page match the POST endpoints below:

Export

Session answers are not item annotations, and the main export leaves session answers out. The schedule export writes sidecar tables instead, and every table has a participant column with the same ID as user_id in the main export:

python -m potato.export --config config.yaml --format schedule --output out/

Delphi ratings go to delphi_ratings.csv, and round sessions appear in schedule_sessions.csv with a status and a latency. The columns of schedule_sessions.csv are:

Example

python potato/flask_server.py start examples/advanced/experience-sampling/config.yaml -p 8000

The example runs the sampling plan above with the log channel and a dry-run Twilio channel. The admin page at /admin/schedule lists each prompt's resume link. The folders examples/advanced/two-wave-panel/ and examples/advanced/delphi-study/ hold a one-week follow-up panel and a three-round Delphi study.

Further reading

• Multi-Phase Workflows - Consent, instructions, and survey pages in the regular flow
• Survey Instruments - Validated questionnaires for prestudy and poststudy pages
• Running a Crowdsourced Study on Prolific and SONA - Recruiting participants and running later waves on Prolific
• Psychology - Potato for online behavioral experiments
• Survey Research - Potato for survey experiments on your own server

For implementation details, see the source documentation.