# Experience Sampling, Follow-Up Waves, and Delphi Rounds

Source: https://www.potatoannotator.com/docs/features/scheduled-sessions

*New in v2.13.0*

With a `schedule:` block, Potato opens short survey sessions after a participant's first visit, and a `notifications:` block sends a message by email, SMS, or Prolific message when each session opens. Survey researchers use follow-up waves for longitudinal panels, psychologists and health researchers use experience sampling for diary and mood studies, and expert panels use Delphi rounds to reach consensus on criteria or guidelines. For every session, Potato records the answers, the time the prompt went out, the time the participant first touched the form, and whether the session was answered, expired, or missed.

The [experience sampling method](https://en.wikipedia.org/wiki/Experience_sampling_method) (ESM) asks participants the same short questions several times a day about the moment of each prompt. Health researchers call the same design [ecological momentary assessment](https://en.wikipedia.org/wiki/Ecological_momentary_assessment) (EMA). In the [Delphi method](https://en.wikipedia.org/wiki/Delphi_method), a panel of experts rates the same items over several rounds and sees the panel's anonymous answers from the previous round before rating again.

- **Time zones.** Each participant picks a time zone at enrollment, and the enrollment page fills in the browser's zone. Clock times such as `"09:00"` are placed in the participant's zone on each local date.
- **Channels.** Messages go out by email through any SMTP server, by SMS through Twilio, or to the participant's Prolific inbox. Credentials are read from environment variables only.
- **Contact details.** Potato asks for an email address or phone number only after a yes to a consent question, and keeps addresses outside the output directory and out of every export.

## Enrollment and the session plan

Each participant goes through these steps once:

1. The participant completes the regular flow of consent, instructions, annotation, and surveys.
2. At the end of the flow, an enrollment page asks for a time zone, and asks for an email address or mobile number only after a yes to the consent question. Participants who were screened out, failed training, or were blocked by attention checks are not enrolled.
3. Potato computes every session and every reminder for the participant at that moment and stores the plan.
4. A background loop sends each message at the message's due time, with a resume link that signs the participant in and opens the session.
5. A participant who returns with the study link goes to the open session, or to a status page that shows when the next session opens in the participant's own time zone.

The plan and the time zone are fixed at enrollment. Editing `waves` later changes the plan only for participants who enroll afterward. A participant who declines contact is still enrolled, gets no messages, and finds open sessions by returning to the study link.

## Configuration

The config below runs a seven-day experience-sampling study with five prompts a day at random times between 09:00 and 21:00. The regular flow is a consent page and one intake item, and finishing the intake item enrolls the participant:

```yaml
annotation_task_name: "Daily mood study"
task_dir: .
output_annotation_dir: annotation_output
data_files: [data/intake.jsonl]
item_properties: {id_key: id, text_key: text}
user_config:
  allow_all_users: true

phases:
  order: [consent, annotation]
  consent:
    type: consent
    file: surveys/consent.json
  annotation:
    type: annotation

annotation_schemes:            # the intake questions, answered once
  - annotation_type: likert
    name: usual_mood
    description: "In general, how would you describe your mood?"
    size: 7
    min_label: "Very low"
    max_label: "Very good"

schedule:
  timezone: "America/Detroit"  # used when the browser reports no time zone
  seed: "daily-mood-2026"
  experience_sampling:
    title: "How are you right now?"
    days: 7
    start_day: 1               # first prompts on the day after enrollment
    daily_window: "09:00-21:00"
    signal: {per_day: 5, min_gap: 90m}
    expires_after: 30m
    questions:
      - {name: mood, annotation_type: likert, size: 7, required: true,
         description: "How do you feel right now?",
         min_label: "Very bad", max_label: "Very good"}
      - {name: company, annotation_type: radio, required: true,
         description: "Who are you with?",
         labels: ["Alone", "Family", "Friends", "Colleagues", "Strangers"]}
    reminders:
      - at: open
      - after_open: 15m

notifications:
  dry_run: true                # pilot: render and log every message, send none
  base_url: "https://mood-study.example.org"
  channels:
    - type: log
    - type: twilio
      account_sid_env: TWILIO_ACCOUNT_SID
      auth_token_env: TWILIO_AUTH_TOKEN
      from_number: "+12025550143"
  contact:
    consent: {question: contact_consent, value: "Yes"}
```

The two reminders send the prompt when the prompt opens and a second message 15 minutes later to participants who have not answered. With `dry_run: true`, Potato renders and addresses every Twilio message and lists each message on the admin page without delivering any. Set `dry_run: false` once the pilot is done. Random prompt times are drawn from `seed`, the participant ID, and the date. Each participant gets an independent draw, and the draw can be recomputed later.

The consent page in `surveys/consent.json` asks the question that `notifications.contact.consent` names:

```json
[
  {"name": "study_consent", "annotation_type": "radio",
   "description": "I agree to answer short surveys five times a day for seven days.",
   "labels": ["I agree", "I do not agree"],
   "label_requirement": {"required_label": ["I agree"]}},
  {"name": "contact_consent", "annotation_type": "radio",
   "description": "May we text you when each survey opens? Your number is kept apart from your answers and deleted when the study ends.",
   "labels": ["Yes", "No"],
   "label_requirement": {"required": true}}
]
```

At startup, Potato checks that `contact_consent` is asked on a phase page and that `Yes` is one of the question's options. Quote `"Yes"` in the YAML because YAML 1.1 reads a bare Yes as a boolean and a true or false consent value fails validation. The `value` may also be a list of answers that allow contact.

## Experience sampling options

| `experience_sampling` key | Default | Meaning |
|---|---|---|
| `days` | required | Number of sampling days, from 1 to 366 |
| `start_day` | `1` | First sampling day counted from the enrollment day, with `0` for the enrollment day itself |
| `daily_window` | `"09:00-21:00"` | Local window the prompts fall in |
| `signal.per_day` | required with `signal` | Random prompts per day, from 1 to 48 |
| `signal.min_gap` | `60m` | Least time between two prompts, at least `expires_after` |
| `interval.every` | none | A prompt at the window start and every interval after the start |
| `interval.times` | none | Fixed local clock times, such as `["10:00", "14:00", "20:00"]` |
| `expires_after` | `30m` | How long each prompt stays open, at least one minute |
| `questions`, `questions_file` | one required | The prompt's questions (see the session questions section below) |
| `reminders` | `[{at: open}]` | `at: open`, `after_open`, and `before_close`. `before_open` is refused |
| `title` | `"Short survey"` | Shown on the prompt page and in messages |
| `intro` | none | Text above the questions |

A study uses exactly one of `signal` (signal-contingent, at random times) or `interval` (interval-contingent, at fixed times). The `interval` block takes exactly one of `every` or `times`. Durations are written as `30m`, `48h`, `7d`, or `2w`, or as a number of seconds.

A plan that cannot fit in the daily window fails validation. Five prompts 90 minutes apart need four gaps of 90 minutes plus the 30 minutes the last prompt stays open, or 6 hours 30 minutes of the window. Nine prompts at the same spacing need 12 hours 30 minutes and fail validation in a 12-hour window. An `interval.every` longer than the window and fixed times closer together than `expires_after` fail the same way.

For each day, Potato places the daily window on that local date in the participant's time zone. The window therefore runs from 09:00 to 21:00 local time on a day when the clocks change. Random times are drawn uniformly over every arrangement that keeps the prompts at least `min_gap` apart and closes the last prompt inside the window. The status page shows the day's window and never the time of the next random prompt. Prompts whose time has passed at enrollment are left out.

## Follow-up waves

A longitudinal panel uses `waves`. The partial block below replaces `schedule` in the config above and adds a one-week and a one-month follow-up:

```yaml
schedule:
  timezone: "America/Detroit"
  waves:
    - name: week1
      title: "One-week follow-up"
      offset: 7d
      at: "09:00"              # local clock time on day 7
      window: 48h
      questions:
        - {name: changed_view, annotation_type: radio, required: true,
           description: "Since last week, has your view changed?",
           labels: ["No", "A little", "A lot"]}
      reminders:
        - before_open: 12h
        - at: open
        - before_close: 12h
      completion_code: "FOLLOWUP-DONE"
    - name: month1
      offset: 30d
      at: "09:00"
      window: 72h
      questions_file: surveys/followup.json
```

With `at`, each wave opens at 09:00 local time on the given day even when the clocks change in between, and `offset` must then be a whole number of days. Without `at`, `offset: 7d` means seven days of elapsed time, and a daylight-saving change moves the local opening time by an hour. Quote clock times such as `"09:00"`. YAML 1.1 reads an unquoted `10:30` as the number 630, and Potato reads 630 as minutes after midnight.

| `waves` key | Default | Meaning |
|---|---|---|
| `name` | required | Short ID of up to 40 letters, digits, `_`, or `-`, used in the export |
| `title` | the `name` | Shown to participants and in messages |
| `offset` | `0` | Time after enrollment |
| `at` | none | Local clock time on the day `offset` days after enrollment |
| `window` | required | How long the wave stays open, at least one minute |
| `questions`, `questions_file` | one required | The wave's questions |
| `reminders` | `[{at: open}]` | When messages go out |
| `completion_code` | none | A code shown after the participant submits the wave, for example to claim payment in a second Prolific study |
| `intro` | none | Text above the questions |

## Delphi rounds

A Delphi study uses `delphi`. The partial block below replaces `schedule` in the experience-sampling config, and `data/criteria.json` holds a list of `{"id", "text"}` items:

```yaml
schedule:
  timezone: "UTC"
  delphi:
    title: "Guideline criteria"
    items_file: data/criteria.json
    scale: {min: 1, max: 9, min_label: "Not important", max_label: "Essential"}
    comments: true
    max_rounds: 3
    round_length: 72h
    gap: 12h
    start: "2026-11-16T09:00:00-05:00"
    min_panel: 3
    stop:
      iqr_at_most: 1
      changed_share_below: 0.15
    reminders:
      - at: open
      - before_close: 24h
```

Rounds are panel-wide. Round 1 opens at `start`, or at the first server start with the config when `start` is not set, and Potato keeps that time across restarts. Each later round opens `gap` after the previous round closes. A panelist who enrolls during round 2 joins round 2 and sees the round-1 feedback without a rating of the panelist's own. A panelist who stops messages still gets each round, without reminders.

From round 2, each item shows the following about the previous round:

- The median, and the middle half of ratings (Q1 to Q3) as a shaded band on the scale, with the panelist's own rating marked.
- The number of ratings.
- The other panelists' comments without names, in an order fixed per panelist. A panelist's own comment is not repeated back.

An item rated by fewer than `min_panel` panelists shows no summary and no comments. With the threshold, a small panel's ratings stay untraceable to one person. The current round's rating is never pre-filled.

When a round closes at the end of the round's window, or when an admin closes the round early on `/admin/schedule`, Potato computes each item's `n`, median, mean, Q1, Q3, and [interquartile range](https://en.wikipedia.org/wiki/Interquartile_range) (IQR). Quartiles use linear interpolation between order statistics, the method R calls type 7 and NumPy's default. From round 2, Potato also computes `median_change` and `changed_share`. The `median_change` value is the absolute change in the median, and `changed_share` is the share of panelists who rated the item in both rounds and changed their rating. An item is settled when the item meets at least one configured criterion:

| `stop` key | Kind | Settled when |
|---|---|---|
| `iqr_at_most` | consensus | The IQR is at most the value |
| `median_change_at_most` | stability, from round 2 | The median moved by at most the value since the previous round |
| `changed_share_below` | stability, from round 2 | The share of panelists who changed their rating is below the value, from 0 to 1 |

The study stops after the first round in which every item is settled, or after `max_rounds`. Without a `stop` block, the study runs all `max_rounds` rounds. The stop reason appears on the panelists' status page and on the admin page.

| `delphi` key | Default | Meaning |
|---|---|---|
| `items`, `items_file` | one required | `[{id, text}]` inline, or a JSON, JSONL, or YAML file |
| `scale` | `{min: 1, max: 9}` | Whole-number rating scale of at most 21 points, with optional `min_label` and `max_label` |
| `comments` | `true` | An optional comment per item of up to 2,000 characters, shown to the panel without names |
| `max_rounds` | `3` | Upper limit on rounds, from 1 to 10 |
| `round_length` | required | How long each round is open |
| `gap` | `0` | Time between one round closing and the next opening |
| `start` | first server start | When round 1 opens, as ISO 8601 with a UTC offset |
| `min_panel` | `3` | Ratings needed before an item's summary and comments are shown |
| `stop` | none | Stopping criteria |
| `reminders` | `[{at: open}]` | When messages go out |
| `title` | `"Delphi round"` | Shown on the round page, followed by the round number |
| `intro` | none | Text above the items |

## Session questions

Wave and prompt questions are written like annotation schemes, inline under `questions` or in a JSON or YAML `questions_file` read relative to the config file. Each question takes `name`, `annotation_type`, `description`, and either `required: true` or `label_requirement: {required: true}`. The six question types take these keys:

| Type | Keys and defaults |
|---|---|
| `radio` | `labels` (at least two), `horizontal` |
| `multiselect` | `labels` (at least two) |
| `likert` | `size` (2 to 11), `min_label`, `max_label` |
| `slider` | `min_value` (`0`), `max_value` (`100`), `starting_value` (the `min_value`), `step` (`1`), `min_label`, `max_label` |
| `number` | `min_value`, `max_value`, `integer` |
| `text` | `multiline` (`true`), `max_length` (`1000`, at most 4,000) |

A key outside the type's list fails validation. Unlike the `slider` annotation scheme, the session `slider` takes `min_label` and `max_label`. A slider counts as answered only after the participant moves the slider, and the server refuses answers outside a question's options or range. Potato stores session answers per session in `<output_annotation_dir>/scheduling/schedule.sqlite` and not in `user_state.json`. In the config above, one participant can answer each prompt question 35 times, once per prompt.

## Reminders

Each wave, the experience-sampling block, and the Delphi block take a `reminders` list built from four rules:

| Rule | When the message goes out |
|---|---|
| `{at: open}` | When the window opens |
| `{before_open: 12h}` | 12 hours before the window opens |
| `{after_open: 2h}` | 2 hours after the window opens, if the session is not yet answered |
| `{before_close: 12h}` | 12 hours before the window closes, if the session is not yet answered |

Potato skips a reminder when the session is already answered or closed, and skips a `before_open` reminder once the session has opened. Reminders that fell due before the participant enrolled are not sent. An `after_open` or `before_close` offset must be shorter than the window. Experience sampling does not accept `before_open`, because the message would announce a random prompt in advance.

## Notifications

The partial block below replaces `notifications` in the config above and sends messages by email and to Prolific inboxes, with a custom text for prompts:

```yaml
notifications:
  base_url: "https://mood-study.example.org"
  channels:
    - type: log
    - type: smtp
      host: smtp.example.org
      port: 587
      security: starttls        # starttls, ssl, or none
      username_env: POTATO_SMTP_USER
      password_env: POTATO_SMTP_PASSWORD
      from: "Mood study <mood-study@example.org>"
    - type: prolific
      token_env: PROLIFIC_API_TOKEN
  templates:
    prompt:
      subject: "{study}: a short survey is open"
      body: "Please answer within {time_left}: {link}\n\nStop messages: {stop_link}"
  contact:
    consent: {question: contact_consent, value: "Yes"}
    delete_after_study: true
```

Each channel type may appear once:

| Channel | Delivers to | Keys and defaults |
|---|---|---|
| `log` | The server log and the admin page, with the full text and links. Delivers nothing to participants | none |
| `smtp` | The email address given at enrollment | `host` (required), `from` (required), `security` (`starttls`), `port` (`587`, or `465` with `ssl`), `username_env` and `password_env` (both or neither), `timeout` (`20` seconds) |
| `twilio` | The mobile number given at enrollment, by SMS | `account_sid_env` (`TWILIO_ACCOUNT_SID`), `auth_token_env` (`TWILIO_AUTH_TOKEN`), exactly one of `from_number` (an [E.164](https://en.wikipedia.org/wiki/E.164) number) or `messaging_service_sid_env`, `timeout` (`20`) |
| `prolific` | The participant's Prolific inbox, by Prolific participant ID | `token_env` (`PROLIFIC_API_TOKEN`), `study_id` (the participant's `STUDY_ID` when unset), `timeout` (`30`) |

The Prolific channel reaches only participants who arrived from Prolific. Potato reads each variable that a `*_env` key names when a message is sent. Potato stops at startup when the YAML holds a credential under a key such as `password`, `token`, or `account_sid`. A missing variable is logged at startup, and each message on that channel then fails with the variable's name. Set the variables in the server's environment before starting Potato:

```bash
export POTATO_SMTP_USER="mood-study@example.org"
export POTATO_SMTP_PASSWORD="<your SMTP password>"
export PROLIFIC_API_TOKEN="<your Prolific API token>"
```

| `notifications` key | Default | Meaning |
|---|---|---|
| `dry_run` | `false` | Renders and addresses every message on every channel and records each message as `dry_run` without delivering |
| `base_url` | `http://localhost:<port>` | The public address used in links. Required for any channel other than `log` unless `dry_run` is on |
| `channels` | none | The channels above |
| `contact.consent` | required with channels | `{question, value}`, the consent question on a phase page and the answer that allows contact |
| `contact.delete_after_study` | `true` | Deletes a participant's contact details once the participant has no sessions left |
| `templates` | built in | Message text per kind |

Template kinds are `before_open`, `open`, `reminder` (for `after_open` and `before_close`), `prompt` (the opening message of an experience-sampling prompt), and `delphi_round`. Each kind takes a `subject` and a `body`. The placeholders are `{study}`, `{session}`, `{opens}`, `{closes}`, `{time_left}`, `{link}`, and `{stop_link}`. Potato shows `{opens}` and `{closes}` in the participant's time zone, and stops at startup when a template names an unknown placeholder. SMS uses the body only, up to Twilio's 1,600-character limit, and a Prolific message joins the subject and the body.

Potato claims each message in the database before sending. Potato therefore sends no message twice, even after a restart or from a second server process. A message interrupted mid-send is marked failed and not resent. After a transient error, Potato tries again 5 and then 10 minutes later while the session is still open, for at most three attempts in all. The transient errors are an SMTP connection error and a Twilio or Prolific 429 or 5xx response. Other errors fail at once.

## Schedule-wide settings

| `schedule` key | Default | Meaning |
|---|---|---|
| `timezone` | `UTC` | The [IANA time zone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) preselected on the enrollment page when the browser reports none |
| `seed` | empty | Text mixed into random prompt times and the order of Delphi comments |
| `poll_seconds` | `30` | How often the background loop runs, from 1 to 3,600 seconds |
| `scheduler` | `true` | Runs the background loop with the server |
| `waves`, `experience_sampling`, `delphi` | none | At least one is required |

The background loop sends due messages, opens and closes Delphi rounds, and deletes finished participants' contact details. With `scheduler: false`, nothing is sent until an admin presses *Send due messages now* on `/admin/schedule` or posts to `/admin/api/schedule/tick`. A `notifications` block without a `schedule` block fails validation.

## Resume links and contact details

A resume link (`/s/<token>`) signs the participant in, opens one session, and expires when the session closes. Links are signed with HMAC-SHA256 under `POTATO_SCHEDULE_LINK_KEY`, or under a random key generated once in `<task_dir>/.potato/scheduling/link.key` when the variable is not set. Set the variable when a server may restart on a fresh disk. When a participant stops messages, Potato invalidates every link sent to the participant before.

Email addresses and phone numbers are kept in `<task_dir>/.potato/scheduling/contacts.sqlite` with file mode 0600, outside the annotation output directory. No exporter, data archive, or deploy bundle reads the contact store, and the notification log records the channel and never the address. A host that wipes the disk at each restart loses the addresses, and participants on that host get no further messages.

Participants can stop messages from the status page or from the stop link in every message. The stop link opens a confirmation page, and only the page's button deletes the details. A mail scanner that opens the link therefore unsubscribes nobody. With `delete_after_study: true`, details are deleted once the participant has no open or upcoming sessions, and a Delphi study keeps the details until the panel finishes.

## Missed and expired sessions

A session closes at the end of the session's window, and the server refuses answers submitted after the close. Potato records a closed session in one of three states:

- **answered** means the participant submitted the session before the close.
- **expired** means the participant opened the session or started answering but did not submit before the close.
- **missed** means the participant never opened the session while the session was open.

Potato does not reschedule a closed session, and pending reminders for the session are skipped. A participant who opens an old link sees when the session closed and when the next session opens, unless the next session is a random prompt. Compliance is answered sessions divided by closed sessions, per participant and per session kind.

## The admin page

`/admin/schedule` needs the admin API key or an RBAC admin role. The page shows each participant's time zone, consent, compliance, median latency, and next session, and lists every message by status, with the full text for `log` and `dry_run` messages. Delphi rounds appear with item statistics. Buttons on the page match the POST endpoints below:

| Endpoint | Method | Action |
|---|---|---|
| `/admin/api/schedule` | GET | The page's data as JSON, without addresses |
| `/admin/api/schedule/tick` | POST | Sends due messages now |
| `/admin/api/schedule/delphi/close_round` | POST | Closes the open Delphi round |
| `/admin/api/schedule/contact/<participant>/delete` | POST | Deletes one participant's contact details |

## Export

Session answers are not item annotations, and the main export leaves session answers out. The `schedule` export writes sidecar tables instead, and every table has a `participant` column with the same ID as `user_id` in the main export:

```bash
python -m potato.export --config config.yaml --format schedule --output out/
```

| File | Rows |
|---|---|
| `schedule_sessions.csv` | One per participant and session, plus one `enrolment` row per participant for the regular flow |
| `schedule_compliance.csv` | Per participant and session kind, with `sessions`, `closed`, `answered`, `expired`, `missed`, `compliance`, `median_latency_seconds` |
| `schedule_notifications.csv` | Every planned message, with `session_id`, `rule`, `template`, `channel`, `due_at_utc`, `status`, `attempts`, `sent_at_utc`, `detail`. No address and no message text |
| `delphi_ratings.csv` | Delphi only, with `participant`, `round`, `item`, `rating`, `comment` |
| `delphi_rounds.csv` | Delphi only, per round and item, with `n`, `median`, `q1`, `q3`, `iqr`, `mean`, `median_change`, `changed_share`, `settled`, `opens_at_utc`, `closed_at_utc`, `round_status`, `stop_reason` |

Delphi ratings go to `delphi_ratings.csv`, and round sessions appear in `schedule_sessions.csv` with a status and a latency. The columns of `schedule_sessions.csv` are:

| Column | Meaning |
|---|---|
| `kind` | `wave`, `prompt`, `round`, or `enrolment` |
| `session_id`, `wave`, `day`, `prompt`, `round`, `title` | Which session the row is. `day` is the 1-based sampling day and `prompt` the prompt's number on that day |
| `timezone` | The participant's time zone |
| `opens_at_utc`, `closes_at_utc`, `opens_at_local`, `closes_at_local` | The window in UTC and in local time |
| `status` | `answered`, `expired`, `missed`, `open`, `started`, or `upcoming` (`enrolled` on the enrollment row) |
| `prompt_at_utc` | When the opening message went out, or the window start when no message was sent |
| `opened_at_utc`, `first_answer_at_utc`, `submitted_at_utc` | When the participant opened the page, first touched the form, and submitted |
| `latency_seconds` | From `prompt_at_utc` to the first answer |
| `response_seconds` | From the first answer to the submission |
| `reminder_channels` | The channels the participant receives messages on, on the enrollment row |
| `answer.<name>` | One column per question, with multiselect answers joined by `;` |

## Example

```bash
python potato/flask_server.py start examples/advanced/experience-sampling/config.yaml -p 8000
```

The example runs the sampling plan above with the `log` channel and a dry-run Twilio channel. The admin page at `/admin/schedule` lists each prompt's resume link. The folders `examples/advanced/two-wave-panel/` and `examples/advanced/delphi-study/` hold a one-week follow-up panel and a three-round Delphi study.

## Further reading

- [Multi-Phase Workflows](/docs/features/surveyflow) - Consent, instructions, and survey pages in the regular flow
- [Survey Instruments](/docs/features/survey-instruments) - Validated questionnaires for prestudy and poststudy pages
- [Running a Crowdsourced Study on Prolific and SONA](/docs/guides/crowdsourcing-prolific-mturk) - Recruiting participants and running later waves on Prolific
- [Psychology](/use-cases/psychology) - Potato for online behavioral experiments
- [Survey Research](/use-cases/survey-research) - Potato for survey experiments on your own server

For implementation details, see the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/workflow/scheduling.md).