Skip to content
Guides8 min read

Running Annotation Studies on Prolific

Run a Potato annotation study on Prolific with URL-direct login, a completion code, attention checks, and API-side submission monitoring.

Potato Team

Point Prolific's study URL at your Potato server with ?PROLIFIC_PID={{%PROLIFIC_PID%}}, set login.type: url_direct and a completion_code that matches the code on the study, and each participant is logged in under their own Prolific ID and sent back to Prolific when they finish.

Prolific is a participant pool built for academic studies, and it is the crowdsourcing platform Potato supports most fully. This guide covers the config Potato needs, the study settings on the Prolific side, and the quality control worth having in place before you spend money.

Why researchers use Prolific for annotation

Prolific screens its participants and pays them through the platform, so the parts of crowdsourcing that usually consume a study budget are handled outside your server. It enforces a minimum hourly rate, filters participants on demographic and language criteria, and records consent, which shortens the IRB paperwork for a labeling study. The trade-off is cost per response against Mechanical Turk, and the MTurk deployment guide covers that platform if you need it.

What you need before you start

  1. A Prolific researcher account.
  2. Potato running on a server with a public HTTPS URL. The server deployment guide covers that step.
  3. A config with your data files, item_properties, and at least one annotation scheme.

The config Potato needs

Four settings turn an ordinary Potato project into one Prolific can drive. login.url_argument names the URL parameter carrying the participant's Prolific ID, completion_code must match a code configured on the Prolific study, and the two redirect keys send participants back to Prolific rather than leaving them on a done page. The assignment keys below it cap how much work any one participant is given, which is what keeps a study's cost predictable.

yaml
annotation_task_name: "Sentiment Annotation Study"
task_description: "Classify the sentiment of short text snippets. This task takes about five minutes."
 
# Participants are logged in from the PROLIFIC_PID URL parameter
login:
  type: url_direct
  url_argument: PROLIFIC_PID
 
# Must match a completion code configured on the Prolific study
completion_code: "C1ABCDEF"
auto_redirect_on_completion: true
auto_redirect_delay: 5000
 
# Keep participants on task
hide_navbar: true
jumping_to_id_disabled: true
 
# Each participant labels 20 items; each item is labeled by 3 participants
assignment_strategy: random
max_annotations_per_user: 20
num_annotators_per_item: 3
 
data_files:
  - data/texts.json
 
item_properties:
  id_key: id
  text_key: text
 
output_annotation_dir: annotation_output/prolific-study/
export_annotation_format: jsonl
 
user_config:
  allow_all_users: true
  users: []
 
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What is the sentiment expressed in this text?"
    labels:
      - name: positive
      - name: neutral
      - name: negative
    sequential_key_binding: true
    label_requirement:
      required: true

Set num_annotators_per_item from what you plan to do with the data. Three is the usual choice when you intend to report agreement or take a majority label, and one is enough once a pilot has shown the task is well defined. The inter-annotator agreement guide covers how to read the result.

Setting up the study on Prolific

In the Prolific study form, set the external study URL to your Potato server with Prolific's placeholders in the query string:

text
https://your-server.com/?PROLIFIC_PID={{%PROLIFIC_PID%}}&STUDY_ID={{%STUDY_ID%}}&SESSION_ID={{%SESSION_ID%}}

Choose "I'll use URL parameters" for the question about how Prolific IDs are recorded, then configure a completion code on the study that matches completion_code in your config. STUDY_ID and SESSION_ID are captured into the participant's session for later tracking.

A participant who clicks the link arrives at Potato with their ID in the URL. Potato creates a passwordless account under that ID, walks the participant through whatever phases you configured, and shows the completion code with a Return to Prolific button linking to https://app.prolific.com/submissions/complete?cc=<code> when they finish. With auto_redirect_on_completion set, the redirect happens on its own after auto_redirect_delay milliseconds.

Write the domain as app.prolific.com. The older app.prolific.co domain no longer serves HTTPS, so a completion URL copied from an older guide sends participants to a dead page after they have done the work.

Most crowdsourced studies need more than the annotation page. Potato's phases block puts consent, screening, instructions, and a post-study survey around the annotation phase, and participants move through them in the order you list:

yaml
phases:
  order: [consent, prestudy, instructions, annotation, poststudy]
 
  consent:
    type: consent
    file: surveyflow/consent.json
 
  prestudy:
    type: prestudy
    file: surveyflow/prescreening.json
 
  instructions:
    type: instructions
    file: surveyflow/instructions.html
 
  poststudy:
    type: poststudy
    file: surveyflow/demographics.json

Add this block to the config above. The annotation phase is the only required one, so a study that needs no consent form can leave every other entry out. Older projects configure the same thing through surveyflow, which Potato still reads but no longer recommends for new projects.

Attention checks

An attention check is an item with a known correct answer, mixed into the queue so an inattentive participant produces a detectable wrong answer. Potato loads them from their own file rather than from your data, so the checks stay out of your dataset:

yaml
attention_checks:
  enabled: true
  items_file: "data/attention_checks.json"
  frequency: 10
  min_response_time: 3.0
  failure_handling:
    warn_threshold: 2
    block_threshold: 5

Add this block to the config above. frequency: 10 inserts one check every ten items, and min_response_time fails a response that arrives faster than anyone could have read the item, whatever it says. The clock runs on the server, from the moment it sent the item to the moment it received the save.

Each check item carries an id, the text the participant sees, and the answer expected for each scheme by name:

json
[
  {
    "id": "attn_001",
    "text": "Please select 'positive' for this item to verify you are reading carefully.",
    "expected_answer": {"sentiment": "positive"}
  }
]

Set block_threshold against the length of your study. A participant who fails five checks in a 20-item task was never annotating; a participant who fails five across 300 items may have been tired for a stretch, and blocking them loses the rest of their work.

Monitoring a running study

Set login.type: prolific and add a prolific block, and Potato polls the Prolific API for submission statuses on top of the URL-direct login it already does:

yaml
login:
  type: prolific
  url_argument: PROLIFIC_PID
 
prolific:
  config_file_path: configs/prolific_config.yaml
 
completion_code: "C1ABCDEF"

Add this block to the config above. The file it points at holds the API token, so keep it out of version control:

yaml
# configs/prolific_config.yaml
token: "your-prolific-api-token"
study_id: "your-study-id"
workload_checker: true
max_concurrent_sessions: 30
workload_checker_period: 300

With the token configured, the admin dashboard gains a Prolific panel that publishes, pauses, and stops the study, approves or rejects submissions, pays bonuses, and previews the cost of adding places. The workload monitor pauses the study when active participants reach max_concurrent_sessions and resumes it when the count drops below 20% of that maximum. It only resumes a study it paused itself, so a study you pause by hand in Prolific stays paused.

Participants who drop out

Prolific reports a submission as RETURNED, TIMED-OUT, or REJECTED when a participant leaves without finishing. Potato releases the items assigned to those participants so other people can pick them up, and keeps the annotations they had already completed. Use instance_reclaim when the reason should change what happens to that completed work:

yaml
instance_reclaim:
  enabled: true
  prolific:
    preserve_completed_annotations: true
    status_policies:
      TIMED-OUT:
        preserve_completed_annotations: true
      REJECTED:
        preserve_completed_annotations: false

Add this block to the config above. The distinction is worth setting deliberately. A timed-out participant usually annotated honestly until something interrupted them, while a rejected one produced work you decided not to pay for, and keeping it in the dataset undercuts the rejection.

What the output contains

Potato writes per-annotator files under output_annotation_dir and, when export_annotation_format is set, a periodic export under <output_annotation_dir>/exports/<format>/. Each exported record carries the instance, the annotator, and the labels by scheme name:

json
{"instance_id": "t_014", "user_id": "5f2c9a1b3d4e5f6a7b8c9d0e", "labels": {"sentiment": {"positive": "positive"}}, "spans": {}, "links": {}}

The user_id is the participant's Prolific ID, because that is the username Potato created them under, which is what lets you match a row back to a Prolific submission for approval or a bonus. Treat that column as identifying data when you publish the dataset.

Running a pilot first

Run 5 to 10 participants before you launch the full study, and read their output rather than only their completion rate. Two failures are expensive to discover afterwards. The first is an instruction that is ambiguous in a way you cannot see because you wrote the task, and the second is a per-item time far from your estimate, which means your pay rate is wrong in one direction or the other.

Price the study from the pilot's median time rather than your own. Keep the interface usable at phone width, since some participants will be on a phone whatever the study description says. Approve submissions quickly, both because people are waiting on the money and because your approval time is visible to future participants deciding whether to take your study.

Next steps

The full reference lives in Crowdsourcing Integration on this site, and the Prolific integration documentation in the Potato repository covers the admin API endpoints behind the dashboard panel. Start there if you need study creation, qualification sync, or paid screen-outs driven from your own code.