Prolific gives you a pre-vetted participant pool built for research studies. This guide walks through setting up an annotation task on Prolific and paying for it.

Why Prolific?

Prolific is aimed at academic studies, so its participants are pre-screened and tend to stick with a task. It enforces minimum-wage pay, lets you filter by a wide range of demographic criteria, and has consent management built in, which makes IRB paperwork easier.

Prerequisites

Prolific researcher account
Potato installed and accessible via public URL
Basic configuration ready

Basic Integration

Potato Configuration

yaml

annotation_task_name: "Research Annotation Study"
 
# Prolific integration via URL-based login
login:
  type: url_direct
  url_argument: PROLIFIC_PID
 
# Completion redirect
finish:
  redirect_url: "https://app.prolific.co/submissions/complete?cc=XXXXXX"

Prolific Study Setup

Create new study on Prolific
Study URL: https://your-server.com/annotate?PROLIFIC_PID={{%PROLIFIC_PID%}}&STUDY_ID={{%STUDY_ID%}}&SESSION_ID={{%SESSION_ID%}}
Completion URL: Get from Potato after task completion

Complete Integration Configuration

yaml

annotation_task_name: "Sentiment Analysis - Prolific Study"
 
# Prolific integration via URL-based login
login:
  type: url_direct
  url_argument: PROLIFIC_PID
 
# Completion handling
finish:
  redirect_url: "https://app.prolific.co/submissions/complete?cc=C1A2B3C4"
  thank_you_message: "Thank you for participating! Your completion code is shown below."
 
# Data and task settings
"data_files": ["data/texts.json"]
 
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    labels: [Positive, Negative, Neutral]
    required: true
 
# Multi-phase workflow using surveyflow
surveyflow:
  on: true
  order:
    - consent
    - prestudy
    - annotation
    - poststudy
 
  consent:
    data_file: "data/consent.json"
 
  prestudy:
    data_file: "data/instructions.json"
 
instances_per_annotator: 50

Setting Up the Study on Prolific

Step 1: Create Study

Go to Prolific Dashboard → New Study
Fill in study details:
- Title: Clear, descriptive name
- Description: What participants will do
- Estimated time: Be accurate (affects pay rate display)

Step 2: Configure Study URL

text

https://your-server.com/annotate?PROLIFIC_PID={{%PROLIFIC_PID%}}&STUDY_ID={{%STUDY_ID%}}&SESSION_ID={{%SESSION_ID%}}

Step 3: Set Completion Code

In Potato config, set a unique completion code

In Prolific, use the redirect completion URL:

text

https://app.prolific.co/submissions/complete?cc=YOUR_CODE

Step 4: Participant Requirements

Filter participants by:

Demographics: Age, gender, nationality
Language: First language, fluency
Approval rate: Minimum past approval %
Custom: Previous study participation

Step 5: Payment

Set fair compensation (Prolific requires minimum wage)
Calculate: (estimated_time_minutes / 60) × hourly_rate
Consider adding bonus for quality

Handling Participant Flow

Use Potato's surveyflow to define the flow of your study:

yaml

surveyflow:
  on: true
  order:
    - consent
    - prestudy
    - annotation
    - poststudy
 
  consent:
    data_file: "data/consent.json"
 
  prestudy:
    data_file: "data/instructions.json"
 
  poststudy:
    data_file: "data/feedback.json"

Quality Control for Crowdsourcing

Attention Checks

You can include gold-standard items in your data file with known correct answers. Configure attention checks by including items with the gold field in your data:

json

{
  "id": "attention_1",
  "text": "Please select 'Positive' for this item.",
  "gold": {"sentiment": "Positive"}
}

Monitoring Quality

Potato's admin dashboard shows completion rates and annotation statistics. You can also open the output files directly to spot patterns like single-label bias or suspiciously fast completion times.

Handling Rejections

Handle quality failures through the quality_control section. You can redirect participants who fail attention checks.

Payment and Bonuses

Bonuses go through Prolific's interface. Once the study finishes, export your annotation data, work out bonuses against your criteria, and upload them to Prolific.

Monitoring Your Study

You have three views into a running study: Potato's admin dashboard, Prolific's study monitoring interface, and your own annotation output files.

Output Format

json

{
  "participant_id": "PROLIFIC_PID_XXXXX",
  "study_id": "STUDY_ID_XXXXX",
  "session_id": "SESSION_ID_XXXXX",
  "annotations": [...],
  "metadata": {
    "start_time": "2026-01-20T10:00:00Z",
    "end_time": "2026-01-20T10:25:00Z",
    "duration_minutes": 25,
    "items_completed": 50,
    "attention_checks_passed": 5,
    "attention_checks_failed": 0
  },
  "quality_metrics": {
    "avg_time_per_item": 28.5,
    "consistency_score": 0.92
  }
}

Tips for Prolific Studies

Run a pilot with 5-10 participants before you launch the full study. Pay research-level rates; Prolific participants expect them and will tell you if they're low. Write clear instructions, since most confusion and rejection traces back to vague ones. Keep the interface usable on a phone, because some participants will be on one. And approve submissions quickly, both as a courtesy and to keep your study reputation up.

Next Steps

Compare with MTurk deployment
Set up quality control in detail
Calculate inter-annotator agreement

For the full crowdsourcing reference, see the website docs at /docs/deployment/crowdsourcing or the source documentation.