# Running Annotation Studies on Prolific

Source: https://www.potatoannotator.com/blog/prolific-integration

Prolific gives you a pre-vetted participant pool built for research studies. This guide walks through setting up an annotation task on Prolific and paying for it.

## Why Prolific?

Prolific is aimed at academic studies, so its participants are pre-screened and tend to stick with a task. It enforces minimum-wage pay, lets you filter by a wide range of demographic criteria, and has consent management built in, which makes IRB paperwork easier.

## Prerequisites

1. Prolific researcher account
2. Potato installed and accessible via public URL
3. Basic configuration ready

## Basic Integration

### Potato Configuration

```yaml
annotation_task_name: "Research Annotation Study"

# Prolific integration via URL-based login
login:
  type: url_direct
  url_argument: PROLIFIC_PID

# Completion redirect
finish:
  redirect_url: "https://app.prolific.co/submissions/complete?cc=XXXXXX"
```

### Prolific Study Setup

1. **Create new study** on Prolific
2. **Study URL**: `https://your-server.com/annotate?PROLIFIC_PID={{%PROLIFIC_PID%}}&STUDY_ID={{%STUDY_ID%}}&SESSION_ID={{%SESSION_ID%}}`
3. **Completion URL**: Get from Potato after task completion

## Complete Integration Configuration

```yaml
annotation_task_name: "Sentiment Analysis - Prolific Study"

# Prolific integration via URL-based login
login:
  type: url_direct
  url_argument: PROLIFIC_PID

# Completion handling
finish:
  redirect_url: "https://app.prolific.co/submissions/complete?cc=C1A2B3C4"
  thank_you_message: "Thank you for participating! Your completion code is shown below."

# Data and task settings
"data_files": ["data/texts.json"]

annotation_schemes:
  - annotation_type: radio
    name: sentiment
    labels: [Positive, Negative, Neutral]
    required: true

# Multi-phase workflow using surveyflow
surveyflow:
  on: true
  order:
    - consent
    - prestudy
    - annotation
    - poststudy

  consent:
    data_file: "data/consent.json"

  prestudy:
    data_file: "data/instructions.json"

instances_per_annotator: 50
```

## Setting Up the Study on Prolific

### Step 1: Create Study

1. Go to Prolific Dashboard → New Study
2. Fill in study details:
   - **Title**: Clear, descriptive name
   - **Description**: What participants will do
   - **Estimated time**: Be accurate (affects pay rate display)

### Step 2: Configure Study URL

```
https://your-server.com/annotate?PROLIFIC_PID={{%PROLIFIC_PID%}}&STUDY_ID={{%STUDY_ID%}}&SESSION_ID={{%SESSION_ID%}}
```

### Step 3: Set Completion Code

1. In Potato config, set a unique completion code
2. In Prolific, use the redirect completion URL:
   ```
   https://app.prolific.co/submissions/complete?cc=YOUR_CODE
   ```

### Step 4: Participant Requirements

Filter participants by:
- **Demographics**: Age, gender, nationality
- **Language**: First language, fluency
- **Approval rate**: Minimum past approval %
- **Custom**: Previous study participation

### Step 5: Payment

- Set fair compensation (Prolific requires minimum wage)
- Calculate: (estimated_time_minutes / 60) × hourly_rate
- Consider adding bonus for quality

## Handling Participant Flow

Use Potato's `surveyflow` to define the flow of your study:

```yaml
surveyflow:
  on: true
  order:
    - consent
    - prestudy
    - annotation
    - poststudy

  consent:
    data_file: "data/consent.json"

  prestudy:
    data_file: "data/instructions.json"

  poststudy:
    data_file: "data/feedback.json"
```

## Quality Control for Crowdsourcing

### Attention Checks

You can include gold-standard items in your data file with known correct answers. Configure attention checks by including items with the `gold` field in your data:

```json
{
  "id": "attention_1",
  "text": "Please select 'Positive' for this item.",
  "gold": {"sentiment": "Positive"}
}
```

### Monitoring Quality

Potato's admin dashboard shows completion rates and annotation statistics. You can also open the output files directly to spot patterns like single-label bias or suspiciously fast completion times.

## Handling Rejections

Handle quality failures through the quality_control section. You can redirect participants who fail attention checks.

## Payment and Bonuses

Bonuses go through Prolific's interface. Once the study finishes, export your annotation data, work out bonuses against your criteria, and upload them to Prolific.

## Monitoring Your Study

You have three views into a running study: Potato's admin dashboard, Prolific's study monitoring interface, and your own annotation output files.

## Output Format

```json
{
  "participant_id": "PROLIFIC_PID_XXXXX",
  "study_id": "STUDY_ID_XXXXX",
  "session_id": "SESSION_ID_XXXXX",
  "annotations": [...],
  "metadata": {
    "start_time": "2026-01-20T10:00:00Z",
    "end_time": "2026-01-20T10:25:00Z",
    "duration_minutes": 25,
    "items_completed": 50,
    "attention_checks_passed": 5,
    "attention_checks_failed": 0
  },
  "quality_metrics": {
    "avg_time_per_item": 28.5,
    "consistency_score": 0.92
  }
}
```

## Tips for Prolific Studies

Run a pilot with 5-10 participants before you launch the full study. Pay research-level rates; Prolific participants expect them and will tell you if they're low. Write clear instructions, since most confusion and rejection traces back to vague ones. Keep the interface usable on a phone, because some participants will be on one. And approve submissions quickly, both as a courtesy and to keep your study reputation up.

## Next Steps

- Compare with [MTurk deployment](/blog/mturk-deployment)
- Set up [quality control](/blog/quality-control-strategies) in detail
- Calculate [inter-annotator agreement](/blog/inter-annotator-agreement)

---

*For the full crowdsourcing reference, see the website docs at [/docs/deployment/crowdsourcing](/docs/deployment/crowdsourcing) or the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/deployment/crowdsourcing.md).*