Crowdsourcing Integration

Potato collects crowdsourced annotations by reading the participant ID out of the URL the platform sends people to, then showing a completion code at the end. That pattern covers Prolific and MTurk, and the generic provider covers any other panel that works the same way.

Running on an Open Port

A crowd worker has to reach the server, so run it on a port open to the internet:

potato start your-project -p 8080

Workers then arrive at https://your-server:8080/. Put a reverse proxy in front of it for TLS, as described in Production Setup.

Crowd Providers

The crowdsourcing block names the platform. Prolific and MTurk have dedicated providers, and generic handles any panel that passes an ID in the URL and accepts a completion code, such as Besample, Positly, User Interviews, Testable Minds, or Cint:

login:
  type: url_direct
  url_argument: response_id        # must match id_param below
 
crowdsourcing:
  provider: generic                # generic | prolific | mturk
  generic:
    platform_label: "Besample"     # shown on the done-page return button
    id_param: response_id          # URL parameter carrying the participant ID
    capture_params: [study]        # extra URL params to record
    completion:
      code: "BSMP-4432"
      redirect_url: "https://besample.app/complete?rid={worker_id}&code={code}"
      failed_code: "BSMP-FAIL"     # optional, for blocked participants

redirect_url accepts the placeholders {code}, {worker_id}, {session_id}, {study_id}, and any name listed in capture_params.

A config with no crowdsourcing block keeps the older behavior, where login.type: url_direct serves Prolific-style and MTurk-style arrivals at the same time.

Prolific

Minimal Setup

Two settings run a Prolific study:

login:
  type: url_direct
  url_argument: PROLIFIC_PID   # the default
 
completion_code: "YOUR-PROLIFIC-CODE"

Workers arriving from Prolific are logged in under their PROLIFIC_PID with no password prompt. When they finish, they see the completion code and a button back to Prolific.

The Study URL

In your Prolific study settings, set the URL to:

https://your-server:8080/?PROLIFIC_PID={{%PROLIFIC_PID%}}&SESSION_ID={{%SESSION_ID%}}&STUDY_ID={{%STUDY_ID%}}

Prolific substitutes the real values. Potato records SESSION_ID and STUDY_ID alongside the participant ID.

Redirecting on Completion

completion_code: "YOUR-PROLIFIC-CODE"
auto_redirect_on_completion: true
auto_redirect_delay: 5000   # milliseconds

Without the redirect, the done page shows the code with click-to-copy and a return button, which is the safer default when you want workers to keep a record of the code.

Prolific API Integration

For larger studies Potato can talk to the Prolific API, which lets it release slots held by returned or timed-out workers and pause the study when the server is busy. Put the credentials in their own file:

# configs/prolific_config.yaml
token: "your-prolific-api-token"
study_id: "your-study-id"
max_concurrent_sessions: 30
workload_checker: true
workload_checker_period: 300

Then point the main config at it:

login:
  type: prolific
 
prolific:
  config_file_path: configs/prolific_config.yaml
 
completion_code: "YOUR-PROLIFIC-CODE"

workload_checker is off unless you turn it on. With it on, Potato pauses the study when active workers exceed max_concurrent_sessions and resumes when they fall to 20% of that number, though only if the monitor paused the study itself. A study you paused by hand is never restarted for you.

Reclaiming Abandoned Work

When a submission comes back RETURNED, TIMED-OUT, or REJECTED, Potato releases that worker's assigned-but-unannotated items so someone else can take them. Annotations they already completed are kept. instance_reclaim changes that per status:

instance_reclaim:
  enabled: true
  prolific:
    preserve_completed_annotations: true
    status_policies:
      TIMED-OUT:
        preserve_completed_annotations: true
      REJECTED:
        preserve_completed_annotations: false

Amazon Mechanical Turk

Minimal Setup

login:
  type: url_direct
  url_argument: workerId
 
completion_code: "COMPLETED"
 
hide_navbar: true
jumping_to_id_disabled: true

The External Question HIT

MTurk sends workers to your server through an External Question:

<?xml version="1.0" encoding="UTF-8"?>
<ExternalQuestion xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2006-07-14/ExternalQuestion.xsd">
  <ExternalURL>https://your-server:8080/?workerId=${workerId}&amp;assignmentId=${assignmentId}&amp;hitId=${hitId}&amp;turkSubmitTo=${turkSubmitTo}</ExternalURL>
  <FrameHeight>800</FrameHeight>
</ExternalQuestion>

Potato reads four parameters from that URL:

Include turkSubmitTo in the URL. Without it the submit button on the completion page has nowhere to send the assignment.

Preview Mode

MTurk sends assignmentId=ASSIGNMENT_ID_NOT_AVAILABLE while a worker is previewing a HIT they have not accepted. Potato shows a preview page instead of starting the task, and refreshes every three seconds so the task begins once the HIT is accepted. Sandbox and production differ only in the host that turkSubmitTo points at, so the same config works in both.

MTurk API Integration

With boto3 installed and AWS credentials that carry MTurk requester permissions, Potato can list assignments and auto-approve them:

# configs/mturk_config.yaml
aws_access_key_id: "YOUR_AWS_ACCESS_KEY"
aws_secret_access_key: "YOUR_AWS_SECRET_KEY"
sandbox: true
hit_id: "YOUR_HIT_ID"

login:
  type: url_direct
  url_argument: workerId
 
mturk:
  enabled: true
  config_file_path: configs/mturk_config.yaml

Keep that file out of version control, and prefer environment variables in production.

Quality Control

Attention Checks

Items with known answers, mixed into the queue:

attention_checks:
  enabled: true
  items_file: data/attention_checks.json
  frequency: 10
  min_response_time: 3.0
  failure_handling:
    warn_threshold: 2
    warn_message: "Please read items carefully before answering."
    block_threshold: 5
    block_message: "You have been blocked due to too many incorrect responses."

min_response_time catches the failure mode crowdsourcing is most exposed to, which is clicking through faster than anyone could read the item.

Gold Standards

Expert-labeled items measure accuracy rather than attention, and they stay silent by default so workers get no feedback that would teach them the answers:

gold_standards:
  enabled: true
  items_file: data/gold_standards.json
  mode: mixed
  frequency: 20
  accuracy:
    min_threshold: 0.7
    evaluation_count: 5

See Quality Control for the item formats and how both are reported.

Custom Completion Pages

The built-in completion page handles a code and a return button. When you need more than that, end the study with a surveyflow page instead:

surveyflow:
  on: true
  order:
    - pre_annotation
    - post_annotation
  pre_annotation:
    - surveyflow/consent.jsonl
  post_annotation:
    - surveyflow/end.jsonl

The end page is a pure_display item carrying whatever link or text you need:

{"id":"1","text":"Thanks for participating. Click below to complete the study.","schema":"pure_display","choices":["<a href=\"https://app.prolific.com/submissions/complete?cc=YOUR-CODE\" target=\"_blank\">Complete Study on Prolific</a>"]}

Full Example: Prolific Study

annotation_task_name: "Sentiment Analysis Study"
task_dir: .
 
login:
  type: url_direct
  url_argument: PROLIFIC_PID
 
completion_code: "SENT2024"
auto_redirect_on_completion: false
 
hide_navbar: true
jumping_to_id_disabled: true
 
user_config:
  allow_all_users: true
 
assignment_strategy: random
max_annotations_per_user: 50
num_annotators_per_item: 3
 
phases:
  order:
    - consent
    - instructions
    - annotation
  consent:
    file: data/consent.html
  instructions:
    file: data/instructions.html
  annotation:
    type: annotation
 
attention_checks:
  enabled: true
  items_file: data/attention_checks.json
  frequency: 10
  failure_handling:
    warn_threshold: 2
    block_threshold: 5
 
data_files:
  - data/main.json
 
item_properties:
  id_key: id
  text_key: text
 
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What is the sentiment of this text?"
    labels:
      - Positive
      - Negative
      - Neutral
    sequential_key_binding: true
 
output_annotation_dir: annotation_output/
export_annotation_format: jsonl

Full Example: MTurk HIT

annotation_task_name: "Image Classification HIT"
task_dir: .
 
login:
  type: url_direct
  url_argument: workerId
 
completion_code: "TASK_COMPLETE"
 
hide_navbar: true
jumping_to_id_disabled: true
 
user_config:
  allow_all_users: true
 
assignment_strategy: random
max_annotations_per_user: 20
num_annotators_per_item: 3
 
data_files:
  - data/items.json
 
item_properties:
  id_key: id
  text_key: image_url
 
instance_display:
  fields:
    - key: image_url
      type: image
      label: "Image"
      display_options:
        max_width: 800
 
annotation_schemes:
  - annotation_type: radio
    name: category
    description: "What is shown in this image?"
    labels:
      - Cat
      - Dog
      - Bird
      - Other
    sequential_key_binding: true
 
output_annotation_dir: annotation_output/

Monitoring the Study

The admin dashboard at /admin reports completion rates, time per instance, and attention-check results while the study runs. Reaching it needs either the shared admin_api_key or a role carrying view_admin_dashboard:

admin_api_key: "your-shared-key"
 
rbac:
  enabled: true
  user_role_assignments:
    researcher@university.edu: admin

Export the collected annotations with the export CLI:

python -m potato.export -c config.yaml -f csv -o ./export/

Best Practices

1. Pilot first - Run with a handful of workers before launching at scale
2. Pay fairly - Time the task yourself and price it against the real duration
3. Write clear instructions - Include examples and the edge cases you expect
4. Use attention checks - Catch clicking-through early rather than at analysis
5. Add gold standards - Attention and accuracy are different measurements
6. Watch the dashboard - Problems are cheap to fix on day one and expensive on day five
7. Decide rejection criteria upfront - Workers should know the standard before they start
8. Export regularly - Do not wait until the study closes

Further Reading

• MTurk Integration - Detailed MTurk setup
• Passwordless Login - URL-based authentication
• Quality Control - Attention checks and gold standards
• User Management - Access lists, roles, and quotas

For implementation details, see the source documentation, the Prolific reference, and the MTurk reference.