Running a Study on Prolific and MTurk
How to run a crowdsourced annotation study with Potato on Prolific or Amazon Mechanical Turk, linking participants, quality control, and fair pay.
Crowdsourcing platforms like Prolific and Amazon Mechanical Turk give you access to many annotators on demand. Potato integrates with both: participants click through from the platform, do the task, and return with a completion code. The hard part isn't the wiring, it's quality control and fair treatment of workers.
See crowdsourcing and Amazon Mechanical Turk for background. For the integration reference, see MTurk Integration and Prolific integration in the deployment docs.
How the flow works
- You deploy a Potato task on a public URL.
- The platform sends each participant to that URL with their ID in the link.
- Potato records annotations against that ID.
- On finish, Potato shows a completion code the worker submits back to the platform to get paid.
Potato captures the participant ID from the URL so annotations tie back to the right worker automatically.
Quality control is non-negotiable
Open crowdsourcing attracts some low-effort and automated responses. Build defenses in from the start:
- Attention checks to catch workers who aren't reading.
- Gold standards to score accuracy against known answers.
- A training/qualification phase so only workers who pass reach the real task.
- Overlap so you can measure agreement and aggregate.
Treat workers fairly
This is both ethical and practical, well-paid, well-briefed workers produce better data:
- Pay at least a fair hourly rate. Time your pilot to estimate how long the task really takes, then price the reward accordingly.
- Write clear instructions and a short consent/intro. See Writing Annotation Guidelines.
- Respond to questions and pay promptly; reputation affects who takes your future tasks.