Skip to content
Guides7 min read

Deploying to Amazon Mechanical Turk

How Potato connects to Mechanical Turk through an ExternalQuestion HIT, what url_direct login does, and what MTurk's 2026 closure means for a study you are planning now.

Potato Team

Potato connects to Mechanical Turk through one config setting, login.type: url_direct with url_argument: workerId, and MTurk handles everything else. Amazon closed MTurk to new requesters on July 30, 2026 and has announced that the service closes on September 30, 2026, so new studies should go to another platform.

Check the closure dates before you build anything on this page. Existing requesters can keep posting work until the service closes, and the integration below keeps working for them, so this guide is written for a study that is already running on MTurk. If you are planning a new crowdsourced study, Prolific uses the same Potato mechanism with a different URL parameter, and running a study on Prolific and MTurk compares the two.

How the integration works

Potato does not create or manage HITs, the unit of work on Mechanical Turk. MTurk recruits the workers and tracks their assignments, your Potato server is the annotation interface, and a single URL parameter connects the two.

  1. You create an ExternalQuestion HIT on MTurk pointing at your Potato server.
  2. A worker accepts the HIT and MTurk redirects them to your server with four query parameters, workerId, assignmentId, hitId and turkSubmitTo.
  3. Potato reads workerId and uses it as the annotator's identity, so nobody creates an account or picks a password.
  4. The worker annotates on your server.
  5. On completion the worker submits back to MTurk through the turkSubmitTo URL, and the assignment appears in your requester console for approval.

The only piece of this that lives in Potato is step 3.

What you need before starting

  1. An MTurk requester account created before the July 2026 cutoff, at requester.mturk.com, funded for production work.
  2. A Potato server on a publicly reachable machine, served over HTTPS, since some browsers block mixed content inside the HIT frame.
  3. A Python environment with Potato installed.
  4. An AWS account if you intend to create HITs programmatically rather than through the requester console.

Configuring Potato for MTurk

url_direct login tells Potato to take the annotator's identity from a URL query parameter instead of a login form, and url_argument names the parameter. MTurk always passes workerId, so those two lines are the whole integration:

yaml
login:
  type: url_direct
  url_argument: workerId

Everything else in the config is ordinary Potato. The settings worth adding for crowdsourcing are about keeping a worker inside the task and capping how much any one of them can take:

yaml
annotation_task_name: "Sentiment Classification"
task_description: "Classify the sentiment of short text snippets."
 
login:
  type: url_direct
  url_argument: workerId
 
# Keep workers inside the task rather than browsing the instance list
hide_navbar: true
jumping_to_id_disabled: true
 
# Assignment
assignment_strategy: random
max_annotations_per_user: 20
max_annotations_per_item: 3
 
data_files:
  - data/items.json
 
item_properties:
  id_key: id
  text_key: text
 
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What is the sentiment of this text?"
    labels:
      - Positive
      - Negative
      - Neutral
 
output_annotation_dir: annotation_output
export_annotation_format: jsonl

max_annotations_per_user: 20 is the setting that decides your unit of work, because it caps a worker's session at 20 items and the HIT reward should pay for exactly that. max_annotations_per_item: 3 gives three independent labels per item, which is the overlap agreement statistics need. Start the server on the port you will expose:

bash
potato start config.yaml -p 8080

Creating the HIT on MTurk

MTurk embeds an external site in a HIT through an XML document called an ExternalQuestion. MTurk substitutes the ${...} placeholders when a worker accepts, and the ampersands have to be escaped as & or the XML is rejected:

xml
<?xml version="1.0" encoding="UTF-8"?>
<ExternalQuestion xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2006-07-14/ExternalQuestion.xsd">
  <ExternalURL>https://your-server.com:8080/?workerId=${workerId}&amp;assignmentId=${assignmentId}&amp;hitId=${hitId}&amp;turkSubmitTo=${turkSubmitTo}</ExternalURL>
  <FrameHeight>800</FrameHeight>
</ExternalQuestion>

Title, description, reward, duration and qualifications are all MTurk-side settings, whether you create the HIT in the requester console or through the AWS SDK. The qualification block decides who reaches your task, and the three filters below are the common starting set:

python
import boto3
 
# Sandbox endpoint; omit endpoint_url for production
mturk = boto3.client(
    'mturk',
    region_name='us-east-1',
    endpoint_url='https://mturk-requester-sandbox.us-east-1.amazonaws.com'
)
 
question_xml = '''<?xml version="1.0" encoding="UTF-8"?>
<ExternalQuestion xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2006-07-14/ExternalQuestion.xsd">
  <ExternalURL>https://your-server.com:8080/?workerId=${workerId}&amp;assignmentId=${assignmentId}&amp;hitId=${hitId}&amp;turkSubmitTo=${turkSubmitTo}</ExternalURL>
  <FrameHeight>800</FrameHeight>
</ExternalQuestion>'''
 
response = mturk.create_hit(
    Title='Sentiment Classification Task',
    Description='Read short texts and classify their sentiment.',
    Keywords='sentiment, classification, text, NLP',
    Reward='0.50',
    MaxAssignments=100,
    LifetimeInSeconds=86400,           # HIT expires after 1 day
    AssignmentDurationInSeconds=3600,  # worker has 1 hour once accepted
    AutoApprovalDelayInSeconds=604800, # auto-approves after 7 days
    Question=question_xml,
    QualificationRequirements=[
        {
            'QualificationTypeId': '000000000000000000L0',  # approval rate
            'Comparator': 'GreaterThanOrEqualTo',
            'IntegerValues': [97]
        },
        {
            'QualificationTypeId': '00000000000000000040',  # HITs approved
            'Comparator': 'GreaterThanOrEqualTo',
            'IntegerValues': [500]
        },
        {
            'QualificationTypeId': '00000000000000000071',  # locale
            'Comparator': 'In',
            'LocaleValues': [
                {'Country': 'US'}, {'Country': 'GB'},
                {'Country': 'CA'}, {'Country': 'AU'}
            ]
        }
    ]
)
 
print(f"Created HIT: {response['HIT']['HITId']}")

AutoApprovalDelayInSeconds is worth setting deliberately, because the delay is the deadline after which unreviewed work pays automatically. The same number is your review window and the longest a worker can wait for money they have earned.

Testing in the sandbox

Run the whole workflow in the sandbox before any money moves. The sandbox is a separate environment with its own accounts and its own worker pool.

ServiceURL
Requester sandboxhttps://requestersandbox.mturk.com
Worker sandboxhttps://workersandbox.mturk.com
API endpointhttps://mturk-requester-sandbox.us-east-1.amazonaws.com

The URL parameter flow can also be exercised without MTurk at all, which is the fastest way to confirm that url_direct is picking up the identity you expect:

bash
# A worker arriving with an accepted assignment
curl "http://localhost:8080/?workerId=TEST_WORKER&assignmentId=TEST_ASSIGN&hitId=TEST_HIT"
 
# Preview mode, before the worker has accepted
curl "http://localhost:8080/?workerId=TEST_WORKER&assignmentId=ASSIGNMENT_ID_NOT_AVAILABLE&hitId=TEST_HIT"

MTurk sends the literal string ASSIGNMENT_ID_NOT_AVAILABLE while a worker is previewing a HIT they have not accepted. Previewing workers should be able to read the instructions and should not be able to submit annotations, so check that path as carefully as the accepted one.

Approving work

Approvals, rejections and bonuses all happen on the MTurk side, through the requester console or through boto3:

python
assignments = mturk.list_assignments_for_hit(
    HITId='YOUR_HIT_ID',
    AssignmentStatuses=['Submitted']
)
 
mturk.approve_assignment(AssignmentId='ASSIGNMENT_ID')
 
mturk.reject_assignment(
    AssignmentId='ASSIGNMENT_ID',
    RequesterFeedback='Did not complete all items.'
)

Approve quickly when the work is fine. A rejection lowers a worker's approval rate, which is the number other requesters filter on, so it carries a cost well beyond your own study and belongs to clear failures rather than borderline quality. For deciding which submissions are actually bad, the checks in quality control strategies run inside Potato and give you evidence that is independent of the platform.

MTurk fees and what a study costs

MTurk charges a commission on top of what you pay workers. As of September 2026 the published requester fees are a 20% commission on rewards and bonuses, an additional 20% on HITs with 10 or more assignments, an extra 5% for the Masters qualification, and a minimum of $0.01 per assignment.

For 100 assignments at a $0.50 reward, the 20% base commission puts the cost at $60.00, and the extra 20% for a HIT of that size takes it to $70.00. Work the reward out from time rather than from the total budget. Estimate how long your 20-item unit takes, divide the target hourly rate by that, and pay at least $12 to $15 an hour, which is both the ethical floor and what keeps experienced workers on a task.

Where to go next