# Deploying to Amazon Mechanical Turk

Source: https://www.potatoannotator.com/blog/mturk-deployment

Amazon Mechanical Turk (MTurk) gives you a large on-demand workforce for annotation tasks. Potato plugs into MTurk through the ExternalQuestion HIT type: your Potato server is the annotation interface, and MTurk takes care of recruiting workers, tracking assignments, and paying them. Here is how to set the whole thing up.

## Prerequisites

1. An AWS account with MTurk enabled
2. An MTurk Requester account (production or sandbox) at [requester.mturk.com](https://requester.mturk.com)
3. A Potato server reachable at a public URL (use HTTPS)
4. A Python environment with Potato installed
5. Some familiarity with MTurk concepts (HITs, Workers, Assignments)

## How the integration works

Potato does not manage MTurk HITs directly. The flow goes like this:

1. You create an ExternalQuestion HIT on MTurk that points to your Potato server URL
2. A worker accepts the HIT on MTurk and gets redirected to your Potato server with query parameters (`workerId`, `assignmentId`, `hitId`, `turkSubmitTo`)
3. Potato reads the `workerId` parameter to identify the worker (via `login.type: url_direct`)
4. The worker does the annotation task on your Potato server
5. When they finish, Potato redirects them back to MTurk's submission endpoint

## Configuration

MTurk integration comes down to one thing: set the login type to `url_direct` with `url_argument: workerId`. That tells Potato to pull the worker's identity from the URL query parameter MTurk passes automatically.

```yaml
login:
  type: url_direct
  url_argument: workerId
```

That is the only MTurk-specific setting in Potato. HIT creation, qualifications, payment, and approval all happen on the MTurk side.

## Complete configuration example

```yaml
annotation_task_name: "Sentiment Classification"
task_description: "Classify the sentiment of short text snippets."

# MTurk login: extract worker ID from URL parameter
login:
  type: url_direct
  url_argument: workerId

# UI settings recommended for crowdsourcing
hide_navbar: true
jumping_to_id_disabled: true

# Assignment settings
assignment_strategy: random
max_annotations_per_user: 20
max_annotations_per_item: 3

# Data
data_files:
  - data/items.json

item_properties:
  id_key: id
  text_key: text

# Annotation scheme
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What is the sentiment of this text?"
    labels:
      - Positive
      - Negative
      - Neutral

# Output
output_annotation_dir: annotation_output
export_annotation_format: json
```

## Setting up on MTurk

### Step 1: Start your Potato server

Launch your Potato server on a publicly reachable machine:

```bash
potato start config.yaml -p 8080
```

Make sure the server is reachable from the internet (for example, `https://your-server.com:8080/`).

### Step 2: Create the ExternalQuestion XML

MTurk embeds external websites in a HIT using an XML format called ExternalQuestion. Here is the XML you need:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<ExternalQuestion xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2006-07-14/ExternalQuestion.xsd">
  <ExternalURL>https://your-server.com:8080/?workerId=${workerId}&amp;assignmentId=${assignmentId}&amp;hitId=${hitId}&amp;turkSubmitTo=${turkSubmitTo}</ExternalURL>
  <FrameHeight>800</FrameHeight>
</ExternalQuestion>
```

Important: use `&amp;` instead of `&` in the XML. MTurk swaps the `${...}` placeholders for real values when a worker accepts the HIT.

### Step 3: Create the HIT on MTurk

You can create HITs through the MTurk Requester Console or programmatically with the AWS SDK (boto3). HIT settings like title, description, reward, duration, and qualifications all live on the MTurk side, not in Potato.

#### Using boto3 (Python)

```python
import boto3

# Use sandbox for testing
mturk = boto3.client(
    'mturk',
    region_name='us-east-1',
    endpoint_url='https://mturk-requester-sandbox.us-east-1.amazonaws.com'
)

# For production, omit endpoint_url or use:
# endpoint_url='https://mturk-requester.us-east-1.amazonaws.com'

question_xml = '''<?xml version="1.0" encoding="UTF-8"?>
<ExternalQuestion xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2006-07-14/ExternalQuestion.xsd">
  <ExternalURL>https://your-server.com:8080/?workerId=${workerId}&amp;assignmentId=${assignmentId}&amp;hitId=${hitId}&amp;turkSubmitTo=${turkSubmitTo}</ExternalURL>
  <FrameHeight>800</FrameHeight>
</ExternalQuestion>'''

response = mturk.create_hit(
    Title='Sentiment Classification Task',
    Description='Read short texts and classify their sentiment as positive, negative, or neutral.',
    Keywords='sentiment, classification, text, NLP',
    Reward='0.50',
    MaxAssignments=100,
    LifetimeInSeconds=86400,         # 1 day
    AssignmentDurationInSeconds=3600, # 1 hour
    AutoApprovalDelayInSeconds=604800, # 7 days
    Question=question_xml,
    QualificationRequirements=[
        {
            'QualificationTypeId': '000000000000000000L0',  # Approval rate
            'Comparator': 'GreaterThanOrEqualTo',
            'IntegerValues': [97]
        },
        {
            'QualificationTypeId': '00000000000000000040',  # Number approved
            'Comparator': 'GreaterThanOrEqualTo',
            'IntegerValues': [500]
        },
        {
            'QualificationTypeId': '00000000000000000071',  # Locale
            'Comparator': 'In',
            'LocaleValues': [
                {'Country': 'US'},
                {'Country': 'GB'},
                {'Country': 'CA'},
                {'Country': 'AU'}
            ]
        }
    ]
)

print(f"Created HIT: {response['HIT']['HITId']}")
```

### Step 4: Set qualifications (on MTurk)

You set worker qualifications on the MTurk side when you create the HIT. Common filters:

- Approval rate: require a minimum HIT approval percentage, say 97% or higher
- HITs approved: require a minimum number of previously approved HITs, say 500 or more
- Locale: limit to workers from specific countries
- Masters: use MTurk's pre-vetted Masters workers (this costs more)
- Custom qualifications: build your own qualification tests through the MTurk console

## Completion handling

When a worker finishes all their assigned items, Potato has to redirect them back to MTurk so the assignment gets submitted. MTurk passes a `turkSubmitTo` URL parameter that tells Potato where to send the completion POST request.

After finishing the task, the worker sees a "Submit HIT" button. Clicking it sends the assignment back to MTurk for your review and approval.

## Testing in the MTurk Sandbox

Test the whole thing in the MTurk Sandbox before you spend real money in production.

| Service | URL |
|---------|-----|
| Requester Sandbox | https://requestersandbox.mturk.com |
| Worker Sandbox | https://workersandbox.mturk.com |
| API Endpoint (Sandbox) | https://mturk-requester-sandbox.us-east-1.amazonaws.com |

### Local testing

You can test the URL parameter flow locally without MTurk:

```bash
# Simulate a worker accessing your task
curl "http://localhost:8080/?workerId=TEST_WORKER&assignmentId=TEST_ASSIGN&hitId=TEST_HIT"

# Simulate the preview mode (before a worker accepts the HIT)
curl "http://localhost:8080/?workerId=TEST_WORKER&assignmentId=ASSIGNMENT_ID_NOT_AVAILABLE&hitId=TEST_HIT"
```

When `assignmentId` is `ASSIGNMENT_ID_NOT_AVAILABLE`, the worker is previewing the HIT and has not accepted it yet.

## Managing HITs and approvals

You manage HITs through MTurk's own tools, whether that is monitoring progress, approving or rejecting assignments, or issuing bonuses:

- MTurk Requester Console: a web interface for managing HITs, reviewing assignments, and messaging workers
- boto3 (the AWS SDK for Python): programmatic access for batch operations

```python
# Example: List assignments for a HIT
assignments = mturk.list_assignments_for_hit(
    HITId='YOUR_HIT_ID',
    AssignmentStatuses=['Submitted']
)

# Approve an assignment
mturk.approve_assignment(AssignmentId='ASSIGNMENT_ID')

# Reject an assignment (use sparingly)
mturk.reject_assignment(
    AssignmentId='ASSIGNMENT_ID',
    RequesterFeedback='Did not complete all items.'
)
```

## Cost calculation

MTurk charges fees on top of the reward you pay workers:

- Base fee: 20% of the reward amount
- Masters qualification: an extra 5%
- 10 or more assignments per HIT: an extra 20%

For example, if you pay $0.50 per assignment across 100 assignments:
- Base cost: 100 x $0.50 x 1.20 = $60.00
- With Masters: 100 x $0.50 x 1.25 = $62.50

## Things worth doing

1. Start in the sandbox. Run the whole workflow there before any money changes hands.
2. Pay fairly. Work out the hourly rate (reward / estimated time x 60) and aim for $12 to $15 an hour or better.
3. Write clear HIT descriptions. Good titles and descriptions pull in better workers.
4. Approve quickly. Workers notice fast payment, so approve as soon as the quality looks fine.
5. Be careful with rejections. They hurt workers' approval rates and your own requester reputation, so use them sparingly.
6. Use HTTPS. Some browsers block mixed content, and HTTPS keeps the iframe working.
7. Set `hide_navbar: true` so workers cannot wander off the task inside Potato.
8. Keep your server up for the whole life of the HIT.

## MTurk vs Prolific

| Aspect | MTurk | Prolific |
|--------|-------|----------|
| Worker pool | Large, diverse | Smaller, research-focused |
| Quality | Variable | Generally higher |
| Pricing | Lower base, + fees | Higher, transparent |
| Setup | More complex | Simpler |
| Best for | Large scale, budget | Research, quality |
| Potato config | `url_argument: workerId` | `url_argument: PROLIFIC_PID` |

## Where to go next

- Compare this with [Prolific integration](/blog/prolific-integration)
- Set up [quality control](/blog/quality-control-strategies)
- Calculate [inter-annotator agreement](/blog/inter-annotator-agreement)

For the full MTurk setup details, see the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/deployment/mturk_integration.md). If you are running a larger crowdsourcing effort, the [crowdsourcing guide](https://github.com/davidjurgens/potato/blob/master/docs/deployment/crowdsourcing.md) covers worker management and quality monitoring across platforms.

---

*Full MTurk documentation at [/docs/deployment/mturk-integration](/docs/deployment/mturk-integration).*