Quality Control for Crowdsourced Annotation

No single check catches every kind of bad annotation, so layer four of them: attention checks for annotators who are not reading, gold standards for annotators who are reading and still wrong, overlap for items that are ambiguous, and timing for work that was submitted too fast to have been done. Each one has its own Potato config key, and each reports into the admin dashboard separately.

Quality control is cheapest before collection and most expensive after it. An attention check that blocks an annotator on their fifth failure costs nothing to configure and saves the items they would have filled in; the same problem found in a post-hoc script costs you the re-collection. The controls below are ordered by when they act.

Quality control across the timeline

What each control catches

The four controls fail in different directions, which is the reason to run more than one.

Agreement in particular is not a catch-all. When annotators review model pre-labels, accepting a suggestion without reading it raises agreement rather than lowering it, so the rubber-stamping annotator looks like the best one on that measure. Agreement cannot catch rubber-stamping covers what to record instead.

Attention checks

Attention checks are items with a known answer, injected into the annotation stream on a schedule, that verify an annotator is reading before answering. Potato injects them, grades them, warns the annotator, and blocks them at a threshold you set, so the whole loop runs on the server rather than in a post-hoc script.

Set the frequency from how long your items take. One check every 10 items is frequent enough to catch a drifting annotator within a session without spending much of your budget on non-data. The min_response_time floor is what catches clicking rather than reading, and a response faster than it fails whatever it says:

attention_checks:
  enabled: true
  items_file: "data/attention_checks.json"
  frequency: 10              # one check every 10 items
  min_response_time: 3.0     # seconds; faster than this fails the check
 
  failure_handling:
    warn_threshold: 2
    warn_message: "Please read items carefully before answering."
    block_threshold: 5
    block_message: "You have been blocked due to too many incorrect responses."

The items file gives each check an id, the text the annotator sees, and the expected answer per scheme:

[
  {
    "id": "attn_001",
    "text": "Please select 'Positive' for this item to verify you are reading carefully.",
    "expected_answer": {
      "sentiment": "positive"
    }
  }
]

Results survive a restart. Potato writes them to quality_control_results.json in the output directory as they are recorded and reads them back at startup, so a deploy does not clear an annotator's failure history or unblock someone who had been blocked.

Gold standards

Gold standards are expert-labeled items mixed into the stream to measure accuracy rather than attention. They are silent by default, so results go to the admin dashboard and the annotator sees no feedback, which measures quality without changing the behavior you are measuring. Turn feedback on when you are training rather than screening.

Gold items are injected every frequency items, so the setting trades measurement precision against how much of the budget goes to items you already have labels for. One in 20 gives roughly 25 graded items across a 500-item shift:

gold_standards:
  enabled: true
  items_file: "data/gold_standards.json"
  mode: mixed                # mixed injects into the stream; this is the implemented mode
  frequency: 20
 
  accuracy:
    min_threshold: 0.7       # annotators below this are named in the dashboard
    evaluation_count: 10     # evaluate once an annotator has seen 10 gold items
 
  feedback:
    show_correct_answer: false
    show_explanation: false

The items file carries the correct annotation per scheme, plus an optional explanation for the feedback case:

[
  {
    "id": "gold_001",
    "text": "The service was absolutely terrible and I will never return.",
    "gold_label": {
      "sentiment": "negative"
    },
    "explanation": "Strong negative language indicates negative sentiment.",
    "difficulty": "easy"
  }
]

Choose gold items that are unambiguous to an expert. An item your own team argues about measures the guidelines rather than the annotator, and it will push competent annotators below the threshold. Potato can also promote items to gold once annotators agree on them unanimously, through gold_standards.auto_promote, which grows the pool without more expert labeling.

Per-annotator accuracy appears at /admin/api/quality_control, which reports configured_correct and configured_total alongside the totals. Quote the configured numbers when comparing annotators, because the configured pool is the same for everyone while the auto-promoted pool depends on when each annotator arrived.

Overlap and agreement

Multiple annotators per item is the control that finds ambiguous items rather than bad annotators. Three per item is the common default, because three gives a majority for aggregation and enough overlap for a coefficient. Set the target and turn on reporting:

num_annotators_per_item: 3
 
agreement_metrics:
  enabled: true
  min_overlap: 2
  refresh_interval: 60

Krippendorff's alpha then appears in the admin dashboard as annotations arrive, per scheme. Measuring inter-annotator agreement covers which coefficient to report and what the thresholds mean.

Timing and telemetry

Potato records how long each item took, not only what was answered. Each instance carries a behavioral_data object whose total_time_ms is the time on that item, alongside the interaction events, the annotation change history, and the AI-suggestion lifecycle. The field is what lets you separate an annotator who is fast because the corpus is easy from one who is fast because they are not looking:

def flag_fast_annotations(export, min_seconds=2.0):
    """Instances submitted faster than a floor, from an exported annotation file."""
    flagged = []
    for instance in export:
        bd = instance.get("behavioral_data") or {}
        seconds = bd.get("total_time_ms", 0) / 1000
        if seconds < min_seconds:
            flagged.append((instance["id"], seconds))
    return flagged

Read timing as a distribution within your own project rather than against a fixed number of seconds, since the same threshold that catches rubber-stamping on a hard corpus catches competence on an easy one. For drawn annotations, annotation_telemetry records the same idea at a finer grain. Its strongest signal is the latency between an AI suggestion appearing and the annotator accepting it. Behavioral tracking and drawing telemetry cover the fields and the flags they feed.

A configuration with all four controls

The four keys are independent, so a project can adopt them one at a time. Together they look like this:

annotation_task_name: "Sentiment Annotation with Quality Control"
task_description: "Classify the sentiment of short texts."
 
data_files:
  - data/items.json
 
item_properties:
  id_key: id
  text_key: text
 
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What is the sentiment of this text?"
    labels:
      - Positive
      - Negative
      - Neutral
 
num_annotators_per_item: 3
 
attention_checks:
  enabled: true
  items_file: data/attention_checks.json
  frequency: 15
  min_response_time: 3.0
  failure_handling:
    warn_threshold: 2
    block_threshold: 5
 
gold_standards:
  enabled: true
  items_file: data/gold_standards.json
  mode: mixed
  frequency: 25
  accuracy:
    min_threshold: 0.7
    evaluation_count: 10
 
agreement_metrics:
  enabled: true
  min_overlap: 2
  refresh_interval: 60
 
output_annotation_dir: annotation_output

Start the thresholds lenient. A block threshold of 5 and an accuracy floor of 0.7 let you see the real distribution of your annotator pool before you decide what counts as failing, and tightening them later costs nothing. Setting them hard on day one ejects annotators on the strength of guidelines that have not been piloted yet.

Platform-level controls

Crowdsourcing platforms screen the pool before anyone reaches your task, which is a different job from the checks above and worth doing as well as them.

• Prescreening. Both Prolific and MTurk filter on approval rate and prior completed work, and both support locale filters. On MTurk, an approval rate of 97% or higher with 500 or more approved HITs is a common starting filter.
• Qualification tests. MTurk supports custom qualifications, so a short test can gate access to the main task.
• Review before payment. Approve quickly when the work is fine, since slow approval is what annotators complain about most, and use rejection sparingly because it damages a worker's standing on the platform.

Platform prescreening and Potato's own checks are complementary. Prescreening decides who reaches the task; attention checks and gold standards measure what they did once they got there.

Before you collect anything

Two decisions are hard to reverse once collection starts, so make them first. Decide which fields you need in the export, because a timing analysis you planned after the fact needs behavioral_data to have been recorded during the run. And run a pilot of a few dozen items with the full control stack turned on, since a pilot is where you find out that your attention checks are ambiguous or your gold items are contested, and it costs one afternoon instead of one dataset.

Further reading

• Gold Standards and Attention Checks
• Quality Control feature reference
• Training Phase for onboarding annotators before the real task
• Writing Effective Annotation Guidelines
• Running a Study on Prolific and MTurk
• Potato's quality control source documentation