Content Moderation Annotation Setup

Grade toxicity on a scale rather than as a yes-or-no question, label the speaker's apparent intent as its own field, and limit how long one annotator stays in the material. The severity boundaries and the intent cases are where a moderation guideline has to be explicit.

Labeling toxic content differs from labeling movie reviews in three ways that change how the task is built. The work is distressing to do for long stretches, the guidelines are never fully determinate, and the items that matter most for a classifier are the ones annotators disagree about. This guide covers the schema design, the quality controls, and the working conditions for a moderation study in Potato, starting with the people doing the work.

Annotator wellbeing

Reading hateful and graphic content for hours has real effects on the people who do it. A few settings help keep the work bearable.

Content warnings and session caps

Potato has no dedicated wellbeing block. The pieces you need are ordinary config keys, used together.

A surveyflow page shown before the task carries the content warning and the support links, and the annotator has to pass through it to reach the items:

surveyflow:
  on: true
  order:
    - pre_annotation
  pre_annotation:
    - "surveyflow/content_warning.jsonl"

max_annotations_per_user caps how much any one person sees, which is the session limit that actually holds:

max_annotations_per_user: 100

Break reminders, cooldowns between sessions, and an in-task exit button are not built in. Where they matter, they belong in the task instructions and in what you tell annotators before they accept the work.

Content blurring

Graphic images are worth hiding behind a click, so that an annotator sees them when they are ready to and not when the page loads. Potato does not blur images, so this one has to come from your data: serve a placeholder in the field the item display reads, and put the real URL behind a link the annotator opens deliberately.

Toxicity classification

A graded severity scale

A binary toxic/not-toxic label collapses the distinction between a rude reply and a credible threat, and moderation policy treats those differently. Four graded levels give the classifier an ordering to learn and give the annotator somewhere to put the middle cases. Each level carries a definition as a tooltip, since a label named moderate means nothing on its own:

annotation_schemes:
  - annotation_type: radio
    name: toxicity_level
    description: "Rate the toxicity level of this content"
    labels:
      - name: "Not toxic"
        tooltip: "No harmful content"
 
      - name: "Mildly toxic"
        tooltip: "Rude or insensitive but not severe"
 
      - name: "Moderately toxic"
        tooltip: "Clearly offensive or harmful"
 
      - name: "Severely toxic"
        tooltip: "Extremely offensive, threatening, or dangerous"

Categories of harm

Severity says how bad an item is, and the category says what kind of harm it carries, which is what routes an item to the right policy. A multiselect fits because one post can be several things at once, and a threat wrapped in a slur is both:

annotation_schemes:
  - annotation_type: multiselect
    name: toxicity_types
    description: "Select all types of toxicity present"
    labels:
      - name: "Profanity or obscenity"
        tooltip: "Swear words, vulgar language"
 
      - name: "Insults"
        tooltip: "Personal attacks, name-calling"
 
      - name: "Threats"
        tooltip: "Threats of violence or harm"
 
      - name: "Hate speech"
        tooltip: "Targeting protected groups"
 
      - name: "Harassment"
        tooltip: "Targeted, persistent hostility"
 
      - name: "Sexual content"
        tooltip: "Explicit or suggestive content"
 
      - name: "Self-harm or suicide"
        tooltip: "Promoting or glorifying self-harm"
 
      - name: "Misinformation"
        tooltip: "Demonstrably false claims"
 
      - name: "Spam or scam"
        tooltip: "Unwanted promotional content"

Hate speech

Which group is targeted

Recording the targeted group separately from the severity lets you measure coverage per group, which is where moderation classifiers are most often unevenly trained:

annotation_schemes:
  - annotation_type: multiselect
    name: target_groups
    description: "Which groups are targeted? Leave empty if this is not hate speech"
    labels:
      - name: "Race or ethnicity"
      - name: "Religion"
      - name: "Gender"
      - name: "Sexual orientation"
      - name: "Disability"
      - name: "Nationality or origin"
      - name: "Age"
      - name: "Other protected group"

Implicit and explicit hate

Coded language is the hardest case for both annotators and classifiers, and grouping it with open slurs hides how much of it your corpus contains. A three-level scale keeps the distinction visible in the data:

annotation_schemes:
  - annotation_type: radio
    name: hate_severity
    description: "Severity of hate speech"
    labels:
      - name: "Implicit"
        tooltip: "Coded language, dog whistles"
 
      - name: "Explicit, mild"
        tooltip: "Clear but not threatening"
 
      - name: "Explicit, severe"
        tooltip: "Dehumanizing, threatening, or violent"

Context and intent

Platform context

The same sentence can be acceptable in one community and a violation in another, so a judgment made without the platform in view is not the judgment moderation needs. Pair the context rating with a free-text field, because the reasoning is what you will read when two annotators split:

annotation_schemes:
  - annotation_type: radio
    name: context_appropriate
    description: "Is this content appropriate for the platform context?"
    labels:
      - name: "Appropriate for context"
      - name: "Borderline"
      - name: "Inappropriate for context"
 
  - annotation_type: text
    name: context_notes
    description: "Explain your contextual reasoning"
    multiline: true
    rows: 3

Apparent intent

A slur quoted in a news report, reclaimed in-group use, and a direct attack can be word-for-word identical, and a classifier trained without the distinction learns to flag the reporting alongside the abuse. Intent is inferred rather than known, so the label asks what the text appears to do and gives annotators somewhere to say they cannot tell:

annotation_schemes:
  - annotation_type: radio
    name: intent
    description: "What is the apparent intent?"
    labels:
      - name: "Genuine attack"
        tooltip: "Intent to harm or offend"
 
      - name: "Satire or parody"
        tooltip: "Mocking toxic behavior"
 
      - name: "Quote or report"
        tooltip: "Reporting or discussing toxic content"
 
      - name: "Reclaimed language"
        tooltip: "In-group use of slurs"
 
      - name: "Unclear intent"

Image moderation

Visual policy violations need the same category and action split as text. The action field is what makes the annotation usable by a moderation queue rather than only by a model, since approve, age-restrict, and remove are different outcomes for the same category:

annotation_schemes:
  - annotation_type: multiselect
    name: image_violations
    description: "Select all policy violations"
    labels:
      - name: "Nudity or sexual content"
      - name: "Graphic violence"
      - name: "Gore or disturbing content"
      - name: "Hate symbols"
      - name: "Dangerous acts"
      - name: "Child safety concern"
      - name: "No violations"
 
  - annotation_type: radio
    name: action_recommendation
    description: "Recommended action"
    labels:
      - name: "Approve"
      - name: "Age-restrict"
      - name: "Add warning label"
      - name: "Remove"
      - name: "Escalate to specialist"

Quality control

Subjective tasks drift, and on this one the drift is usually downward severity as annotators habituate to the material. Gold items with known labels catch that, three judgements per item give you something to measure agreement on, and adjudication moves the disagreements to someone senior:

# Items with known labels, mixed into the regular queue
gold_standards:
  enabled: true
  items_file: data/moderation_gold.json
  mode: mixed
  frequency: 20
  accuracy:
    min_threshold: 0.7
    evaluation_count: 10
 
# Three judgements on every item
num_annotators_per_item: 3
 
# Disagreements go to a specialist queue
adjudication:
  enabled: true
  adjudicator_users:
    - trust_safety_lead
  min_annotations: 3
  agreement_threshold: 0.67
  require_notes_on_override: true
  show_agreement_scores: true

Two cautions. These blocks belong at the config root: Potato reads gold_standards from the top level, so nesting it under a quality_control: block leaves it unread and warns about nothing. And there is no built-in distribution monitoring, so nothing will alert you when the severity mix skews. Watching for that drift means exporting periodically and checking the label distribution yourself.

A complete configuration

The config below is the smaller version of the schemas above, cut to the four schemes one study needs. A severity level, the categories, a confidence rating, and a notes field cover the judgment without asking for nine fields per item. Three annotators per item is the floor for measuring agreement on a subjective task, and the confidence rating gives you a second way to find the items worth adjudicating:

annotation_task_name: "Content Moderation"
task_dir: "."
output_annotation_dir: "annotations/"
 
data_files:
  - "data/reports.json"
 
item_properties:
  id_key: id
  text_key: text
 
num_annotators_per_item: 3
 
annotation_schemes:
  - annotation_type: radio
    name: toxicity
    description: "Toxicity level"
    labels:
      - name: "None"
      - name: "Mild"
      - name: "Moderate"
      - name: "Severe"
 
  - annotation_type: multiselect
    name: categories
    description: "Types of harmful content (select all)"
    labels:
      - name: "Hate speech"
      - name: "Harassment"
      - name: "Violence or threats"
      - name: "Sexual content"
      - name: "Self-harm"
      - name: "Spam"
      - name: "None"
 
  - annotation_type: likert
    name: confidence
    description: "How confident are you in this judgment?"
    size: 5
    min_label: "Uncertain"
    max_label: "Very confident"
 
  - annotation_type: text
    name: notes
    description: "Additional notes (optional)"
    multiline: true
    rows: 4
 
export_annotation_format: jsonl

Writing the guidelines

Most disagreement between annotators traces back to vague guidelines, so this is where the effort pays off. Spell out what separates "mild" from "moderate" rather than leaving annotators to infer it from the label names. Show the edge cases with an explanation of why each one landed where it did, since the explanation is what generalizes to the next edge case. Say how platform and audience change the call, and tell annotators what to do when intent is genuinely unclear rather than leaving them to guess at a default. Expect to revise all of it as new kinds of content arrive.

Supporting annotators

Rotate the work rather than parking one person on toxic content all day. Keep mental health resources one click away, give people a real channel to raise concerns, and treat the workload as a study design decision rather than a scheduling one.

For how the underlying classification schemes work, see the text annotation documentation. The quality control guide covers attention checks, gold standards, and agreement metrics in more detail, and Annotation Schemes lists every schema type Potato accepts.