Annotator Demographics with Consent

A subjective offensiveness-rating task wrapped in an informed-consent page and a standardized demographic survey, so you can analyze labels by annotator background.

Configuration Fileconfig.yaml

yaml

annotation_task_name: "Offensiveness Rating with Annotator Demographics"
task_dir: "."

port: 8000

# Data configuration
data_files:
  - "sample-data.json"

item_properties:
  id_key: id
  text_key: text

# Multi-phase workflow: consent, then a standardized demographic survey,
# then the rating task, then a short post-study survey.
phases:
  order: [consent, prestudy, annotation, poststudy]

  consent:
    type: consent
    data_file: "data/consent.json"

  prestudy:
    type: prestudy
    # Standardized demographic battery from the built-in instrument library.
    # Swap for gss-demographics, acs-demographics, ipums-demographics, ... or
    # combine several with an `instruments:` list.
    instrument: "anes-demographics"
    # Append a study-specific question after the battery.
    file: "data/extra_demographics.json"

  # annotation phase is always enabled

  poststudy:
    type: poststudy
    data_file: "data/feedback.json"

# The labeling task itself
annotation_schemes:
  - annotation_type: likert
    name: offensiveness
    description: "How offensive is this comment?"
    size: 5
    min_label: "Not at all offensive"
    max_label: "Very offensive"
    keyboard_shortcuts: true

  - annotation_type: radio
    name: target
    description: "If offensive, who is the target? (optional)"
    labels:
      - An individual
      - A group
      - No clear target
      - Not offensive

# User settings
require_password: false

# Output
output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

Sample Datasample-data.json

json

[
  {
    "id": "c_001",
    "text": "Honestly, a take like that is exactly what I'd expect from someone who's never had to deal with it."
  },
  {
    "id": "c_002",
    "text": "Thanks for putting this together, it clarified a lot for me."
  }
]

// ... and 4 more items

Get This Design

This design is available in our showcase. Copy the configuration below to get started.

Quick start:

# Create your project folder
mkdir annotator-demographics-consent
cd annotator-demographics-consent
# Copy config.yaml from above
potato start config.yaml

Details

Annotation Types

likertradio

Domain

Data EthicsContent Moderation

Use Cases

annotator demographicssubjective annotationresponsible data collection

Related Designs

Dynamic Hate Speech Detection

Hate speech classification with fine-grained type labels based on the Dynamically Generated Hate Speech Dataset (Vidgen et al., ACL 2021). Classify content as hateful or not, then identify hate type (animosity, derogation, dehumanization, threatening, support for hateful entities) and target group.

likertmultiselect

Implicit Hate Speech Detection

Detect and categorize implicit hate speech using a six-category taxonomy. Based on ElSherief et al., EMNLP 2021. Identifies grievance, incitement, stereotypes, inferiority, irony, and threats.