Skip to content
Documentation
Docs/Features

Task Assignment

Control how annotation instances are distributed to annotators.

Task Assignment

Potato provides flexible task assignment strategies to control how annotation instances are distributed to annotators.

Overview

Task assignment controls:

  • Which items each annotator sees
  • How many items each annotator completes
  • How many annotations each item receives
  • The order in which items are presented

Key Configuration Options

OptionDescriptionDefault
assignment_strategyStrategy for assigning itemsrandom
max_annotations_per_userMaximum items per annotatorunlimited
max_annotations_per_itemTarget annotations per item3

Assignment Strategies

Random Assignment

Assigns items randomly to annotators, ensuring unbiased distribution.

yaml
assignment_strategy: random
max_annotations_per_item: 3

Best for: General annotation tasks where order doesn't matter.

Fixed Order Assignment

Assigns items in the order they appear in the dataset.

yaml
assignment_strategy: fixed_order
max_annotations_per_item: 2

Best for: Tasks where annotators should see items in a specific sequence.

Least-Annotated Assignment

Prioritizes items with the fewest existing annotations, ensuring even distribution.

yaml
assignment_strategy: least_annotated
max_annotations_per_item: 5

Best for: Ensuring all items receive adequate coverage before any item gets excessive annotations.

Max-Diversity Assignment

Prioritizes items with the highest disagreement among existing annotations.

yaml
assignment_strategy: max_diversity
max_annotations_per_item: 4

Best for: Quality control and resolving ambiguous items.

Active Learning Assignment

Uses machine learning to prioritize uncertain instances.

yaml
assignment_strategy: active_learning
 
active_learning:
  enabled: true
  schema_names: ["sentiment"]
  min_annotations_per_instance: 2
  min_instances_for_training: 20
  update_frequency: 10

See Active Learning for full configuration.

Configuration

Modern Configuration (Recommended)

yaml
# Strategy selection
assignment_strategy: random
 
# Limits
max_annotations_per_user: 10    # -1 for unlimited
max_annotations_per_item: 3     # -1 for unlimited
 
# Optional: nested configuration
assignment:
  strategy: random
  max_annotations_per_item: 3
  random_seed: 1234

Legacy Configuration

The older automatic_assignment configuration is still supported:

yaml
automatic_assignment:
  on: true
  output_filename: task_assignment.json
  sampling_strategy: random    # 'random' or 'ordered'
  labels_per_instance: 3       # Annotations per item
  instance_per_annotator: 5    # Items per annotator
  test_question_per_annotator: 0

Test Questions

Insert attention check questions into the annotation queue:

Defining Test Questions

Add _testing to the instance ID in your data file:

csv
text,id
"This is test question 1",0_testing
"Regular item",dkjfd

Or in JSON:

json
[
  {"id": "0_testing", "text": "This is a test question"},
  {"id": "regular_001", "text": "Normal annotation item"}
]

Configuration

yaml
automatic_assignment:
  on: true
  test_question_per_annotator: 2  # Insert 2 test questions per annotator

Example Configurations

Basic Random Assignment

yaml
annotation_task_name: "Sentiment Analysis"
assignment_strategy: random
max_annotations_per_user: 20
max_annotations_per_item: 3

Quality-Focused Assignment

yaml
annotation_task_name: "Quality Annotation"
assignment_strategy: max_diversity
max_annotations_per_item: 5
max_annotations_per_user: 50

Crowdsourcing Setup

yaml
annotation_task_name: "Crowdsourced Task"
assignment_strategy: random
max_annotations_per_user: 10
max_annotations_per_item: 3
 
# Crowdsourcing settings
hide_navbar: true
jumping_to_id_disabled: true
 
login:
  type: url_direct
  url_argument: workerId

Active Learning Setup

yaml
assignment_strategy: active_learning
 
active_learning:
  enabled: true
  schema_names: ["sentiment", "topic"]
  min_annotations_per_instance: 2
  min_instances_for_training: 20
  update_frequency: 10
  classifier_name: "sklearn.linear_model.LogisticRegression"
  vectorizer_name: "sklearn.feature_extraction.text.TfidfVectorizer"

Admin Dashboard Integration

Monitor and adjust assignment settings through the Admin Dashboard:

  1. Navigate to /admin
  2. Go to the Configuration tab
  3. Modify:
    • Max Annotations per User
    • Max Annotations per Item
    • Assignment Strategy

Changes take effect immediately without server restart.

Further Reading

For implementation details, see the source documentation.

PreviousPasswordless LoginNextCategory Assignment