Skip to content

Task Assignment

Control instance distribution in Potato — sequential, random, uniform overlap, and per-annotator custom assignments to manage workload and coverage precisely.

Task Assignment

Potato provides flexible task assignment strategies to control how annotation instances are distributed to annotators.

Overview

Task assignment controls:

  • Which items each annotator sees
  • How many items each annotator completes
  • How many annotations each item receives
  • The order in which items are presented

Key Configuration Options

OptionDescriptionDefault
assignment_strategyStrategy for assigning itemsrandom
max_annotations_per_userMaximum items per annotatorunlimited
max_annotations_per_itemTarget annotations per item3

Assignment Strategies

Random Assignment

Assigns items randomly to annotators, ensuring unbiased distribution.

yaml
assignment_strategy: random
max_annotations_per_item: 3

Best for: General annotation tasks where order doesn't matter.

Fixed Order Assignment

Assigns items in the order they appear in the dataset.

yaml
assignment_strategy: fixed_order
max_annotations_per_item: 2

Best for: Tasks where annotators should see items in a specific sequence.

Least-Annotated Assignment

Prioritizes items with the fewest existing annotations, ensuring even distribution.

yaml
assignment_strategy: least_annotated
max_annotations_per_item: 5

Best for: Ensuring all items receive adequate coverage before any item gets excessive annotations.

Max-Diversity Assignment

Prioritizes items with the highest disagreement among existing annotations.

yaml
assignment_strategy: max_diversity
max_annotations_per_item: 4

Best for: Quality control and resolving ambiguous items.

Active Learning Assignment

Uses machine learning to prioritize uncertain instances.

yaml
assignment_strategy: active_learning
 
active_learning:
  enabled: true
  schema_names: ["sentiment"]
  min_annotations_per_instance: 2
  min_instances_for_training: 20
  update_frequency: 10

See Active Learning for full configuration.

Configuration

Modern Configuration (Recommended)

yaml
# Strategy selection
assignment_strategy: random
 
# Limits
max_annotations_per_user: 10    # -1 for unlimited
max_annotations_per_item: 3     # -1 for unlimited
 
# Optional: nested configuration
assignment:
  strategy: random
  max_annotations_per_item: 3
  random_seed: 1234

Legacy Configuration

The older automatic_assignment configuration is still supported:

yaml
automatic_assignment:
  on: true
  output_filename: task_assignment.json
  sampling_strategy: random    # 'random' or 'ordered'
  labels_per_instance: 3       # Annotations per item
  instance_per_annotator: 5    # Items per annotator
  test_question_per_annotator: 0

Test Questions

Insert attention check questions into the annotation queue:

Defining Test Questions

Add _testing to the instance ID in your data file:

csv
text,id
"This is test question 1",0_testing
"Regular item",dkjfd

Or in JSON:

json
[
  {"id": "0_testing", "text": "This is a test question"},
  {"id": "regular_001", "text": "Normal annotation item"}
]

Configuration

yaml
automatic_assignment:
  on: true
  test_question_per_annotator: 2  # Insert 2 test questions per annotator

Example Configurations

Basic Random Assignment

yaml
annotation_task_name: "Sentiment Analysis"
assignment_strategy: random
max_annotations_per_user: 20
max_annotations_per_item: 3

Quality-Focused Assignment

yaml
annotation_task_name: "Quality Annotation"
assignment_strategy: max_diversity
max_annotations_per_item: 5
max_annotations_per_user: 50

Crowdsourcing Setup

yaml
annotation_task_name: "Crowdsourced Task"
assignment_strategy: random
max_annotations_per_user: 10
max_annotations_per_item: 3
 
# Crowdsourcing settings
hide_navbar: true
jumping_to_id_disabled: true
 
login:
  type: url_direct
  url_argument: workerId

Active Learning Setup

yaml
assignment_strategy: active_learning
 
active_learning:
  enabled: true
  schema_names: ["sentiment", "topic"]
  min_annotations_per_instance: 2
  min_instances_for_training: 20
  update_frequency: 10
  classifier_name: "sklearn.linear_model.LogisticRegression"
  vectorizer_name: "sklearn.feature_extraction.text.TfidfVectorizer"

Admin Dashboard Integration

Monitor and adjust assignment settings through the Admin Dashboard:

  1. Navigate to /admin
  2. Go to the Configuration tab
  3. Modify:
    • Max Annotations per User
    • Max Annotations per Item
    • Assignment Strategy

Changes take effect immediately without server restart.

Further Reading

For implementation details, see the source documentation.