Skip to content

User Management

Manage annotators in Potato with user_config access lists, URL-direct login, RBAC roles and permissions, per-annotator workload quotas, and training gates.

Potato decides three separate things about a person: whether they may sign up, what they are allowed to do once in, and how much work they are given. Each is a different config block, and mixing them up is the usual cause of an annotator who can log in but never receives an item.

Access Modes

Open Access

Potato allows self-registration by default, so a new annotator creates their own account from the login page:

yaml
user_config:
  allow_all_users: true

Restricted Access

Set allow_all_users: false and list the addresses allowed to register. Nobody else can create an account:

yaml
user_config:
  allow_all_users: false
  users:
    - alice@example.com
    - bob@example.com
    - researcher@university.edu

Both keys live under user_config. Written at the top level they are silently ignored, which reads as an access list that does nothing.

Authentication

Password Login

The default. Annotators register with an email and a password, which Potato stores as a per-user salted PBKDF2 hash.

yaml
login:
  type: password

Direct Login with a URL Argument

url_direct takes the username straight from a URL parameter and asks for no password, which is how crowdsourcing arrivals are handled:

yaml
login:
  type: url_direct
  url_argument: PROLIFIC_PID

Annotators then arrive at http://your-server:8000/?PROLIFIC_PID=abc123 and are logged in as abc123.

url_argument also accepts a list, in which case the annotator's output folder is named by joining the values together:

yaml
login:
  type: url_direct
  url_argument:
    - PROLIFIC_PID
    - STUDY_ID
    - SESSION_ID

The three login types are password, url_direct, and prolific, the last adding Prolific API integration on top of URL-direct login. See Crowdsourcing Integration for the platform side.

Roles and Permissions

The Shared Admin Key

admin_api_key is a single shared key sent as the X-API-Key header, and it bypasses every permission check. When unset, Potato generates one and writes it to {task_dir}/admin_api_key.txt.

yaml
admin_api_key: "your-shared-key"

Role-Based Access Control

With an rbac block, named people reach the admin dashboard by logging in normally, without holding the shared key:

yaml
rbac:
  enabled: true
 
  # Custom roles, merged over the built-in ones
  roles:
    lead: [view_admin_dashboard, export_data, annotate]
 
  # Assign roles to specific users
  user_role_assignments:
    carol@example.com: admin
    dave@example.com: lead
 
  # Map an SSO claim to a role
  sso_role_mapping:
    "org:my-github-org": adjudicator

The permissions are view_admin_dashboard, manage_assignment, adjudicate, export_data, and annotate. The built-in admin role holds all five, adjudicator holds adjudicate, export_data and annotate, and annotator holds annotate.

A user's permissions are the union of every role they hold. With no rbac block at all, only the shared key and the adjudication.adjudicator_users list confer elevated access, which is how projects written before RBAC keep working.

Workload

Caps Per Annotator

max_annotations_per_user sets one number for everyone:

yaml
max_annotations_per_user: 100

per_annotator_quota sets different numbers for different people. It accepts default, by_user, and by_user_role, and rejects any other sub-key at load rather than ignoring it:

yaml
per_annotator_quota:
  default: 50
  by_user:
    alice@example.com: 200
  by_user_role:
    expert: 150
    novice: 40

A cap is resolved by looking at by_user first, then by_user_role, then default. The role labels come from user_roles:

yaml
user_roles:
  alice@example.com: expert
  bob@example.com: novice

A label here is only a workload label. It confers permissions only when it also names a role in rbac.roles, so expert and novice stay quota labels and nothing more.

Annotators Per Item

num_annotators_per_item is how many people see each item, which is what decides whether you can report agreement at the end:

yaml
num_annotators_per_item: 3
min_annotators_per_instance: 2

max_annotations_per_item is a deprecated alias for the same cap. Setting both to different values is refused at load rather than resolved silently.

Assignment Strategy

yaml
assignment_strategy: random
random_seed: 1234

The strategies are random, fixed_order, least_annotated, max_diversity, diversity_clustering, category_based, batch, priority, model_review, llm_confidence, psychometric, and active_learning. random_seed defaults to 1234, so fix it when a run has to be reproducible. See Task Assignment for what each strategy does.

Qualifying Annotators

A training phase puts practice items with known answers in front of annotators before the real task, and passing_criteria decides who continues:

yaml
training:
  enabled: true
  data_file: data/training.json
  passing_criteria:
    min_correct: 8
    max_mistakes: 2
    max_mistakes_per_question: 1
  allow_retry: true
  failure_action: block

See Training Phase for the item format and the feedback options.

Checking Attention During the Task

Training filters people at the start. Attention checks keep measuring once the real work is underway:

yaml
attention_checks:
  enabled: true
  items_file: data/attention_checks.json
  frequency: 10
  min_response_time: 3.0
  failure_handling:
    warn_threshold: 2
    warn_message: "Please read items carefully before answering."
    block_threshold: 5
    block_message: "You have been blocked due to too many incorrect responses."

See Quality Control for gold standards and agreement reporting.

Sessions

yaml
secret_key: "a-long-random-string"
session_timeout_minutes: 60
persist_sessions: true
session_lifetime_days: 7

secret_key signs the session cookie, and it can come from the POTATO_SECRET_KEY environment variable instead. Without one, Potato generates a new key at every startup, which logs everyone out on each restart.

session_timeout_minutes clears a session that has been idle that long, and the clock restarts on every request. Potato 2.11 is the first release that enforces it, with a default of 480 minutes. At 60, as in this example, an annotator who steps away for an hour has to log in again when they return.

Example: Research Team Setup

yaml
annotation_task_name: "Sentiment Annotation"
task_dir: .
 
user_config:
  allow_all_users: false
  users:
    - researcher1@university.edu
    - researcher2@university.edu
    - student1@university.edu
    - student2@university.edu
 
rbac:
  enabled: true
  user_role_assignments:
    researcher1@university.edu: admin
 
user_roles:
  researcher1@university.edu: expert
  student1@university.edu: novice
 
per_annotator_quota:
  default: 200
  by_user_role:
    expert: 300
    novice: 120
 
num_annotators_per_item: 2
assignment_strategy: least_annotated
 
data_files:
  - data/main.json
 
item_properties:
  id_key: id
  text_key: text
 
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What is the sentiment of this text?"
    labels:
      - Positive
      - Negative
      - Neutral
    sequential_key_binding: true
 
output_annotation_dir: annotation_output/

Example: Crowdsourcing Setup

yaml
annotation_task_name: "Sentiment Annotation Study"
task_dir: .
 
login:
  type: url_direct
  url_argument: PROLIFIC_PID
 
completion_code: "POTATO2024"
 
user_config:
  allow_all_users: true
 
max_annotations_per_user: 50
num_annotators_per_item: 3
assignment_strategy: random
 
hide_navbar: true
jumping_to_id_disabled: true
 
attention_checks:
  enabled: true
  items_file: data/attention_checks.json
  frequency: 10
  failure_handling:
    warn_threshold: 2
    block_threshold: 5
 
data_files:
  - data/main.json
 
item_properties:
  id_key: id
  text_key: text
 
output_annotation_dir: annotation_output/

Further Reading

For implementation details, see the source documentation and the roles and permissions reference.