Admin Dashboard

Potato 2.0 includes an admin dashboard for monitoring annotation progress, tracking annotator performance, and managing task configuration in real time.

Accessing the Dashboard

API Key Authentication

The dashboard is gated by a shared key sent as the X-API-Key header, so set one before you expose the server. Configure it in your YAML:

admin_api_key: "your-secure-api-key-here"

Potato reads the key from three places, in priority order: the admin_api_key config key, the POTATO_ADMIN_API_KEY environment variable, and an auto-generated key. When you set none of them, Potato generates a random key at startup, logs it to the console, and persists it to {task_dir}/admin_api_key.txt so it survives a restart.

Access the dashboard at /admin and enter your API key when prompted.

Authentication Methods

1. Direct navigation - Go to /admin and enter the API key
2. Header-based - Include X-API-Key header in requests
3. Debug mode - With debug: true, requests from the server's own machine open the admin pages without a key

Debug mode opens the admin pages without a key only for requests from the server's own machine, meaning a loopback address with no X-Forwarded-For, X-Real-IP or Forwarded header. A tunnel or reverse proxy on the same machine that adds no forwarding header makes every caller look local, so never set debug on a deployed server:

# Development only
debug: true  # Admin pages open without a key for requests from this machine

Dashboard Tabs

Overview Tab

High-level metrics for your annotation task:

• Total Users - Number of registered annotators
• Active Users - Annotators currently in the annotation phase
• Total Annotations - Total annotations completed
• Completion Percentage - Percentage of items with annotations
• Total Items - Items in the dataset
• Working Time - Total time spent by all annotators
• Task Configuration - Task name, assignment limits, strategy, and debug status

Annotators Tab

Detailed information about each annotator:

The tab renders the full table with a Refresh Data button. Sorting and filtering controls live on the Instances tab, not here.

Instances Tab

Browse and manage annotation instances:

Features:

• Pagination (25, 50, or 100 per page)
• Sort by annotation count, completion percentage, disagreement, ID, or average time
• Sort ascending or descending
• Filter to all instances, completed only, or incomplete only

Beyond these three, the dashboard also carries Questions, Behavioral Analytics, Crowdsourcing, BWS Scoring, MACE, Embedding Visualization, and Devices tabs.

Configuration Tab

Modify settings in real time without restarting:

• Max Annotations per User - Limit how many items each user can annotate (-1 for unlimited)
• Max Annotations per Item - Cap annotations per instance (-1 for unlimited)
• Assignment Strategy - How instances are assigned to users

Changes apply to the running server immediately. They are held in memory and are not written back to your config file, so reapply them in the YAML if you want them to survive a restart.

Assignment Strategies

The Configuration tab exposes six strategies:

The config key is assignment_strategy at the top level. Potato accepts twelve values in total, including category_based, diversity_clustering, batch, priority, psychometric, and model_review, which you set in the YAML rather than from the dashboard.

Timing Analytics

The dashboard tracks detailed timing data:

• Total Working Time - Sum of all annotation sessions
• Average Time per Annotation - Mean time per item
• Annotations per Hour - Throughput rate

Timing data is derived from behavioral tracking and page interaction events.

Configuration

Basic Setup

One key turns the dashboard on, because the key is what the login form checks:

# Enable admin dashboard
admin_api_key: "your-secure-api-key"

Full Configuration

Potato has no admin configuration block. The settings that shape the admin surface are top-level keys, and the three assignment keys are the ones the Configuration tab edits at runtime:

admin_api_key: "your-secure-api-key"
 
# How long a session may sit idle before the server clears it.
# The clock restarts on every request, so it bounds inactivity,
# not the length of a shift. Default 480.
session_timeout_minutes: 480
 
# Editable from the Configuration tab while the server runs
assignment_strategy: least_annotated
max_annotations_per_user: 100
max_annotations_per_item: 3

The dashboard has no refresh interval, page-size, or export-size settings. Page size is a per-session dropdown on the Instances tab, and the tables refresh when you press their Refresh buttons.

API Endpoints

The admin dashboard is powered by REST API endpoints. Every one of them takes the X-API-Key header. Note the path order is /admin/api/..., not /api/admin/....

Stats

GET /admin/api/overview

Returns user statistics, annotation progress, and a configuration summary.

Annotators

GET /admin/api/annotators
GET /admin/user_state/<user_id>

List all annotators with timing and throughput metrics, or get detailed state for one user.

Instances

GET /admin/api/instances?page=1&page_size=25&sort_by=annotation_count&sort_order=desc&filter_completion=
GET /admin/item_state/<item_id>

List instances with pagination, sorting and filtering, or get detailed state for one item. sort_by takes annotation_count, completion_percentage, disagreement, id, or average_time; filter_completion takes completed, incomplete, or an empty value for all.

Configuration

GET /admin/api/config
POST /admin/api/config

Get or update task configuration. The POST body accepts max_annotations_per_user, max_annotations_per_item, and assignment_strategy, and rejects anything else.

Training Progress

GET /admin/training/api/status
GET /admin/training/api/runs
GET /admin/training/api/runs/<run_id>

Model training status and run history, including the held-out score and the deployed precision from review verdicts. The training routes cover classifier training runs. Potato exposes no admin API for per-annotator progress through the annotator training phase.

Active Learning

GET /admin/active-learning/stats

Which schemas have a fitted model, where the schema cycler is, and the recorded training runs.

Monitoring Features

Suspicious Activity Detection

The dashboard computes a suspicion score per annotator from three process signals:

• Fast annotation rate, the share of annotations completed in under two seconds
• Low interaction rate, the share of instances with minimal interaction
• No-change rate, the share of instances where the initial selection was never changed

Annotators scoring above 50% are highlighted. The score ranks annotators for review and does not classify anyone, since fast work has innocent explanations.

Quality Metrics

Track annotation quality indicators:

• Inter-annotator agreement, via GET /admin/api/agreement
• Time-per-annotation distribution
• Label distribution balance, on the Questions tab
• AI acceptance rate, where AI assistance is enabled

Security Best Practices

API Key Management

Keeping the key out of the config file keeps it out of version control, and Potato substitutes ${VAR} references for admin_api_key and secret_key:

# Use environment variables
admin_api_key: ${ADMIN_API_KEY}

export ADMIN_API_KEY="your-secure-key"

Potato also reads POTATO_ADMIN_API_KEY directly, so exporting that variable works without a config entry.

Access Control

• Use strong, unique API keys
• Rotate keys periodically
• Don't expose admin endpoints publicly
• Use HTTPS in production

Audit Logging

Potato keeps no admin audit log. Configuration changes, exports, and access attempts are not recorded to a durable trail, so treat the API key as the whole of the access control and restrict who holds it. The one audited surface is the MCP server, which writes each authorized call to the file named by mcp.audit_log.

Example Configuration

annotation_task_name: "Sentiment Analysis"
task_dir: "."
port: 8000
 
# Admin dashboard
admin_api_key: ${ADMIN_API_KEY}
session_timeout_minutes: 480
 
# Data and annotation configuration
data_files:
  - "data/reviews.json"
 
item_properties:
  id_key: id
  text_key: text
 
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "How does this review read?"
    labels:
      - Positive
      - Negative
      - Neutral
 
# Assignment settings (editable from the Configuration tab)
assignment_strategy: least_annotated
max_annotations_per_user: 100
max_annotations_per_item: 3
 
output_annotation_dir: "output/"
export_annotation_format: jsonl
 
user_config:
  allow_all_users: true

Three annotations per item is the smallest number that lets a majority break a tie, and it is what the agreement statistics on the dashboard need to report anything useful.

Performance Considerations

Large Datasets

For datasets with thousands of instances:

• Use the Instances tab pagination rather than the 100-per-page view
• Load items lazily with partial_loading instead of reading every file at startup
• Consider a database backend for better performance

Many Annotators

For tasks with many concurrent annotators:

• Monitor server resources
• Use a database backend for scalability
• Cap max_annotations_per_user so no annotator drains the queue

# For large-scale deployments
database:
  type: mysql
  host: localhost
  database: potato_db
  username: ${DB_USER}
  password: ${DB_PASSWORD}

Troubleshooting

Can't Access Dashboard

• Verify the API key is set in config, in POTATO_ADMIN_API_KEY, or read it from {task_dir}/admin_api_key.txt
• Check the URL is /admin
• Ensure the API key matches exactly
• Check for typos in configuration

Slow Dashboard Loading

• Reduce the Instances page size
• Load items lazily with partial_loading
• Consider a database backend
• Check server resources

Stats Not Updating

The dashboard polls nothing on its own, so the numbers you see are from the last load.

• Press the Refresh button on the tab you are reading
• Reload the page
• Check for JavaScript errors in the browser console

Further Reading

• Behavioral Tracking - Detailed interaction tracking
• Quality Control - Attention checks and gold standards
• API Overview - Admin API endpoints

For implementation details, see the source documentation.