Solo Mode: How One Annotator Can Label 10,000 Examples

You have 10,000 product reviews to label for sentiment (Positive, Neutral, Negative). Hiring three annotators to label all of them takes weeks and costs thousands of dollars. Solo Mode lets one domain expert get comparable quality by labeling only 500-1,000 instances; an LLM handles the rest, and the human reviews whatever the LLM is unsure about.

This tutorial walks through the whole process.

What you will need

• Potato 2.3.0 or later, with the AI extras for the LLM providers: pip install potato-annotation[ai]
• An OpenAI or Anthropic API key (for the LLM component)
• Your dataset in JSONL format
• One knowledgeable annotator (that could be you)

Step 1: Prepare your data

Create data/reviews.jsonl with one review per line:

{"id": "rev_001", "text": "Absolutely love this product! Best purchase I've made all year.", "source": "amazon"}
{"id": "rev_002", "text": "It works fine. Nothing special but gets the job done.", "source": "amazon"}
{"id": "rev_003", "text": "Broke after two weeks. Complete waste of money.", "source": "amazon"}
{"id": "rev_004", "text": "The quality is decent for the price point. I might buy again.", "source": "amazon"}
{"id": "rev_005", "text": "Arrived damaged and customer service was unhelpful.", "source": "amazon"}

For this tutorial, imagine this file contains 10,000 reviews.

Step 2: Create the configuration

Create config.yaml:

annotation_task_name: "Product Review Sentiment (Solo Mode)"
task_dir: "."
 
data_files:
  - "data/reviews.jsonl"
 
item_properties:
  id_key: id
  text_key: text
 
# --- Solo Mode Configuration ---
solo_mode:
  enabled: true
 
  # The model that labels. A list, so you can give fallbacks.
  labeling_models:
    - endpoint_type: openai
      model: "gpt-4o"
      api_key: ${OPENAI_API_KEY}
      temperature: 0.1
      max_tokens: 1024
 
  embedding:
    model_name: "all-MiniLM-L6-v2"
 
  uncertainty:
    strategy: direct_confidence
 
  # When you are allowed to stop labeling
  thresholds:
    end_human_annotation_agreement: 0.93
    minimum_validation_sample: 50
    confidence_low: 0.5
    confidence_high: 0.85
    periodic_review_interval: 100
 
  # What decides which instance you see next
  instance_selection:
    low_confidence_weight: 0.30
    disagreement_weight: 0.25
    diversity_weight: 0.20
    llm_predicted_weight: 0.15
    random_weight: 0.10
 
  batches:
    llm_labeling_batch: 25
    max_parallel_labels: 200
 
  labeling_functions:
    enabled: true
    auto_extract: true
    max_functions: 15
    min_confidence: 0.92
    min_coverage: 3
 
  edge_case_rules:
    enabled: true
    auto_extract_on_labeling: true
 
  confusion_analysis:
    enabled: true
    min_instances_for_pattern: 3
    max_patterns: 20
 
  prompt_optimization:
    enabled: true
    target_accuracy: 0.93
 
  refinement_loop:
    enabled: true
    trigger_interval: 50
    min_improvement: 0.02
    max_cycles: 3
 
  state_dir: "solo_state_reviews"
 
# --- Annotation Schema ---
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What is the overall sentiment of this review?"
    labels:
      - "Positive"
      - "Neutral"
      - "Negative"
    label_requirement:
      required: true
    sequential_key_binding: true
 
output_annotation_dir: "output/"
export_annotation_format:
  - jsonl
  - parquet

Step 3: Start the server

potato start config.yaml -p 8000

Open http://localhost:8000 and log in. The Solo Mode dashboard appears, showing the project in setup, ready for you to review the prompt it synthesised.

Step 4: Parallel annotation -- your first 50

Potato has picked 50 diverse reviews using embedding-based clustering. These are not random; they are chosen to cover your data distribution as broadly as possible.

Label each one. This is the phase that matters most, since the quality of your seed labels sets the ceiling for what the LLM can learn. Take your time and stay consistent.

Time estimate: 15-25 minutes at 20-30 seconds per instance.

Once enough of these are in, the LLM starts labeling alongside you without being asked.

Step 5: The LLM catches up

This phase runs on its own. Potato sends the LLM a batch of 200 instances with your 50 seed labels as few-shot examples, then compares the predictions against 10 held-out seed labels to estimate baseline accuracy.

A progress indicator shows up in the dashboard. This usually takes 1-2 minutes depending on the LLM provider.

Typical result: The LLM lands at 75-85% accuracy on the first calibration. That's expected. It hasn't learned your annotation style yet.

Step 6: Confusion analysis

Potato displays a confusion matrix showing where the LLM disagrees with your labels. A typical output:

Confusion Analysis (Round 1)
============================
Overall Accuracy: 0.82 (target: 0.93)

Top Confusion Pairs:
  Neutral -> Positive:  14 instances (7.0%)
  Negative -> Neutral:   9 instances (4.5%)
  Positive -> Neutral:   4 instances (2.0%)

This points to the LLM's main weakness here: it keeps upgrading neutral reviews to positive. That's a common one, since LLMs tend to lean positive.

Your action: Look through the confusion pairs. Click each pair to see the specific instances the LLM got wrong, which is the quickest way to understand how it fails.

Step 7: Back to prompt review

Based on the confusion analysis, Potato drafts refined guidelines for the LLM and shows them side by side: the current prompt on one side, and the specific edits it proposes from the error patterns on the other.

For example, Potato might suggest adding:

"Reviews that describe a product as 'fine', 'okay', or 'decent' without strong emotion should be labeled Neutral, even if they mention buying again."

Go through each suggested edit and approve, modify, or reject it. You can also write in your own clarifications.

Time estimate: 5-10 minutes.

Step 8: Labeling functions

Potato generates programmatic labeling functions from patterns in your seed labels. These are fast, deterministic rules that handle easy cases:

Generated Labeling Functions:
  LF1: Strong positive words (love, amazing, best, excellent)
       Precision: 0.97, Coverage: 0.06
  LF2: Strong negative words (terrible, awful, worst, waste)
       Precision: 0.95, Coverage: 0.04
  LF3: Exclamation + positive adjective
       Precision: 0.94, Coverage: 0.03
  LF4: Return/refund mention + negative context
       Precision: 0.92, Coverage: 0.02
  ...
  Total coverage: 0.18 (1,800 of 10,000 instances)

Labeling functions cover 18% of your dataset at 92%+ precision. Those instances get labeled automatically, which frees up the LLM and the human for the harder cases.

Your action: Look over the generated functions and disable any that seem unreliable. This step is optional, since Potato only keeps functions above your configured precision threshold anyway.

Step 9: Active annotation (125-375 instances)

This is where you do most of your labeling. Potato picks instances using the weighted pools set in instance_selection above:

• low_confidence_weight (0.30): Reviews where the LLM's confidence is lowest
• disagreement_weight (0.25): Reviews where the LLM and the labeling functions disagree
• diversity_weight (0.20): Reviews chosen to spread across the embedding space rather than cluster
• llm_predicted_weight (0.15): LLM-labeled reviews pulled back for human comparison
• random_weight (0.10): Random reviews for calibration

Two more weights exist and are left at their default of zero here, so they contribute nothing: edge_case_rule_weight, for instances matching a discovered edge-case rule, and cartography_weight, for instances whose confidence swings between rounds. Potato checks that the weights sum to 1.0 and logs a warning when they do not, but it uses the values as written, so make them add up yourself.

You label these in batches of 25. After each batch, Potato updates the LLM's accuracy estimate and decides whether to keep going.

Typical trajectory:

• Batch 1-3 (75 instances): Accuracy climbs from 82% to 87%
• Batch 4-6 (150 instances): Accuracy reaches 90%
• Batch 7-10 (250 instances): Accuracy plateaus at 91-92%

If agreement reaches your end_human_annotation_agreement, Solo Mode moves toward autonomous labeling. Otherwise it keeps cycling through review and refinement.

Time estimate: 45-90 minutes total, depending on how many batches are needed.

Step 10: The refinement loop

If accuracy is still under threshold after active labeling, Potato runs another round of the refinement loop:

1. The LLM re-labels the full dataset with updated guidelines and more few-shot examples
2. Potato recomputes accuracy against all human labels
3. It finds the new confusion patterns
4. It refines the guidelines again

This phase is mostly hands-off. You only need to approve the guideline changes.

Typical result: Accuracy improves by 2-4% per refinement round.

Step 11: Disagreement resolution

Potato presents the most contentious instances: cases where the LLM, labeling functions, and nearest-neighbor analysis all give different answers. For each instance, you see:

• The review text
• LLM prediction and confidence
• Labeling function votes
• 3 nearest labeled examples with their labels
• The LLM's chain-of-thought reasoning

These are the hard cases, and your labels here carry more weight than elsewhere in the process.

Time estimate: 20-30 minutes for 100-150 instances.

Step 12: Edge case synthesis

Potato generates synthetic reviews targeting the remaining confusion patterns. For example, if the LLM still struggles with "neutral reviews that mention buying again," it generates examples like:

"It's an okay product for the price. I might get another one if there's a sale."

You label these synthetic examples, and Potato adds them to the LLM's few-shot context.

Time estimate: 10-15 minutes for 30 examples.

Step 13: Periodic review

By now the LLM has labeled most of the dataset. Potato ranks all of its labels by confidence and sends you the lowest-confidence ones in batches of 25.

Confidence Escalation Progress:
  Batch 1: 25 instances, 23/25 correct (92%)
  Batch 2: 25 instances, 24/25 correct (96%)
  Batch 3: 25 instances, 25/25 correct (100%)
  -> Stopping: last 3 batches stable

Once the accuracy stops moving across three batches in a row, Solo Mode treats the remaining high-confidence labels as trustworthy. Note that this is stability, not perfection: the run above stops after 92%, 96% and 100%, because the number has settled, not because the last batch was clean.

Time estimate: 15-20 minutes.

Step 14: Prompt optimization

This phase runs on its own. Potato tries 8 prompt variants and keeps the one with the highest F1 score on your accumulated human labels:

Prompt Optimization Results:
  Variant 1 (direct, 5 examples):     F1=0.91
  Variant 2 (CoT, 5 examples):        F1=0.93
  Variant 3 (direct, 10 examples):    F1=0.92
  Variant 4 (CoT, 10 examples):       F1=0.94  <-- selected
  Variant 5 (direct, 15 examples):    F1=0.92
  Variant 6 (CoT, 15 examples):       F1=0.93
  Variant 7 (self-consistency, 5x):   F1=0.94
  Variant 8 (self-consistency, 10x):  F1=0.94

It then uses the best prompt for a final re-labeling pass.

Step 15: Final validation

Potato pulls 100 random LLM-labeled instances for you to review. You label them, and Potato compares your labels against the LLM's.

Final Validation:
  Reviewed: 100 instances
  LLM correct: 94/100 (94%)
  Threshold: 93%
  -> PASSED

If the LLM clears your threshold, the project is complete. If not, it drops back to active annotation for another round -- that fallback is the one transition final validation is allowed.

Time estimate: 10-15 minutes.

Results Summary

At the end, the dashboard reports where the labels came from. Solo Mode has no command-line interface -- it runs inside the server and is driven from its own pages -- so this is read from the running instance rather than a terminal:

Solo Mode Complete
==================
Dataset: 10,000 instances
Total human labels: 612
  Seed: 50
  Active labeling: 275
  Disagreement exploration: 137
  Edge case synthesis: 30
  Confidence escalation: 75
  Final validation: 45

LLM labels: 8,200 (accuracy: 94.1%)
LF labels: 1,800 (precision: 95.3%)
Unlabeled: 0

Final label distribution:
  Positive: 4,823 (48.2%)
  Neutral:  3,011 (30.1%)
  Negative: 2,166 (21.7%)

Total human time: ~3.5 hours
Estimated multi-annotator cost (3x): ~$4,500
Solo Mode cost: ~$450 (API fees) + ~$175 (annotator time)
Savings: ~88%

The human labeled 612 of 10,000 instances, about 6%. The LLM and labeling functions handled the rest at 94%+ accuracy.

Exporting Results

Solo Mode writes through the same storage as any other project, so the labels come out through the ordinary exporter:

python -m potato.export -c config.yaml -f jsonl -o ./export/

Each line includes the label and its source:

{"id": "rev_001", "sentiment": "Positive", "source": "human", "confidence": 1.0}
{"id": "rev_002", "sentiment": "Neutral", "source": "llm", "confidence": 0.91}
{"id": "rev_003", "sentiment": "Negative", "source": "labeling_function", "confidence": 0.97}

For Parquet export:

import pandas as pd
df = pd.read_parquet("output/exports/parquet/annotations.parquet")
print(df["sentiment"].value_counts())

annotations.parquet is one row per instance and annotator, with each annotation scheme flattened into its own column, so you select the scheme by name rather than a generic value column.

Quality Assurance: Hybrid Verification

For publication-quality datasets, add a second annotator to review a sample. The Solo Mode source documentation describes the verification options in more detail.

Solo Mode has no verification block of its own, so this is an ordinary second pass: export the labels, run them back through Potato as a normal project with one reviewer over a 10% sample, and compute agreement between the reviewer and the Solo Mode labels the way you would for any two annotators. (icl_labeling does have a verification block, but it belongs to in-context labeling, not to Solo Mode.)

Troubleshooting

LLM accuracy plateaus below threshold

• Increase seed count: Try 75-100 seed instances instead of 50
• Switch LLM: Try claude-sonnet-4-20250514 instead of GPT-4o (or vice versa)
• Lower the threshold: If 93% is not achievable, consider whether 90% is acceptable for your use case
• Check your data: Some datasets are inherently ambiguous. If human-human agreement would only be 90%, do not expect the LLM to do better

Active annotation takes too many batches

• Increase batch size: Change batch_size from 25 to 50
• Adjust pool weights: If most escalated instances come from the low-confidence pool, reduce low_confidence_weight and raise disagreement_weight or diversity_weight

Labeling functions have low coverage

• This is normal for tasks without strong lexical signals (e.g., sarcasm detection, implicit sentiment)
• Labeling functions work best for explicit, keyword-driven patterns
• Solo Mode still works without labeling functions; the LLM covers those instances instead

Further Reading

• Solo Mode Documentation -- full configuration reference
• Active Learning -- the underlying selection algorithm
• AI Support -- LLM provider configuration
• Quality Control -- additional quality assurance options
• Parquet Export -- efficient data export