Skip to content

Solo Mode

Run a complete annotation pipeline solo in Potato — one expert and an LLM working through a phase state machine that covers prompting, parallel annotation, disagreement resolution and final validation.

New in v2.3.0

Traditional annotation projects require multiple annotators, inter-annotator agreement computation, adjudication rounds, and significant coordination overhead. For many research teams, this is the primary bottleneck: not the annotation interface, but the logistics of hiring, training, and managing a team.

Solo Mode replaces the multi-annotator paradigm with a single human expert collaborating with an LLM. The human provides high-quality labels on a small, strategically selected subset. The LLM learns from those labels, proposes labels for the rest, and the human reviews only the cases where the LLM is uncertain or likely wrong. A phase state machine orchestrates this, advancing on the thresholds you set and looping back to the prompt whenever the prompt turns out to be what is wrong.

In internal benchmarks, Solo Mode achieved 95%+ agreement with full multi-annotator pipelines while requiring only 10-15% of the total human labels.

The Workflow

Solo Mode is a state machine. SoloPhaseController tracks which phase a project is in and which phases it is allowed to move to next, so the route is not a straight line: several phases can loop back when the prompt still needs work.

Setup

Task description and data. Generate prompt asks a revision model (revision_models) to write the first prompt from the description and the annotation schemes. Without a revision model, or when it returns nothing, the description itself becomes the first prompt and the prompt history says so. The only phase setup can leave to is prompt review.

Prompt review

Potato synthesises a prompt from the task description and you edit it. Several later phases return here when the prompt turns out to be the problem, which makes this the hub of the whole workflow rather than a one-time step.

From here the project either goes through the edge-case loop or straight to parallel annotation.

Edge case synthesis, then edge case labeling

The LLM generates boundary examples and you label them. Labeling can return to prompt review if the examples show the prompt is wrong.

Prompt validation

Your labels on the edge cases are checked against what the prompt produces. It either proceeds to parallel annotation or goes back to prompt review.

Parallel annotation

You and the LLM annotate the same data at the same time. This is the phase that produces the agreement figure everything else keys off.

Disagreement resolution

Where you and the LLM differ, you settle it. It returns to parallel annotation, or to prompt review when the disagreements point at the prompt rather than at individual items.

Active annotation

The main annotation phase, with instances chosen by the weighted pools in instance_selection. From here Potato can go to periodic review, rule review, or on to autonomous labeling.

Periodic review

Low-confidence LLM labels, surfaced every thresholds.periodic_review_interval items. Returns to active annotation, moves to rule review, or falls back to prompt review.

Rule review

The edge case rules Potato extracted from your labeling, presented for approval. Returns to active annotation or to prompt review.

Autonomous labeling

The LLM labels what is left. Its only exit is final validation.

Final validation

You check a sample of the LLM-only labels. Clearing thresholds.end_human_annotation_agreement completes the project; failing it drops back to active annotation.

Completed

Terminal state.

Configuration

Quick Start

A minimal Solo Mode configuration:

yaml
annotation_task_name: "Sentiment Classification"
task_dir: "."
 
data_files:
  - "data/reviews.jsonl"
 
item_properties:
  id_key: id
  text_key: text
 
solo_mode:
  enabled: true
 
  # The models that label. A list: the first is used, the rest are fallbacks.
  labeling_models:
    - endpoint_type: openai
      model: "gpt-4o"
      api_key: ${OPENAI_API_KEY}
      temperature: 0.1
      max_tokens: 1024
 
  # When the project may stop asking the human
  thresholds:
    end_human_annotation_agreement: 0.92
    minimum_validation_sample: 50
    confidence_low: 0.5
 
  state_dir: "solo_state_sentiment"
 
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What is the sentiment of this text?"
    labels:
      - Positive
      - Neutral
      - Negative
 
output_annotation_dir: "output/"
export_annotation_format: "jsonl"

Full Configuration Reference

yaml
solo_mode:
  enabled: true
 
  # Models that label, and models that revise. Both are lists; revision_models
  # defaults to labeling_models when omitted.
  labeling_models:
    - endpoint_type: openai       # openai, anthropic, ollama, vllm, ...
      model: "gpt-4o"
      api_key: ${OPENAI_API_KEY}
      temperature: 0.1
      max_tokens: 1024
  revision_models:
    - endpoint_type: anthropic
      model: "claude-sonnet-4-20250514"
      api_key: ${ANTHROPIC_API_KEY}
      max_tokens: 1024
 
  # Sentence embeddings, used for diversity and neighbour lookups
  embedding:
    model_name: "all-MiniLM-L6-v2"
 
  # How the LLM's uncertainty is measured
  uncertainty:
    strategy: direct_confidence
    sampling_diversity:
      num_samples: 5
      temperature: 1.0
 
  # When the human can stop, and what counts as agreement per scheme type
  thresholds:
    end_human_annotation_agreement: 0.90
    minimum_validation_sample: 50
    confidence_low: 0.5
    confidence_high: 0.8
    periodic_review_interval: 100
    likert_tolerance: 1
    multiselect_jaccard_threshold: 0.5
    textbox_embedding_threshold: 0.7
    span_overlap_threshold: 0.5
 
  # Weights for the pools that choose what the human sees next
  instance_selection:
    low_confidence_weight: 0.3
    disagreement_weight: 0.1
    diversity_weight: 0.2
    llm_predicted_weight: 0.3
    random_weight: 0.1
    cartography_weight: 0.0
    edge_case_rule_weight: 0.0
 
  batches:
    llm_labeling_batch: 50
    max_parallel_labels: 200
 
  labeling_functions:
    enabled: true
    auto_extract: true
    max_functions: 50
    min_confidence: 0.85
    min_coverage: 3               # instances, not a fraction
    vote_threshold: 0.5
 
  edge_case_rules:
    enabled: true
    auto_extract_on_labeling: true
    confidence_threshold: 0.75
    min_rules_for_clustering: 10
    target_cluster_size: 15
    reannotation_enabled: true
    reannotation_confidence_threshold: 0.60
    max_reannotations_per_instance: 2
 
  confusion_analysis:
    enabled: true
    min_instances_for_pattern: 3
    max_patterns: 20
    auto_suggest_guidelines: false
 
  prompt_optimization:
    enabled: true
    find_smallest_model: true
    target_accuracy: 0.85
    optimization_interval_seconds: 300
    accuracy_weight: 0.7
    length_weight: 0.2
 
  # Send cheap cases to a small model and hard ones to a larger one
  confidence_routing:
    enabled: false
    tiers:
      - name: cheap
        confidence_threshold: 0.8
        model:
          endpoint_type: ollama
          model: "llama3.2"
 
  refinement_loop:
    enabled: true
    trigger_interval: 50
    min_improvement: 0.02
    max_cycles: 5
    patience: 2
 
  state_dir: "solo_state_sentiment"

Key Capabilities

Confusion Analysis

After each labeling round, Potato builds a confusion matrix between human and LLM labels. The admin dashboard shows:

  • Per-class precision, recall, and F1 from the LLM's perspective
  • Most common confusion pairs (e.g., "neutral misclassified as positive: 23 instances")
  • Example instances for each confusion pair
  • Trend charts showing improvement across refinement rounds

Solo Mode has no command-line interface. It runs inside the server and is driven from its own pages under the running instance, so the analysis is read from the dashboard rather than a CLI. What it reports looks like this:

text
Confusion Analysis (Round 2)
============================
Overall Accuracy: 0.87 (target: 0.92)

Top Confusion Pairs:
  neutral -> positive:  23 instances (15.3%)
  negative -> neutral:  11 instances (7.3%)
  positive -> neutral:   5 instances (3.3%)

Per-Class Performance:
  Positive:  P=0.91  R=0.94  F1=0.92
  Neutral:   P=0.78  R=0.71  F1=0.74
  Negative:  P=0.93  R=0.88  F1=0.90

Automated Refinement Loop

The refinement loop iterates between LLM labeling, confusion analysis, and guideline updates. Each iteration:

  1. LLM labels the full dataset with current guidelines
  2. Potato compares against all available human labels
  3. If accuracy is below threshold, confusion analysis runs
  4. LLM proposes guideline edits based on error patterns
  5. Human reviews and approves edits
  6. Cycle repeats (up to max_cycles)
yaml
solo_mode:
  revision_models:
    - endpoint_type: anthropic
      model: "claude-sonnet-4-20250514"
      api_key: ${ANTHROPIC_API_KEY}
 
  refinement_loop:
    enabled: true
    trigger_interval: 50           # run a cycle every 50 new human labels
    min_improvement: 0.02          # stop if a cycle gains less than 2%
    max_cycles: 5
    patience: 2                    # cycles below min_improvement before stopping

Labeling Functions (ALCHEmist-Inspired)

Potato generates lightweight labeling functions from patterns observed in human annotations. These are not LLM calls; they are fast, deterministic rules.

Example generated labeling functions:

python
# Auto-generated labeling function 1
# Precision: 0.96, Coverage: 0.08
def lf_strong_positive_words(text):
    positive = {"excellent", "amazing", "fantastic", "outstanding", "perfect"}
    if any(w in text.lower() for w in positive):
        if not any(neg in text.lower() for neg in {"not", "never", "no"}):
            return "Positive"
    return None  # abstain
 
# Auto-generated labeling function 2
# Precision: 0.93, Coverage: 0.05
def lf_explicit_negative(text):
    negative = {"terrible", "awful", "horrible", "worst", "disgusting"}
    if any(w in text.lower() for w in negative):
        return "Negative"
    return None

Configure labeling function behavior:

yaml
solo_mode:
  labeling_functions:
    enabled: true
    auto_extract: true
    max_functions: 50
    min_confidence: 0.85
    min_coverage: 3          # minimum instances a function must cover
    vote_threshold: 0.5      # agreement needed when functions disagree

Disagreement Explorer

The disagreement explorer presents instances where different signals conflict. For each instance, the annotator sees:

  • The LLM's predicted label and confidence
  • Labeling function votes (if any)
  • Nearest labeled neighbors in embedding space
  • The raw text/content

This is the highest-value annotation activity: each label resolves a genuine ambiguity.

yaml
solo_mode:
  # Disagreement is a phase, not a configurable block. What you control is how
  # heavily disagreement weighs when Potato picks the next instance.
  instance_selection:
    disagreement_weight: 0.3
    low_confidence_weight: 0.3
    diversity_weight: 0.2
    llm_predicted_weight: 0.1
    random_weight: 0.1

Cascaded Confidence Escalation

After the bulk of the dataset is labeled by the LLM, Potato ranks all LLM-labeled instances by confidence and escalates the least confident ones to the human. This continues in batches until quality stabilizes.

yaml
solo_mode:
  thresholds:
    confidence_low: 0.5           # below this, the human sees it
    confidence_high: 0.8          # above this, the LLM's label stands
    periodic_review_interval: 100 # surface low-confidence labels this often
 
  batches:
    llm_labeling_batch: 50
    max_parallel_labels: 200

Decision Models

From Potato 2.11, a decision model can label in Solo Mode. A decision model answers a fixed-choice question in one forward pass and returns a probability for every option, which makes it faster and cheaper than an LLM at labeling, and it cannot write prompts, rules or explanations. Two endpoint types reach one. laya runs Laya inside the Potato process on CPU and needs pip install 'potato-annotation[decision]', and systemone talks to any server with the /v1/systemone API, such as hosted Jev or OpenJev.

The usual place for a decision model is the first tier of confidence routing, where it labels everything and only the items it is unsure about go on to the LLM:

yaml
solo_mode:
  labeling_models:
    - endpoint_type: laya
      model: convaiinnovations/laya
  revision_models:                     # required: decision models cannot write text
    - endpoint_type: vllm
      model: google/gemma-4-12B-it-qat-w4a16-ct
      base_url: http://localhost:8001/v1
  confidence_routing:
    enabled: true
    tiers:
      - name: laya
        confidence_threshold: 0.9
        model:
          endpoint_type: laya
          model: convaiinnovations/laya
          device: cpu
      - name: llm
        confidence_threshold: 0.5
        model:
          endpoint_type: vllm
          model: google/gemma-4-12B-it-qat-w4a16-ct
          base_url: http://localhost:8001/v1

revision_models must be a generative model. It defaults to labeling_models, so a study whose only model is a decision model has to set it. Decision models label radio, select, likert and slider schemes, and Solo Mode refuses to start if one is set to label multiselect, span or free-text schemes. On rating scales, Laya's scores can barely follow the scale, so check a few items with Try on 10 items before routing a likert or slider scheme through it. The confidence is the probability the model gave its chosen option, and its calibration varies by domain, so set the tier threshold from your own data. config-decision-model.yaml is a runnable version.

Blind Suggestions

By default the annotate page shows the LLM's label, its confidence and an Accept button before you answer, which is fast and also pulls your answer toward the suggestion. Solo Mode hands the rest of the corpus to the LLM when your agreement with it crosses end_human_annotation_agreement, so agreement measured on labels you gave after seeing the suggestion can end your part of the work too early. From Potato 2.11, solo_mode.suggestions controls when the suggestion appears:

yaml
solo_mode:
  suggestions:
    display: blind_sample      # before_label | after_label | blind_sample
    blind_rate: 0.2
    minimum_blind_sample: 20

blind_sample hides the suggestion on a blind_rate fraction of items, and after_label never shows it first. Once there are minimum_blind_sample blind comparisons, the handoff compares blind agreement with end_human_annotation_agreement. The dashboard's Blind Agreement card shows blind and shown agreement side by side, and a large positive gap means you agree with the LLM more often when you can see its answer.

Potato 2.11 also has Solo Mode check the project's spend cap (ai_budget.cap_usd) before labeling starts and before each batch, and adds a response cache (solo_mode.cache), a gold set with prompt trials (solo_mode.gold), a prompt-sensitivity check, a labeling jury (solo_mode.jury) and curriculum ordering (solo_mode.instance_selection.curriculum). Solo Mode Advanced Features documents each one. When the labeling engine is on, Solo Mode also writes its model predictions into the engine's store, so the engine can calibrate and certify them.

Multi-Signal Instance Prioritization

Across all phases that involve human labeling, Potato uses a weighted pool system to select the most informative instances. Seven pools feed into a unified priority queue:

yaml
solo_mode:
  instance_selection:
    low_confidence_weight: 0.3
    diversity_weight: 0.2
    llm_predicted_weight: 0.3
    disagreement_weight: 0.1
    random_weight: 0.1
    cartography_weight: 0.0
    edge_case_rule_weight: 0.0
  • low_confidence: the LLM's confidence falls below thresholds.confidence_low
  • diversity: far from what you have already labeled, in embedding space
  • llm_predicted: the LLM has a label and it has not been checked
  • disagreement: the signals available for the instance do not agree
  • random: a small random draw, to keep the estimate honest and catch blind spots
  • cartography: training-dynamics signal, off by default
  • edge_case_rule: matches a rule extracted from your earlier labeling, off by default

Edge Case Synthesis

Potato uses the LLM to generate synthetic examples that target known weaknesses:

yaml
solo_mode:
  edge_case_rules:
    enabled: true
    auto_extract_on_labeling: true   # derive rules as you label
    confidence_threshold: 0.75
    min_rules_for_clustering: 10
    target_cluster_size: 15
    reannotation_enabled: true       # revisit items a new rule contradicts
    reannotation_confidence_threshold: 0.60
    max_reannotations_per_instance: 2

The LLM generates examples that are ambiguous between the specified label pairs. The human labels them, and these labels are added to the few-shot context for subsequent LLM labeling rounds.

Prompt Optimization (DSPy-Inspired)

Potato runs automated prompt optimization to find the best instruction format for the LLM, and will look for the smallest model that still holds the target accuracy:

yaml
solo_mode:
  prompt_optimization:
    enabled: true
    find_smallest_model: true          # prefer the cheapest model that holds up
    target_accuracy: 0.85
    optimization_interval_seconds: 300
    accuracy_weight: 0.7               # how the candidates are scored
    length_weight: 0.2

Monitoring Progress

The admin dashboard shows Solo Mode progress in real time:

  • Current phase and progress within each phase
  • Human labels completed vs. total budget
  • LLM accuracy over time (per round)
  • Labeling function coverage and precision
  • Confidence distribution histogram
  • Estimated time to completion

There is no potato.solo module and no Solo Mode CLI: the whole workflow lives behind the server's own routes, so progress is read from the dashboard. It reports:

text
Solo Mode Status
================
Current Phase: 6 (Active Labeling) - Batch 3/10
Human Labels: 142 / ~300 estimated total
LLM Accuracy: 0.89 (target: 0.92)
LF Coverage: 0.23 (labeling functions cover 23% of data)
Dataset Size: 10,000 instances
  - Human labeled: 142
  - LF labeled: 2,300
  - LLM labeled: 7,558
  - Unlabeled: 0

When to Use Solo Mode vs. Traditional Multi-Annotator

Use Solo Mode when:

  • You have a domain expert who can provide high-quality labels
  • Budget or logistics prevent hiring multiple annotators
  • The task has clear, well-defined categories
  • You need to label a large dataset (1,000+ instances)
  • Speed matters more than measuring inter-annotator agreement

Use traditional multi-annotator when:

  • You need inter-annotator agreement statistics for publication
  • The task is highly subjective (e.g., offensiveness, humor)
  • You need to study annotator disagreement patterns
  • Regulatory requirements mandate multiple independent annotators
  • The label space is complex or evolving (annotation guidelines are still being developed)

Hybrid approach: Use Solo Mode for the initial bulk labeling, then assign a second annotator to a random 10-20% sample to compute agreement statistics. This gives you the efficiency of Solo Mode with the quality assurance of multi-annotator verification.

Solo Mode has no verification block of its own. Run the second annotator as an ordinary Potato pass over the exported labels, and compute agreement between their labels and Solo Mode's the way you would for any two annotators. (icl_labeling does have a verification block, but that belongs to in-context labeling, not to Solo Mode.)

Further Reading

For implementation details, see the source documentation.