ICL Labeling

ICL (in-context learning) labeling feeds an LLM the annotations your annotators already agree on, then has the model label the rest of the data. Potato records a confidence score for every prediction and routes a sample of them back to humans without showing the model's answer, so the reported accuracy comes from independent labels rather than from the model's own confidence.

Overview

The ICL labeling system:

1. Collects High-Confidence Examples: Identifies instances where annotators agree (e.g., 80%+ agreement)
2. Labels with LLM: Uses examples to prompt an LLM for labeling unlabeled instances
3. Tracks Confidence: Records LLM confidence scores for each prediction
4. Verifies Accuracy: Routes a sample of LLM-labeled instances to humans for blind verification
5. Reports Metrics: Calculates and displays LLM accuracy based on verification results

Features

Automatic Example Collection

Potato rebuilds the example pool on an interval from the instances where annotators already agree, so the prompt tracks the annotators rather than a snapshot taken when the server started:

• Configurable agreement threshold (default: 80%)
• Minimum annotator count requirement (default: 2)
• Automatic refresh on a configurable interval
• Per-schema example pools

The minimum annotator count is the setting most likely to leave the pool empty. It defaults to 2, so a study running with num_annotators_per_item: 1 can never produce an example and the worker reports zero examples across zero schemas forever. Set min_annotators_per_instance: 1 for a single-annotator study, or raise num_annotators_per_item.

LLM Labeling with Limits

Three limits keep the model from labeling the whole dataset before anyone has checked whether its labels are any good:

• Max total labels: Limit the total number of LLM predictions
• Max unlabeled ratio: Only label a percentage of remaining data
• Pause on low accuracy: Pause automatically when measured accuracy drops below a threshold

Blind Verification

Verification is blind. The annotator sees the instance as an ordinary task with no sign of the model's prediction, so their label is an independent second opinion rather than a check on an answer already in front of them:

• Configurable sample rate (default: 20% of LLM labels)
• Multiple selection strategies: low_confidence, random, mixed
• Verification tasks mixed in with regular assignments

Configuration

ICL labeling reuses the endpoint configured under ai_support, so that block has to be present and enabled:

# AI endpoint configuration (required)
ai_support:
  enabled: true
  endpoint_type: "openai"
  ai_config:
    model: "gpt-4o-mini"
    api_key: "${OPENAI_API_KEY}"
 
# ICL labeling configuration
icl_labeling:
  enabled: true
 
  # Example selection settings
  example_selection:
    min_agreement_threshold: 0.8      # 80% annotators must agree
    min_annotators_per_instance: 2    # Minimum annotations for consensus
    max_examples_per_schema: 10       # Max examples per schema in prompt
    refresh_interval_seconds: 300     # How often to refresh examples
 
  # LLM labeling settings
  llm_labeling:
    batch_size: 20
    trigger_threshold: 5              # Min examples before LLM labeling starts
    confidence_threshold: 0.7         # Min confidence to accept prediction
    batch_interval_seconds: 600
    max_total_labels: 100             # Max instances to label total
    max_unlabeled_ratio: 0.5          # Max portion of unlabeled to label
    pause_on_low_accuracy: true
    min_accuracy_threshold: 0.7
 
  # Human verification settings
  verification:
    enabled: true
    sample_rate: 0.2                  # 20% of LLM labels verified
    selection_strategy: "low_confidence"
    mix_with_regular_assignments: true
    assignment_mix_rate: 0.2

All four example-selection settings live under example_selection, all eight labeling settings under llm_labeling, and all five verification settings under verification. A flat block written directly under icl_labeling is reported as an unrecognized key rather than accepted and ignored.

Two values are worth setting deliberately. max_total_labels defaults to unlimited, so a first run wants a number small enough that a bad prompt costs a few dozen predictions instead of the corpus. confidence_threshold is the floor a prediction has to clear to be stored at all, so raising it discards the model's least certain guesses and lowering it keeps them.

Selection Strategies

• low_confidence: Prioritizes verifying LLM's least confident predictions first
• random: Random sampling from all predictions
• mixed: 50% low confidence + 50% random

Admin API

Status Endpoint

GET /admin/api/icl/status

Returns overall ICL labeler status including examples per schema, predictions made, verification queue size, and accuracy metrics.

Examples Endpoint

GET /admin/api/icl/examples?schema=sentiment

Returns high-confidence examples, optionally filtered by schema.

Predictions Endpoint

GET /admin/api/icl/predictions?schema=sentiment&status=pending

Returns LLM predictions, optionally filtered by schema and verification status.

Accuracy Endpoint

GET /admin/api/icl/accuracy?schema=sentiment

Returns accuracy metrics based on human verification results.

Manual Trigger Endpoint

POST /admin/api/icl/trigger
Content-Type: application/json
 
{"schema_name": "sentiment"}

Triggers a batch labeling run for one schema without waiting for the interval.

Usage Workflow

1. Configure Your Project

Start with a cap low enough that you can read every prediction, and a verification rate high enough to estimate accuracy from that cap:

ai_support:
  enabled: true
  endpoint_type: "openai"
  ai_config:
    model: "gpt-4o-mini"
    api_key: "${OPENAI_API_KEY}"
 
icl_labeling:
  enabled: true
  example_selection:
    min_agreement_threshold: 0.8
  llm_labeling:
    max_total_labels: 50  # Start small
  verification:
    enabled: true
    sample_rate: 0.3  # Verify 30% initially

At max_total_labels: 50 and sample_rate: 0.3, about 15 predictions come back verified, which is enough to see a model that is badly wrong and not enough to separate 85% accuracy from 90%.

2. Collect Human Annotations

Have annotators label data normally. Instances that reach the agreement threshold become available as examples on the next refresh.

3. Monitor Progress

curl http://localhost:8000/admin/api/icl/status

4. Review Accuracy

curl http://localhost:8000/admin/api/icl/accuracy

5. Iterate

Based on accuracy:

• If accuracy is high (>80%), increase max_total_labels
• If accuracy is low, add more human examples before continuing

Best Practices

1. Start Small: Begin with conservative limits (max_total_labels: 50) to assess accuracy before scaling

2. Verify Early: Use a higher sample_rate initially (0.3-0.5) to get confident accuracy estimates

3. Monitor Actively: Check accuracy metrics regularly through the admin API

4. Adjust Thresholds: If LLM accuracy is low:

   • Increase min_agreement_threshold for cleaner examples
   • Increase trigger_threshold so more examples accumulate before labeling starts
   • Increase confidence_threshold to discard the model's least certain predictions

5. Use Selection Strategies:

   • low_confidence: Best for identifying problematic categories
   • random: Best for unbiased accuracy estimates
   • mixed: Balanced approach

Troubleshooting

LLM Not Labeling

1. Check if ai_support is properly configured
2. Verify enough high-confidence examples exist (/admin/api/icl/status)
3. Check if labeling is paused due to limits or low accuracy

Low Accuracy

1. Increase min_agreement_threshold for cleaner examples
2. Add more annotation guidelines and instructions
3. Review examples being used (/admin/api/icl/examples)

Verification Tasks Not Appearing

1. Verify verification.enabled is true
2. Check mix_with_regular_assignments is true
3. Verify there are pending verifications in the queue

Further Reading

• AI Support - General AI endpoint configuration
• Active Learning - Related AI-assisted features
• Quality Control - Accuracy tracking

For implementation details, see the source documentation.