Skip to content

Active Learning for Annotation

What active learning is, when it helps, and which query strategies Potato supports (uncertainty, diversity, BADGE, BALD), so you label fewer items for the same model quality.

Active learning chooses which items to annotate next so a model reaches the same accuracy with far fewer labels. Instead of labeling at random, you label the items the model finds most informative.

See active learning for background. For the feature reference, see Active Learning.

The loop

  1. Label a small seed set.
  2. Train a quick model on what you have.
  3. Score the unlabeled pool and pick the most informative items.
  4. Annotate those, add them, retrain. Repeat.

Each round spends the annotation budget where the model learns most.

Query strategies Potato supports

  • Uncertainty sampling: pick items the model is least confident about (near the decision boundary). The simplest default, and often good enough.
  • Diversity sampling: pick items that are different from each other, so you don't waste budget on near-duplicates.
  • BADGE: combines uncertainty and diversity using gradient embeddings.
  • BALD: Bayesian strategy that selects items expected to most reduce model uncertainty.
  • Hybrid: blends strategies.
yaml
active_learning:
  enabled: true
  schema_names: [sentiment]
  query_strategy: uncertainty   # or diversity, badge, bald, hybrid
  min_instances_for_training: 20

When it helps

It helps when labels are expensive, the pool is large, and a useful model can be trained on a small seed. It helps less when:

  • The task is so easy that random labeling already saturates quickly.
  • You need an unbiased held-out test set. Active learning skews what gets labeled, so keep the evaluation data randomly sampled.
  • Labels are cheap relative to engineering effort.

Further reading