Integrating LLMs for Smart Annotation Hints

A good LLM hook makes annotation faster and more consistent. This guide wires up OpenAI, Anthropic, Gemini, and local models so annotators get useful suggestions, and keeps the decision with the person doing the labeling.

What the assistants do

With ai_support on, each annotation scheme carries three buttons the annotator presses when they want help:

• Hint gives guidance on how to think about the item, sometimes carrying a suggested label
• Keyword marks the words bearing on each label, drawn as boxes over the text
• Rationale explains why each available label might apply

Label suggestions are not a separate button. When a hint carries a suggested label, that label is marked with an amber border and a sparkle.

For how the AI support layer is built, see the source documentation.

Basic OpenAI integration

annotation_task_name: "AI-Assisted Sentiment Analysis"
task_dir: "."
 
ai_support:
  enabled: true
  endpoint_type: openai
  ai_config:
    model: gpt-4o-mini
    api_key: ${OPENAI_API_KEY}
    temperature: 0.3
    max_tokens: 800
    include:
      all: true
 
data_files:
  - data/texts.json
 
item_properties:
  id_key: id
  text_key: content
 
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What is the sentiment of this text?"
    labels: [Positive, Negative, Neutral]
 
output_annotation_dir: annotation_output/
export_annotation_format: jsonl

Two things about this block are worth knowing, because neither fails loudly:

include.all is the switch. It is off by default, and without it no assistant button appears anywhere. The page renders an empty AI panel, validation passes, and nothing warns you.

The model settings live inside ai_config. ai_support.model is ignored. potato validate --strict names the key, which is the fastest way to catch it.

Leave max_tokens at 800 or above. The Keyword and Rationale assistants answer for every label in one reply, so a smaller budget truncates the response and the assistant renders empty with no error.

Supported providers

OpenAI

ai_support:
  enabled: true
  endpoint_type: openai
  ai_config:
    model: gpt-4o-mini
    api_key: ${OPENAI_API_KEY}
    temperature: 0.3
    max_tokens: 800
    include:
      all: true

Anthropic

ai_support:
  enabled: true
  endpoint_type: anthropic
  ai_config:
    model: claude-sonnet-4-20250514
    api_key: ${ANTHROPIC_API_KEY}
    temperature: 0.3
    max_tokens: 800
    include:
      all: true

Google Gemini

The endpoint is gemini:

ai_support:
  enabled: true
  endpoint_type: gemini
  ai_config:
    model: gemini-2.0-flash-exp
    api_key: ${GOOGLE_API_KEY}
    max_tokens: 800
    include:
      all: true

Local models with Ollama

ai_support:
  enabled: true
  endpoint_type: ollama
  ai_config:
    model: llama3.2
    base_url: http://localhost:11434
    max_tokens: 800
    include:
      all: true

A local endpoint is worth the quality drop when the corpus cannot leave your machine, or when the per-item cost over a large study would otherwise dominate the budget. vllm and huggingface take the same shape.

Turning assistants on per scheme

include.all enables every assistant everywhere. special_include is the alternative, mapping a page index to an annotation id to the assistants that scheme should carry:

ai_support:
  enabled: true
  endpoint_type: openai
  ai_config:
    model: gpt-4o-mini
    api_key: ${OPENAI_API_KEY}
    max_tokens: 800
    include:
      all: false
      special_include:
        "0":
          "0": ["hint", "keyword"]
        "1":
          "1": ["hint"]

The inner key is the annotation id, which is the scheme's position in annotation_schemes counting from zero. This matters on a task with several schemes, where turning all three assistants on everywhere puts nine buttons on the page.

Highlighting the likely labels

Where the problem is a long option list rather than long text, option_highlighting marks the most likely options and dims the rest. Every option stays clickable:

ai_support:
  enabled: true
  endpoint_type: openai
  ai_config:
    model: gpt-4o-mini
    api_key: ${OPENAI_API_KEY}
    max_tokens: 800
 
  option_highlighting:
    enabled: true
    top_k: 3
    dim_opacity: 0.4

It works on radio, multiselect, likert and select, and has no effect on spans, free text, or sliders. See Option Highlighting.

Caching and prefetching

An LLM call adds seconds to a page load, and an annotator who waits stops pressing the button:

ai_support:
  enabled: true
  endpoint_type: openai
  ai_config:
    model: gpt-4o-mini
    api_key: ${OPENAI_API_KEY}
    max_tokens: 800
    include:
      all: true
 
  cache_config:
    disk_cache:
      enabled: true
      path: annotation_output/ai_cache.json
    prefetch:
      warm_up_page_count: 20
      on_next: 10
      on_prev: 3

The cache survives restarts, so an item an annotator revisits costs nothing the second time.

Complete configuration

annotation_task_name: "AI-Assisted NER Annotation"
task_dir: "."
 
ai_support:
  enabled: true
  endpoint_type: openai
  ai_config:
    model: gpt-4o-mini
    api_key: ${OPENAI_API_KEY}
    temperature: 0.2
    max_tokens: 800
    include:
      all: true
 
  cache_config:
    disk_cache:
      enabled: true
      path: annotation_output/ai_cache.json
    prefetch:
      warm_up_page_count: 20
      on_next: 5
      on_prev: 2
 
data_files:
  - data/texts.json
 
item_properties:
  id_key: id
  text_key: content
 
annotation_schemes:
  - annotation_type: span
    name: entities
    description: "Label the named entities in this text"
    labels:
      - name: PERSON
        color: "#FF6B6B"
      - name: ORG
        color: "#4ECDC4"
      - name: LOC
        color: "#45B7D1"
      - name: DATE
        color: "#96CEB4"
 
output_annotation_dir: annotation_output/
export_annotation_format: jsonl

Custom prompts

Prompts are JSON files in potato/ai/prompt/, one per annotation type:

Each file holds a template per assistant with a name, the prompt, and an output_format. The variables are ${text}, ${description}, ${labels}, ${min_label}, ${max_label}, ${size}, ${min_value}, and ${max_value}.

Working with AI suggestions

The saved annotation is always the annotator's, so the human stays in the loop by construction. What that does not guarantee is independence: an annotator who accepts suggestions without checking them produces labels that agree with each other because they came from the same model, and agreement computed over those labels measures the model rather than the task.

Watch the acceptance rate for that reason. A rate near 100% usually means suggestions are being confirmed rather than checked.

Tips

Start cautious and review every suggestion until the model has earned trust on your data. A low acceptance rate usually means the prompt needs work rather than the annotators. Tune prompts against the errors you actually see rather than the ones you expect. And track AI labels against human ones over time, since that comparison is the only thing that tells you how accurate the assistance really is.

Next steps

• Enable active learning to prioritize uncertain items
• Set up quality control
• Read AI Support for the full endpoint reference
• Explore option highlighting for guided annotation

Full AI documentation at AI Support.