AI Support

With AI support on, an LLM writes hints, highlights keywords, and explains why each label might apply while the annotator works. It does not fill the form in. The annotator still decides, and every assistant sits behind a button they press.

Installation

The provider SDKs are optional dependencies. A base install runs without any of them, and each SDK is imported only when a config selects that endpoint_type:

pip install "potato-annotation[ai]"

That covers the common ones. If a config selects an endpoint whose SDK is missing, the server reports which package to install rather than failing at import.

Supported Providers

Cloud: openai, anthropic, gemini, huggingface

Local: ollama, vllm

Vision: ollama_vision, openai_vision, anthropic_vision, yolo

See Visual AI Support for the vision endpoints and the image and video assistants.

Configuration

ai_support:
  enabled: true
  endpoint_type: "openai"
  ai_config:
    model: "gpt-4o-mini"
    api_key: "${OPENAI_API_KEY}"
    temperature: 0.7
    max_tokens: 800
    include:
      all: true

Two things about this block are worth stating plainly, because nothing catches either one at runtime:

The model settings go inside ai_config, one level deeper than the obvious guess. ai_support.model and ai_support.base_url are ignored, though potato validate --strict names them.

include.all is off by default. Without it no assistant button appears anywhere, the page renders an empty AI panel, validation passes clean, and nothing warns you.

Configuration Options

openai and openai_vision accept an API key three ways, and validation takes all three: the api_key value, OPENAI_API_KEY in the environment, or a base_url pointing at a self-hosted server that needs no key. anthropic, huggingface and gemini read only the first, so for those the key really is required.

Leave max_tokens at its default unless you have a reason. The Rationale and Keyword assistants answer for every label in one reply, so a few hundred tokens cuts the response mid-object and the assistant renders empty with no error.

The Assistants

Annotators get three buttons per scheme, and a fourth behavior that rides along with hints.

Hint gives contextual guidance on how to think about the item, optionally carrying a suggested label.

Keyword marks the words and phrases bearing on each label, drawn as box overlays on the text rather than as background highlighting.

Rationale explains why each available label might apply, giving balanced reasoning across all of them. It is the one worth enabling when annotators are being trained or the decisions are genuinely hard.

Label suggestions are not a separate button. When the hint carries a suggested label, that label is marked with an amber border and a sparkle indicator.

What Each Endpoint Can Do

"Text items only" is decided per item rather than per endpoint. A vision model is still a language model, so on a text item it offers all three assistants, and on an image item the Keyword button is filtered out because there are no words to mark.

Items With Both a Picture and Text

A support ticket with a screenshot, or a product photo with a review, should reach the model as both. Potato takes the text from item_properties.text_key and the picture from whichever of these resolves first:

1. ai_support.image_key
2. item_properties.image_key
3. an instance_display field of type: image, or the first displayed field whose value looks like one

So a config that already declares its photo through instance_display does not name it twice:

instance_display:
  fields:
    - {key: body, type: text, label: Ticket}
    - {key: screenshot, type: image, label: Screenshot}

Sending only the picture is how a hint ends up confidently describing the wrong half of the item. A model handed a stock photo and asked to triage "the export button spins forever" will report that it was given no context.

The field takes an http(s) URL, a data URI, or a filename under media_directory. A local file is read from disk and sent as base64. A value that is neither a URL nor a file Potato can find is logged and the model is asked without the picture, so a hint that reasons only about the text is a reason to check that log line.

Enabling Assistants Per Scheme

include.all turns every assistant on everywhere. special_include is the alternative, mapping a page index to an annotation id to the list of assistants that scheme should carry:

ai_support:
  enabled: true
  endpoint_type: "openai"
  ai_config:
    model: "gpt-4o-mini"
    api_key: "${OPENAI_API_KEY}"
    include:
      all: false
      special_include:
        "0":
          "0": ["hint", "keyword"]
        "1":
          "1": ["hint"]
        "2":
          "0": ["hint", "keyword"]
          "1": ["hint", "keyword"]

The inner key is the annotation id, which is the scheme's position in annotation_schemes counting from zero.

This matters more than it looks when the form is laid out in a grid. The assistant row renders per scheme, so three assistants across three schemes is nine buttons, one row per cell. Below about 260px of cell width the row collapses to icons on a single line, keeping the label as the button's accessible name and on its title. If you are combining layout.grid with AI support, name the assistants you want per scheme rather than turning all of them on everywhere.

Caching and Prefetching

An LLM call adds seconds to a page load, and an annotator who waits stops pressing the button. Caching also stops you paying twice for an item somebody revisits:

ai_support:
  enabled: true
  endpoint_type: "openai"
  ai_config:
    model: "gpt-4o-mini"
    api_key: "${OPENAI_API_KEY}"
    include:
      all: true
  cache_config:
    disk_cache:
      enabled: true
      path: "annotation_output/ai_cache.json"
    prefetch:
      warm_up_page_count: 20
      on_next: 10
      on_prev: 3

Warmup pre-generates replies for the first N instances at startup. Prefetch generates them ahead of the cursor as annotators navigate, and the larger on_next against on_prev reflects that most movement is forward. The disk cache survives restarts.

External AI Config File

Endpoint details tend to be the parts you cannot commit: API keys, internal server URLs, whichever model that machine has. ai_config_file moves them into their own file:

# config.yaml, committed, no secrets
ai_support:
  enabled: true
  ai_config_file: ai-config.yaml
 
  ai_config:
    temperature: 0.7
    max_tokens: 800
    include:
      all: true

# ai-config.yaml, gitignored
endpoint_type: ollama
model: qwen3:0.6b
base_url: http://localhost:11434

The external file sets endpoint_type, and its other values merge into ai_config taking precedence over the inline ones. The merge is flat: the external file holds model, base_url and api_key at its top level. Wrapping them in an ai_config: block inside that file produces ai_config.ai_config, which nothing reads, and the boot then fails with a missing-API-key error pointing at the wrong key.

If ai_config_file is set but the file is absent, AI support switches itself off with a warning and the server still starts. potato validate --strict treats that warning as an error, so a study whose subject is AI assistance cannot pass its own check with the AI off. Ship an ai-config.yaml.example beside the config for people to copy. The ai-config.yaml and ai-config*.yaml patterns are gitignored.

Custom Prompts

Prompts live as JSON files in potato/ai/prompt/, one per annotation type:

Each file holds a template per assistant, with a name, the prompt, an output_format, and an icon:

{
  "hint": {
    "name": "Hint",
    "prompt": "TASK: Generate annotation guidance...\n- Text to annotate: \"${text}\"\n- Annotation task: ${description}\n- Available labels: ${labels}",
    "output_format": "default_hint",
    "img": "/static/ai_assistant_img/blub.svg"
  }
}

The variables are ${text}, ${description}, ${labels}, ${min_label}, ${max_label}, ${size}, ${min_value}, and ${max_value}.

output_format names the reply shape the renderer expects. default_hint returns {hint, suggestive_choice}, default_keyword returns {label_keywords: [{label, keywords}]}, and default_rationale returns {rationales: [{label, reasoning}]}.

Provider Configuration

OpenAI

ai_support:
  enabled: true
  endpoint_type: "openai"
  ai_config:
    model: "gpt-4o-mini"
    api_key: "${OPENAI_API_KEY}"
    temperature: 0.7
    max_tokens: 800
    include:
      all: true

Anthropic

ai_support:
  enabled: true
  endpoint_type: "anthropic"
  ai_config:
    model: "claude-sonnet-4-20250514"
    api_key: "${ANTHROPIC_API_KEY}"
    temperature: 0.7
    max_tokens: 800
    include:
      all: true

Google Gemini

ai_support:
  enabled: true
  endpoint_type: "gemini"
  ai_config:
    model: "gemini-2.0-flash-exp"
    api_key: "${GOOGLE_API_KEY}"
    max_tokens: 800
    include:
      all: true

Ollama

ai_support:
  enabled: true
  endpoint_type: "ollama"
  ai_config:
    model: "llama3.2"
    base_url: "http://localhost:11434"
    max_tokens: 800
    include:
      all: true

vLLM

ai_support:
  enabled: true
  endpoint_type: "vllm"
  ai_config:
    model: "meta-llama/Llama-3.2-3B-Instruct"
    base_url: "http://localhost:8000"
    max_tokens: 800
    include:
      all: true

HuggingFace

ai_support:
  enabled: true
  endpoint_type: "huggingface"
  ai_config:
    model: "meta-llama/Llama-3.2-3B-Instruct"
    api_key: "${HF_API_KEY}"
    max_tokens: 800
    include:
      all: true

OpenRouter is not one of the ten values ai_support.endpoint_type accepts, so a config naming it here does not load. It is available to the chat sidebar instead, as chat_support.endpoint_type: openrouter — spelled without the underscore.

A Separate Endpoint for Images

A text model and a vision model can run side by side, so each content type gets the better one:

ai_support:
  enabled: true
  endpoint_type: "ollama"
  visual_endpoint_type: "ollama_vision"
  ai_config:
    model: "llama3.2"
    include:
      all: true
  visual_ai_config:
    model: "qwen2.5-vl:7b"

Full Example

annotation_task_name: "Sentiment Analysis with AI Support"
task_dir: "."
 
data_files:
  - "data/reviews.json"
 
item_properties:
  id_key: id
  text_key: text
 
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What is the sentiment of this text?"
    labels: ["positive", "negative", "neutral"]
 
  - annotation_type: multiselect
    name: topics
    description: "What topics are discussed?"
    labels: ["product", "service", "price", "quality"]
 
ai_support:
  enabled: true
  endpoint_type: "openai"
  ai_config:
    model: "gpt-4o-mini"
    api_key: "${OPENAI_API_KEY}"
    temperature: 0.7
    max_tokens: 800
    include:
      all: true
  cache_config:
    disk_cache:
      enabled: true
      path: "annotation_output/ai_cache.json"
    prefetch:
      warm_up_page_count: 20
      on_next: 10
      on_prev: 3
 
output_annotation_dir: "output/"
export_annotation_format: jsonl
 
user_config:
  allow_all_users: true

Environment Variables

Keep keys out of the config:

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GOOGLE_API_KEY="..."

Reference them with ${VARIABLE_NAME}, which works in the main config and in the external AI config file.

Troubleshooting

The tooltip is empty, but the server answered

The model replied and the request returned 200, and the tooltip shows nothing. Four things produce that same symptom, so check them in this order:

1. include.all is missing. There is no assistant button on the page at all, and nothing warns about it.
2. max_tokens is too low. The Rationale and Keyword assistants answer for every label at once, so the reply is cut mid-object. The log says the model hit max_tokens.
3. The server ignored the schema. Check the reply is JSON rather than prose. A server without constrained decoding answers in whatever shape the prompt suggested.
4. The keys are in the wrong place. Everything about the model belongs in ai_config. validate --strict names any key it does not recognize.

No buttons at all, and the boot log looked fine

Look for "AI endpoint ready" in the log. If it says the endpoint failed to start, the reason is in the warning above that line.

Connection failures on Ollama or vLLM

Confirm the server is running and the model is pulled, with ollama list for Ollama, and check the configured base_url is reachable.

Best Practices

1. Treat it as assistance - The annotator makes the decision, and a study that accepts the suggestion by default is measuring the model
2. Enable caching in production - It cuts latency and repeat costs
3. Use warmup - Pre-generate the first instances so the first annotator does not wait
4. Watch the bill - Cloud providers charge per request, and an assistant on every scheme multiplies through the corpus
5. Consider a local endpoint - Ollama or vLLM for high-volume work, or a corpus that cannot leave the machine
6. Keep keys in the environment - Never commit them

Further Reading

• Visual AI Support - Vision endpoints for image and video
• Option Highlighting - Dimming the less likely labels
• Diversity Ordering - Embedding-based item ordering
• ICL Labeling - In-context learning over your own annotations
• Active Learning - Model-driven instance prioritization

For implementation details, see the source documentation.