A good LLM hook can make annotation both faster and more consistent. This guide covers wiring up OpenAI, Claude, Gemini, and local models so your annotators get useful suggestions while keeping the final call.

What LLM integration gets you

A model can pre-fill labels for the annotator to review, highlight the terms that matter in the text, flag annotations that look wrong, and explain the tricky cases so a stuck annotator has something to work with.

For how the AI support layer is built, see the source documentation.

Basic OpenAI integration

yaml

annotation_task_name: "AI-Assisted Sentiment Analysis"
 
# AI configuration
ai_support:
  enabled: true
  endpoint_type: openai
 
  ai_config:
    model: gpt-4
    api_key: ${OPENAI_API_KEY}
    temperature: 0.3
    max_tokens: 500
 
  features:
    hints:
      enabled: true
    keyword_highlighting:
      enabled: true
    label_suggestions:
      enabled: true
 
# ... rest of config
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    labels: [Positive, Negative, Neutral]

Supported providers

OpenAI

yaml

ai_support:
  enabled: true
  endpoint_type: openai
 
  ai_config:
    model: gpt-4  # or gpt-4o, gpt-3.5-turbo
    api_key: ${OPENAI_API_KEY}
    temperature: 0.3
    max_tokens: 500

Anthropic Claude

yaml

ai_support:
  enabled: true
  endpoint_type: anthropic
 
  ai_config:
    model: claude-3-sonnet-20240229
    api_key: ${ANTHROPIC_API_KEY}
    temperature: 0.3
    max_tokens: 500

Google Gemini

yaml

ai_support:
  enabled: true
  endpoint_type: google
 
  ai_config:
    model: gemini-1.5-pro
    api_key: ${GOOGLE_API_KEY}

Local Models (Ollama)

yaml

ai_support:
  enabled: true
  endpoint_type: ollama
 
  ai_config:
    model: llama2  # or mistral, mixtral, etc.
    base_url: http://localhost:11434

Feature: label suggestions

The model can propose a label for the annotator to weigh:

yaml

ai_support:
  enabled: true
  endpoint_type: openai
 
  ai_config:
    model: gpt-4
    api_key: ${OPENAI_API_KEY}
 
  features:
    label_suggestions:
      enabled: true
      show_confidence: true
 
annotation_schemes:
  - annotation_type: radio
    name: category
    labels: [News, Opinion, Satire, Other]

Feature: keyword highlighting

With keyword highlighting on, the LLM marks the relevant terms in the annotation text on its own:

AI keyword highlighting in Potato

To highlight important terms automatically:

yaml

ai_support:
  enabled: true
  endpoint_type: openai
 
  ai_config:
    model: gpt-4
    api_key: ${OPENAI_API_KEY}
 
  features:
    keyword_highlighting:
      enabled: true

Feature: hints

Give annotators a nudge without handing them the answer:

yaml

ai_support:
  enabled: true
  endpoint_type: openai
 
  ai_config:
    model: gpt-4
    api_key: ${OPENAI_API_KEY}
 
  features:
    hints:
      enabled: true

Hints show up as guidance rather than a verdict, so a hard case gets easier to reason through but the choice still belongs to the annotator.

Complete AI-assisted configuration

yaml

annotation_task_name: "AI-Assisted NER Annotation"
 
# AI Configuration
ai_support:
  enabled: true
  endpoint_type: openai
 
  ai_config:
    model: gpt-4
    api_key: ${OPENAI_API_KEY}
    temperature: 0.2
    max_tokens: 500
 
  features:
    hints:
      enabled: true
    keyword_highlighting:
      enabled: true
    label_suggestions:
      enabled: true
      show_confidence: true
 
  cache_config:
    disk_cache:
      enabled: true
      path: "ai_cache/cache.json"
    prefetch:
      warm_up_page_count: 50
      on_next: 5
      on_prev: 2
 
data_files:
  - data/texts.json
 
item_properties:
  id_key: id
  text_key: content
 
annotation_schemes:
  - annotation_type: span
    name: entities
    description: "Label named entities (AI suggestions provided)"
    labels:
      - name: PERSON
        color: "#FF6B6B"
      - name: ORG
        color: "#4ECDC4"
      - name: LOC
        color: "#45B7D1"
      - name: DATE
        color: "#96CEB4"
 
output_annotation_dir: "annotation_output/"
export_annotation_format: "json"

Working with AI suggestions

With AI support on, suggestions sit next to the annotation interface, and the annotator can accept, change, or ignore them. The saved annotation is always the annotator's call, so a human stays in the loop.

When caching is on, responses are stored, so the same instance never costs you a second API call.

Custom prompts

Potato includes default prompts for each annotation type, stored in potato/ai/prompt/. You can customize these by editing the prompt files:

Annotation Type	Prompt File
Radio buttons	`radio_prompt.txt`
Likert scales	`likert_prompt.txt`
Checkboxes	`checkbox_prompt.txt`
Span annotation	`span_prompt.txt`
Text input	`text_prompt.txt`

Prompts support variable substitution with {text}, {labels}, and {description}.

Tips for AI-assisted annotation

Start cautious and review every suggestion until you trust the model on your task. Watch the acceptance rate, because a low one usually means the prompt needs work, not the annotators. Tune the prompts against the errors you actually see. Keep the human in charge: the AI assists, the person decides. And track AI labels against human ones over time so you know how accurate the model really is.

New in v2.2: option highlighting

Potato 2.2 adds option highlighting, which reads the content and marks the most likely options for discrete tasks (radio, multiselect, likert). The top-k options get a star, the less likely ones dim down, and everything stays clickable.

yaml

ai_support:
  option_highlighting:
    enabled: true
    top_k: 3
    dim_opacity: 0.4

Read the full Option Highlighting documentation →

Next steps

Enable active learning to prioritize uncertain items
Set up quality control with AI metrics
Learn about local models for privacy
Explore option highlighting for guided annotation

Full AI documentation at /docs/features/ai-support.