# Integrating LLMs for Smart Annotation Hints

Source: https://www.potatoannotator.com/blog/llm-integration-guide

A good LLM hook can make annotation both faster and more consistent. This guide covers wiring up OpenAI, Claude, Gemini, and local models so your annotators get useful suggestions while keeping the final call.

## What LLM integration gets you

A model can pre-fill labels for the annotator to review, highlight the terms that matter in the text, flag annotations that look wrong, and explain the tricky cases so a stuck annotator has something to work with.

For how the AI support layer is built, see the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/ai_support.md).

## Basic OpenAI integration

```yaml
annotation_task_name: "AI-Assisted Sentiment Analysis"

# AI configuration
ai_support:
  enabled: true
  endpoint_type: openai

  ai_config:
    model: gpt-4
    api_key: ${OPENAI_API_KEY}
    temperature: 0.3
    max_tokens: 500

  features:
    hints:
      enabled: true
    keyword_highlighting:
      enabled: true
    label_suggestions:
      enabled: true

# ... rest of config
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    labels: [Positive, Negative, Neutral]
```

## Supported providers

### OpenAI

```yaml
ai_support:
  enabled: true
  endpoint_type: openai

  ai_config:
    model: gpt-4  # or gpt-4o, gpt-3.5-turbo
    api_key: ${OPENAI_API_KEY}
    temperature: 0.3
    max_tokens: 500
```

### Anthropic Claude

```yaml
ai_support:
  enabled: true
  endpoint_type: anthropic

  ai_config:
    model: claude-3-sonnet-20240229
    api_key: ${ANTHROPIC_API_KEY}
    temperature: 0.3
    max_tokens: 500
```

### Google Gemini

```yaml
ai_support:
  enabled: true
  endpoint_type: google

  ai_config:
    model: gemini-1.5-pro
    api_key: ${GOOGLE_API_KEY}
```

### Local Models (Ollama)

```yaml
ai_support:
  enabled: true
  endpoint_type: ollama

  ai_config:
    model: llama2  # or mistral, mixtral, etc.
    base_url: http://localhost:11434
```

## Feature: label suggestions

The model can propose a label for the annotator to weigh:

```yaml
ai_support:
  enabled: true
  endpoint_type: openai

  ai_config:
    model: gpt-4
    api_key: ${OPENAI_API_KEY}

  features:
    label_suggestions:
      enabled: true
      show_confidence: true

annotation_schemes:
  - annotation_type: radio
    name: category
    labels: [News, Opinion, Satire, Other]
```

## Feature: keyword highlighting

With keyword highlighting on, the LLM marks the relevant terms in the annotation text on its own:

![AI keyword highlighting in Potato](/images/blog/keyword-highlights.png)

To highlight important terms automatically:

```yaml
ai_support:
  enabled: true
  endpoint_type: openai

  ai_config:
    model: gpt-4
    api_key: ${OPENAI_API_KEY}

  features:
    keyword_highlighting:
      enabled: true
```

## Feature: hints

Give annotators a nudge without handing them the answer:

```yaml
ai_support:
  enabled: true
  endpoint_type: openai

  ai_config:
    model: gpt-4
    api_key: ${OPENAI_API_KEY}

  features:
    hints:
      enabled: true
```

Hints show up as guidance rather than a verdict, so a hard case gets easier to reason through but the choice still belongs to the annotator.

## Complete AI-assisted configuration

```yaml
annotation_task_name: "AI-Assisted NER Annotation"

# AI Configuration
ai_support:
  enabled: true
  endpoint_type: openai

  ai_config:
    model: gpt-4
    api_key: ${OPENAI_API_KEY}
    temperature: 0.2
    max_tokens: 500

  features:
    hints:
      enabled: true
    keyword_highlighting:
      enabled: true
    label_suggestions:
      enabled: true
      show_confidence: true

  cache_config:
    disk_cache:
      enabled: true
      path: "ai_cache/cache.json"
    prefetch:
      warm_up_page_count: 50
      on_next: 5
      on_prev: 2

data_files:
  - data/texts.json

item_properties:
  id_key: id
  text_key: content

annotation_schemes:
  - annotation_type: span
    name: entities
    description: "Label named entities (AI suggestions provided)"
    labels:
      - name: PERSON
        color: "#FF6B6B"
      - name: ORG
        color: "#4ECDC4"
      - name: LOC
        color: "#45B7D1"
      - name: DATE
        color: "#96CEB4"

output_annotation_dir: "annotation_output/"
export_annotation_format: "json"
```

## Working with AI suggestions

With AI support on, suggestions sit next to the annotation interface, and the annotator can accept, change, or ignore them. The saved annotation is always the annotator's call, so a human stays in the loop.

When caching is on, responses are stored, so the same instance never costs you a second API call.

## Custom prompts

Potato includes default prompts for each annotation type, stored in `potato/ai/prompt/`. You can customize these by editing the prompt files:

| Annotation Type | Prompt File |
|-----------------|-------------|
| Radio buttons | `radio_prompt.txt` |
| Likert scales | `likert_prompt.txt` |
| Checkboxes | `checkbox_prompt.txt` |
| Span annotation | `span_prompt.txt` |
| Text input | `text_prompt.txt` |

Prompts support variable substitution with `{text}`, `{labels}`, and `{description}`.

## Tips for AI-assisted annotation

Start cautious and review every suggestion until you trust the model on your task. Watch the acceptance rate, because a low one usually means the prompt needs work, not the annotators. Tune the prompts against the errors you actually see. Keep the human in charge: the AI assists, the person decides. And track AI labels against human ones over time so you know how accurate the model really is.

## New in v2.2: option highlighting

Potato 2.2 adds option highlighting, which reads the content and marks the most likely options for discrete tasks (radio, multiselect, likert). The top-k options get a star, the less likely ones dim down, and everything stays clickable.

```yaml
ai_support:
  option_highlighting:
    enabled: true
    top_k: 3
    dim_opacity: 0.4
```

[Read the full Option Highlighting documentation →](/docs/features/option-highlighting)

## Next steps

- Enable [active learning](/blog/active-learning-efficiency) to prioritize uncertain items
- Set up [quality control](/blog/quality-control-strategies) with AI metrics
- Learn about [local models](/docs/features/ai-support) for privacy
- Explore [option highlighting](/docs/features/option-highlighting) for guided annotation

---

*Full AI documentation at [/docs/features/ai-support](/docs/features/ai-support).*