Option Highlighting

Option Highlighting is an AI-assisted feature that helps annotators identify the most likely correct options for discrete annotation tasks. Using an LLM, the system analyzes the content and task description to predict the top-k most likely options, displaying them at full opacity while dimming less-likely options.

Overview

When enabled, Option Highlighting:

1. Analyzes instance content using an LLM
2. Identifies the most probable options based on task context
3. Highlights likely options with a star indicator
4. Dims less-likely options (configurable opacity)
5. Keeps all options fully clickable - this is guidance, not restriction

This is particularly useful for tasks with many options, training new annotators, reducing cognitive load, and providing a "second opinion" to increase annotation confidence.

Configuration

Add the option_highlighting section under ai_support in your config:

ai_support:
  enabled: true
  endpoint_type: "openai"
  ai_config:
    model: "gpt-4o-mini"
    api_key: "${OPENAI_API_KEY}"
    temperature: 0.3
 
  option_highlighting:
    enabled: true
    top_k: 3                    # Number of options to highlight (1-10)
    dim_opacity: 0.4            # Opacity for non-highlighted options (0.1-0.9)
    auto_apply: true            # Apply on page load vs manual trigger
    schemas: null               # null = all schemas, or ["schema1", "schema2"]
    prefetch_count: 20          # Items to prefetch
 
  cache_config:
    disk_cache:
      enabled: true
      path: annotation_output/ai_cache.json
    prefetch:
      warm_up_page_count: 10
      on_next: 3
      on_prev: 1

Configuration Reference

Supported Annotation Types

Option highlighting works with discrete choice types:

• radio - Single choice selection
• multiselect - Multiple checkbox selection
• likert - Likert scale ratings
• select - Dropdown selection

It does not apply to: span, textbox, slider, image_annotation, or video_annotation.

Visual Appearance

• Highlighted options: Full opacity with a gold star indicator and subtle background highlight
• Dimmed options: Reduced opacity (configurable), brighten to 0.7 on hover, still fully clickable
• Form indicator: Gold left border on annotation forms with highlighting active, small "AI" badge

Prefetching

Since LLM queries can be slow, option highlighting uses aggressive prefetching:

1. Warmup: First N items prefetched on server start
2. On navigation: Next items prefetched when user moves forward/backward
3. Background processing: Prefetch happens asynchronously
4. Caching: Results cached to disk for reuse

Best Practices

1. Use a fast model - gpt-4o-mini provides good balance of speed and accuracy
2. Set appropriate top_k - for binary choices use 1, for 4-5 options use 2-3
3. Enable prefetching - higher prefetch_count (20+) ensures smooth navigation
4. Train annotators - explain that highlights are suggestions, not requirements

Further Reading

• AI Support - General AI assistant configuration
• Diversity Ordering - Complementary AI feature for item ordering

For implementation details, see the source documentation.