Triage

The triage annotation schema provides a Prodigy-style binary accept/reject/skip interface optimized for rapid data curation tasks. This schema is ideal for filtering large datasets, performing data quality checks, or any task requiring quick binary decisions.

Overview

The triage schema presents three large, visually distinct buttons:

• Keep (green) - Accept the item for inclusion
• Discard (red) - Reject the item
• Unsure (gray) - Skip when uncertain

With auto-advance enabled (default), annotators can process hundreds of items per hour using just keyboard shortcuts.

Quick Start

annotation_schemes:
  - annotation_type: triage
    name: data_quality
    description: Is this data sample suitable for training?
    auto_advance: true
    show_progress: true

Configuration Options

Examples

Data Quality Filtering

annotation_schemes:
  - annotation_type: triage
    name: quality_filter
    description: Is this text high quality and suitable for training?
    auto_advance: true

Content Moderation

annotation_schemes:
  - annotation_type: triage
    name: content_moderation
    description: Does this content violate community guidelines?
    accept_label: "Safe"
    reject_label: "Violates"
    skip_label: "Review Later"
    auto_advance: true

Relevance Filtering

annotation_schemes:
  - annotation_type: triage
    name: relevance
    description: Is this document relevant to the query?
    accept_label: "Relevant"
    reject_label: "Not Relevant"
    skip_label: "Partially Relevant"

Combined with Other Schemas

Triage can be combined with other annotation types for both filtering and detailed annotation:

annotation_schemes:
  - annotation_type: triage
    name: include_item
    description: Should this item be included in the dataset?
    auto_advance: false  # Don't auto-advance when combined
 
  - annotation_type: radio
    name: category
    description: What category does this belong to?
    labels: ["News", "Opinion", "Review", "Other"]

Keyboard Shortcuts

The keys 1, 2, 3 are adjacent on the keyboard, allowing fast annotation without looking at the keyboard.

Output Format

{
  "data_quality": {
    "labels": {
      "decision": "accept"
    }
  }
}

Possible values: "accept", "reject", "skip".

Best Practices

1. Use auto-advance for high-throughput tasks - significantly improves throughput for simple decisions
2. Disable auto-advance for complex decisions - when annotators need time to think or make additional annotations
3. Provide clear descriptions - make triage criteria unambiguous for consistent annotations
4. Use meaningful labels - customize labels to match your specific use case
5. Combine with attention checks - for crowdsourcing, insert known items to verify quality

Further Reading

• AI Support - AI-assisted annotation hints
• Quality Control - Attention checks and gold standards
• Productivity Features - Keyboard shortcuts

For implementation details, see the source documentation.