Name: Potato
Author: Potato

標註者間一致性（IAA）衡量不同標註者對相同項目標註的一致程度。高一致性表明標註可靠；低一致性則說明指南不清晰或任務具有主觀性。

Choosing an agreement metric: two raters use Cohen's kappa, many raters use Fleiss' kappa, ordinal or missing data use Krippendorff's alpha Which agreement metric to use

為什麼要衡量一致性？

驗證指南：低一致性說明指南不明確
評估任務難度：某些任務本身就具有主觀性
評價標註者：識別需要更多培訓的人
報告可靠性：學術發表的必要要求
聚合標籤：決定如何合併標註結果

一致性指標

Cohen's Kappa（2個標註者）

用於比較兩個標註者的分類資料：

text

κ = (Po - Pe) / (1 - Pe)

其中：

Po = 觀察到的一致率
Pe = 偶然預期的一致率

解讀標準：

Kappa 值	解讀
< 0	低於偶然水平
0.01-0.20	輕微一致
0.21-0.40	一般一致
0.41-0.60	中等一致
0.61-0.80	較高一致
0.81-1.00	幾乎完全一致

Fleiss' Kappa（3個及以上標註者）

用於多個標註者的分類資料：

yaml

quality_control:
  agreement:
    metrics:
      - fleiss_kappa

解讀標準與 Cohen's Kappa 相同。

Krippendorff's Alpha

最靈活的指標——支援：

任意數量的標註者
缺失資料
各種資料類型（名義、序數、區間、比率）

yaml

quality_control:
  agreement:
    metrics:
      - krippendorff_alpha
    alpha_level: nominal  # or ordinal, interval, ratio

解讀標準：

α ≥ 0.80：可靠
0.67 ≤ α < 0.80：勉強可接受
α < 0.67：不可靠

在 Potato 中配置一致性

基本設定

yaml

quality_control:
  agreement:
    enabled: true
    calculate_on_overlap: true
 
    metrics:
      - cohens_kappa
      - fleiss_kappa
      - krippendorff_alpha
 
    # Per annotation scheme
    per_scheme: true
 
    # Reporting
    report_interval: 100  # Every 100 annotations
    export_file: agreement_report.json

重疊配置

yaml

quality_control:
  redundancy:
    # How many annotators per item
    annotations_per_item: 3
 
    # Minimum overlap for calculations
    min_overlap_for_agreement: 2
 
    # Sampling for agreement
    agreement_sample_size: 100  # Calculate on 100 items
    agreement_sample_method: random  # or stratified, all

計算一致性

在儀表板中

Potato 在管理儀表板中顯示一致性指標：

yaml

quality_control:
  dashboard:
    show_agreement: true
    agreement_chart: true
    update_frequency: 60  # seconds

通過 API

bash

# Get current agreement metrics
curl http://localhost:8000/api/quality/agreement
 
# Response:
{
  "overall": {
    "fleiss_kappa": 0.72,
    "krippendorff_alpha": 0.75
  },
  "per_scheme": {
    "sentiment": {
      "fleiss_kappa": 0.78,
      "krippendorff_alpha": 0.80
    },
    "topic": {
      "fleiss_kappa": 0.65,
      "krippendorff_alpha": 0.68
    }
  },
  "sample_size": 150,
  "annotator_pairs": 10
}

通過命令列

bash

# Calculate agreement from output files
potato agreement --annotations annotation_output/ --output agreement_report.json
 
# With specific metric
potato agreement --annotations annotation_output/ --metric krippendorff --level ordinal

不同標註類型的一致性

分類型（單選、多選）

yaml

quality_control:
  agreement:
    schemes:
      sentiment:
        type: nominal
        metrics: [cohens_kappa, fleiss_kappa]
 
      urgency:
        type: ordinal  # Low < Medium < High
        metrics: [krippendorff_alpha]

李克特量表

yaml

quality_control:
  agreement:
    schemes:
      quality_rating:
        type: ordinal
        metrics: [krippendorff_alpha, weighted_kappa]
 
        # Weighted kappa for ordinal
        weighting: linear  # or quadratic

Span 標註

對於 NER，span 需要特殊處理：

yaml

quality_control:
  agreement:
    schemes:
      entities:
        type: span
        span_matching: overlap  # or exact, token
 
        # What to compare
        compare: label_and_span  # or label_only, span_only
 
        # Overlap threshold for "match"
        overlap_threshold: 0.5
 
        metrics:
          - span_f1
          - span_precision
          - span_recall

排名

yaml

quality_control:
  agreement:
    schemes:
      preference_rank:
        type: ranking
        metrics:
          - kendall_tau
          - spearman_rho

成對 vs 總體一致性

成對（每對）

yaml

quality_control:
  agreement:
    pairwise: true
    output_matrix: true  # Agreement matrix
 
# Output:
# annotator1 × annotator2: κ = 0.75
# annotator1 × annotator3: κ = 0.68
# annotator2 × annotator3: κ = 0.82

總體（所有標註者）

yaml

quality_control:
  agreement:
    overall: true
    metrics:
      - fleiss_kappa  # Designed for 3+ annotators
      - krippendorff_alpha

處理低一致性

識別問題區域

yaml

quality_control:
  agreement:
    diagnostics:
      enabled: true
 
      # Items with most disagreement
      show_disagreed_items: true
      disagreement_threshold: 0.5
 
      # Labels with most confusion
      confusion_matrix: true
 
      # Annotators with low agreement
      per_annotator_agreement: true

低一致性時的操作

yaml

quality_control:
  agreement:
    alerts:
      - threshold: 0.6
        action: notify
        message: "Agreement below 0.6 - review guidelines"
 
      - threshold: 0.4
        action: pause
        message: "Agreement critically low - pausing task"
 
    # Automatic guideline reminders
    show_guidelines_on_low_agreement: true
    guideline_threshold: 0.5

完整配置

yaml

annotation_task_name: "Agreement-Tracked Annotation"
 
quality_control:
  # Redundancy setup
  redundancy:
    annotations_per_item: 3
    assignment_method: random
 
  # Agreement calculation
  agreement:
    enabled: true
 
    # Metrics
    metrics:
      - fleiss_kappa
      - krippendorff_alpha
 
    # Per-scheme configuration
    schemes:
      sentiment:
        type: nominal
        metrics: [fleiss_kappa, cohens_kappa]
 
      intensity:
        type: ordinal
        metrics: [krippendorff_alpha]
        alpha_level: ordinal
 
      entities:
        type: span
        span_matching: overlap
        overlap_threshold: 0.5
        metrics: [span_f1]
 
    # Calculation settings
    calculate_on_overlap: true
    min_overlap: 2
    sample_size: all  # or number
 
    # Pairwise analysis
    pairwise: true
    pairwise_output: agreement_matrix.csv
 
    # Diagnostics
    diagnostics:
      confusion_matrix: true
      disagreed_items: true
      per_annotator: true
 
    # Alerts
    alerts:
      - metric: fleiss_kappa
        threshold: 0.6
        action: notify
 
    # Reporting
    report_file: agreement_report.json
    report_interval: 50
 
  # Dashboard
  dashboard:
    show_agreement: true
    charts:
      - agreement_over_time
      - per_scheme_agreement
      - annotator_comparison

輸出報告

json

{
  "timestamp": "2024-10-25T15:30:00Z",
  "sample_size": 500,
  "annotators": ["ann1", "ann2", "ann3"],
 
  "overall_agreement": {
    "fleiss_kappa": 0.72,
    "krippendorff_alpha": 0.75
  },
 
  "per_scheme": {
    "sentiment": {
      "fleiss_kappa": 0.78,
      "confusion_matrix": {
        "Positive": {"Positive": 180, "Negative": 5, "Neutral": 15},
        "Negative": {"Positive": 8, "Negative": 165, "Neutral": 12},
        "Neutral": {"Positive": 12, "Negative": 10, "Neutral": 93}
      }
    }
  },
 
  "pairwise": {
    "ann1_ann2": 0.75,
    "ann1_ann3": 0.70,
    "ann2_ann3": 0.72
  },
 
  "per_annotator": {
    "ann1": {"avg_agreement": 0.73, "items_annotated": 500},
    "ann2": {"avg_agreement": 0.74, "items_annotated": 500},
    "ann3": {"avg_agreement": 0.71, "items_annotated": 500}
  },
 
  "most_disagreed_items": [
    {"id": "item_234", "disagreement_rate": 1.0},
    {"id": "item_567", "disagreement_rate": 0.67}
  ]
}

最佳實踐

儘早計算：不要等到最後才計算
使用合適的指標：區分名義型、序數型和 span
調查低一致性：通常能發現指南問題
在論文中報告：學術工作的必需項
設定閾值：提前定義可接受的水平

下一步

通過品質控制提升一致性
新增訓練階段進行校準
學習如何匯出資料並包含一致性資訊

完整的一致性文件請參閱使用者管理。