Skip to content

培訓階段

在主要任務之前使用練習題培訓和篩選標註者。

Potato 2.0 包含一個可選的培訓階段，幫助在標註者開始主要標註任務之前對其進行資格篩選。標註者回答具有已知正確答案的練習題，並獲得即時的表現反饋。

訓練階段如何為標註者把關 — 在有已知答案的題目上練習；只有合格的標註者進入真實任務訓練階段如何為標註者把關

用例

  • 確保標註者理解任務
  • 過濾低品質標註者
  • 在真實標註前提供指導性練習
  • 收集基線品質指標
  • 通過示例教授標註指南

工作原理

  1. 標註者完成一組培訓題
  2. 每次回答後獲得即時反饋
  3. 進度根據通過標準進行追蹤
  4. 只有通過的標註者才能進入主要任務

配置

基本設定

yaml
phases:
  training:
    enabled: true
    data_file: "data/training_data.json"
    schema_name: sentiment  # Which annotation scheme to train
 
    # Passing criteria
    passing_criteria:
      min_correct: 8  # Must get at least 8 correct
      total_questions: 10

完整配置

yaml
phases:
  training:
    enabled: true
    data_file: "data/training_data.json"
    schema_name: sentiment
 
    passing_criteria:
      # Different criteria options (choose one or combine)
      min_correct: 8
      require_all_correct: false
      max_mistakes: 3
      max_mistakes_per_question: 2
 
    # Allow retries
    retries:
      enabled: true
      max_retries: 3
 
    # Show explanations for incorrect answers
    show_explanations: true
 
    # Randomize question order
    randomize: true

通過標準

你可以為培訓階段設定各種通過標準：

最少正確數

yaml
passing_criteria:
  min_correct: 8
  total_questions: 10

標註者必須在 10 題中至少答對 8 題。

要求全部正確

yaml
passing_criteria:
  require_all_correct: true

標註者必須答對每一題才能通過。

最大錯誤數

yaml
passing_criteria:
  max_mistakes: 3

標註者在累計 3 次錯誤後被取消資格。

每題最大錯誤數

yaml
passing_criteria:
  max_mistakes_per_question: 2

標註者在任何單題上犯 2 次錯誤後被取消資格。

組合標準

yaml
passing_criteria:
  min_correct: 8
  max_mistakes_per_question: 3

必須答對 8 題，且任何單題不能錯超過 3 次。

培訓資料格式

培訓資料必須包含正確答案和可選的解釋：

json
[
  {
    "id": "train_1",
    "text": "I absolutely love this product! Best purchase ever!",
    "correct_answers": {
      "sentiment": "Positive"
    },
    "explanation": "This text expresses strong positive sentiment with words like 'love' and 'best'."
  },
  {
    "id": "train_2",
    "text": "This is the worst service I've ever experienced.",
    "correct_answers": {
      "sentiment": "Negative"
    },
    "explanation": "The words 'worst' and the overall complaint indicate negative sentiment."
  },
  {
    "id": "train_3",
    "text": "The package arrived on time.",
    "correct_answers": {
      "sentiment": "Neutral"
    },
    "explanation": "This is a factual statement without emotional indicators."
  }
]

多模式培訓

對於有多個標註模式的任務：

json
{
  "id": "train_1",
  "text": "Apple announced new iPhone features yesterday.",
  "correct_answers": {
    "sentiment": "Neutral",
    "topic": "Technology"
  },
  "explanation": {
    "sentiment": "This is a factual news statement.",
    "topic": "The text discusses Apple and iPhone, which are tech topics."
  }
}

使用者體驗

培訓流程

  1. 使用者看到"培訓階段"指示器
  2. 顯示帶標註表單的問題
  3. 使用者提交答案
  4. 立即顯示反饋：
    • 正確：綠色對勾，進入下一題
    • 錯誤：紅色叉號，顯示解釋，重試選項

反饋展示

當標註者回答錯誤時：

  • 高亮正確答案
  • 顯示提供的解釋
  • 出現重試按鈕（如果啟用重試）
  • 顯示通過標準的進度

管理員監控

在管理員儀表板中追蹤培訓表現：

  • 完成率
  • 平均正確答案數
  • 通過/失敗率
  • 培訓花費時間
  • 每題準確率

通過 /admin API 端點訪問：

text
GET /api/admin/training/stats
GET /api/admin/training/user/{user_id}

示例：情感分析培訓

yaml
task_name: "Sentiment Analysis"
task_dir: "."
port: 8000
 
# Main annotation data
data_files:
  - "data/reviews.json"
 
item_properties:
  id_key: id
  text_key: text
 
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What is the sentiment of this review?"
    labels:
      - Positive
      - Negative
      - Neutral
 
# Training phase configuration
phases:
  training:
 
 
 
 
output_annotation_dir: "output/"
output_annotation_format: "json"
allow_all_users: true

示例：NER 培訓

yaml
annotation_schemes:
  - annotation_type: span
    name: entities
    description: "Highlight named entities"
    labels:
      - Person
      - Organization
      - Location
      - Date
 
phases:
  training:

片段標註的培訓資料：

json
{
  "id": "train_1",
  "text": "Tim Cook announced that Apple will open a new store in New York on March 15.",
  "correct_answers": {
    "entities": [
      {"start": 0, "end": 8, "label": "Person"},
      {"start": 24, "end": 29, "label": "Organization"},
      {"start": 54, "end": 62, "label": "Location"},
      {"start": 66, "end": 74, "label": "Date"}
    ]
  },
  "explanation": "Tim Cook is a Person, Apple is an Organization, New York is a Location, and March 15 is a Date."
}

最佳實踐

1. 從簡單開始

先從簡單的例子開始，再引入邊界情況：

json
[
  {"text": "I love this!", "correct_answers": {"sentiment": "Positive"}},
  {"text": "I hate this!", "correct_answers": {"sentiment": "Negative"}},
  {"text": "It arrived yesterday.", "correct_answers": {"sentiment": "Neutral"}}
]

2. 覆蓋所有標籤

確保培訓包含每個可能標籤的示例：

json
[
  {"correct_answers": {"sentiment": "Positive"}},
  {"correct_answers": {"sentiment": "Negative"}},
  {"correct_answers": {"sentiment": "Neutral"}}
]

3. 編寫清晰的解釋

解釋應該教授標註指南：

json
{
  "explanation": "While this text mentions a problem, the overall tone is constructive and the reviewer expresses satisfaction with the resolution. This makes it Positive rather than Negative."
}

4. 設定合理的標準

不必要時不要要求完美：

yaml
# Too strict - may lose good annotators
passing_criteria:
  require_all_correct: true
 
# Better - allows for learning
passing_criteria:
  min_correct: 8
  total_questions: 10

5. 包含邊界情況

新增棘手的例子以準備標註者：

json
{
  "text": "Not bad at all, I guess it could be worse.",
  "correct_answers": {"sentiment": "Neutral"},
  "explanation": "Despite negative words like 'not bad' and 'worse', this is actually a lukewarm endorsement - neutral rather than positive or negative."
}

與工作流整合

培訓與多階段工作流整合：

yaml
phases:
  consent:
    enabled: true
    data_file: "data/consent.json"
 
  prestudy:
    enabled: true
    data_file: "data/demographics.json"
 
  instructions:
    enabled: true
    content: "data/instructions.html"
 
  training:
    enabled: true
    data_file: "data/training.json"
    schema_name: sentiment
    passing_criteria:
      min_correct: 8
 
  annotation:
    # Main task - always enabled
    enabled: true
 
  poststudy:
    enabled: true
    data_file: "data/feedback.json"

效能注意事項

  • 培訓資料在啟動時載入
  • 進度按會話儲存在記憶體中
  • 對主標註效能影響最小
  • 考慮將複雜培訓分為多個階段

延伸閱讀

有關實現細節，請參閱原始碼文件。