Skip to content

眾包整合

與 Prolific、MTurk 和其他眾包平臺進行整合。

Potato 與 Prolific 和 Amazon Mechanical Turk 等眾包平臺無縫整合，用於大規模標註任務。

Prolific 整合

基本設定

yaml
crowdsourcing:
  platform: prolific
  enabled: true
  completion_code: "POTATO2024"  # Code shown on completion

URL 參數

Prolific 通過 URL 參數傳遞參與者資訊：

yaml
crowdsourcing:
  platform: prolific
  url_params:
    - PROLIFIC_PID    # Participant ID
    - STUDY_ID        # Study ID
    - SESSION_ID      # Session ID

工作者通過以下方式訪問：

text
https://your-server.com/?PROLIFIC_PID=xxx&STUDY_ID=xxx&SESSION_ID=xxx

Prolific 配置

在你的 Prolific 研究設定中：

  1. 將研究 URL設定為你的 Potato 伺服器
  2. 新增 URL 參數：?PROLIFIC_PID={{%PROLIFIC_PID%}}&STUDY_ID={{%STUDY_ID%}}&SESSION_ID={{%SESSION_ID%}}
  3. 將完成程式碼設定為與你的配置匹配

驗證

驗證 Prolific 參與者：

yaml
crowdsourcing:
  platform: prolific
  validate_participant: true
  completion_code: "POTATO2024"

Amazon MTurk 整合

基本設定

yaml
crowdsourcing:
  platform: mturk
  enabled: true

HIT 配置

建立指向你伺服器的外部問題 HIT：

xml
<?xml version="1.0" encoding="UTF-8"?>
<ExternalQuestion xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2006-07-14/ExternalQuestion.xsd">
  <ExternalURL>https://your-server.com/?workerId=${workerId}&amp;assignmentId=${assignmentId}&amp;hitId=${hitId}</ExternalURL>
  <FrameHeight>800</FrameHeight>
</ExternalQuestion>

URL 參數

yaml
crowdsourcing:
  platform: mturk
  url_params:
    - workerId
    - assignmentId
    - hitId

沙盒測試

先用 MTurk 沙盒進行測試：

yaml
crowdsourcing:
  platform: mturk
  sandbox: true  # Use sandbox environment

工作者管理

追蹤工作者

yaml
crowdsourcing:
  track_workers: true
  worker_id_field: worker_id

限制每個工作者的實例數

yaml
instances_per_annotator: 50

阻止重複工作者

防止工作者重複參與任務：

yaml
crowdsourcing:
  prevent_retakes: true

品質控制

注意力檢查

插入測試問題：

yaml
attention_checks:
  enabled: true
  frequency: 10  # Every 10 instances
  fail_threshold: 2
  action: warn  # or 'block'

金標準問題

json
{
  "id": "gold_1",
  "text": "The sky is typically blue during a clear day.",
  "gold_label": "True",
  "is_gold": true
}
yaml
quality_control:
  gold_questions: true
  gold_percentage: 10  # 10% of instances
  min_gold_accuracy: 70

時間限制

yaml
crowdsourcing:
  min_time_per_instance: 5  # seconds
  max_time_total: 3600  # 1 hour

拒絕低品質工作

yaml
quality_control:
  auto_reject:
    enabled: true
    conditions:
      - gold_accuracy_below: 50
      - completion_time_under: 300  # seconds

完成處理

顯示完成程式碼

yaml
completion:
  show_code: true
  code: "POTATO2024"
  message: "Thank you! Your completion code is: {code}"

完成後重定向

yaml
completion:
  redirect: true
  redirect_url: "https://prolific.co/submissions/complete?cc={code}"

自定義完成頁面

yaml
completion:
  custom_template: templates/completion.html

支付等級

基於品質

yaml
payment:
  tiers:
    - name: bonus
      condition:
        gold_accuracy_above: 90
      amount: 0.50
    - name: standard
      condition:
        gold_accuracy_above: 70
      amount: 0.00
    - name: reject
      condition:
        gold_accuracy_below: 50

完整示例：Prolific 研究

yaml
task_name: "Sentiment Analysis Study"
 
# Crowdsourcing settings
crowdsourcing:
  platform: prolific
  enabled: true
  completion_code: "SENT2024"
  url_params:
    - PROLIFIC_PID
    - STUDY_ID
    - SESSION_ID
  prevent_retakes: true
 
# Open access for crowdworkers
allow_all_users: true
 
# Task assignment
instances_per_annotator: 50
annotation_per_instance: 3
 
# Quality control
attention_checks:
  enabled: true
  frequency: 10
  fail_threshold: 2
 
quality_control:
  gold_questions: true
  gold_percentage: 5
  min_gold_accuracy: 70
 
# Data
data_files:
  - path: data/main.json
    text_field: text
 
# Annotation scheme
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What is the sentiment?"
    labels:
      - Positive
      - Negative
      - Neutral
    sequential_key_binding:
      Positive: "1"
      Negative: "2"
      Neutral: "3"
 
# Completion
completion:
  show_code: true
  code: "SENT2024"
  message: |
    ## Thank you for participating!
 
    Your completion code is: **{code}**
 
    Please return to Prolific and enter this code to receive payment.

完整示例：MTurk HIT

yaml
task_name: "Image Classification HIT"
 
crowdsourcing:
  platform: mturk
  enabled: true
  url_params:
    - workerId
    - assignmentId
    - hitId
 
allow_all_users: true
instances_per_annotator: 20
 
# Time constraints
crowdsourcing:
  min_time_per_instance: 3
  max_time_total: 1800
 
# MTurk form submission
completion:
  mturk_submit: true
  submit_url: "https://www.mturk.com/mturk/externalSubmit"
 
annotation_schemes:
  - annotation_type: radio
    name: category
    description: "What is shown in this image?"
    labels:
      - Cat
      - Dog
      - Bird
      - Other

監控工作者

管理員儀表板

yaml
admin_users:
  - researcher@university.edu
 
admin_dashboard:
  enabled: true
  show_worker_stats: true

訪問 /admin 可檢視：

  • 工作者完成率
  • 每個實例的平均時間
  • 金標準準確率
  • 注意力檢查結果

匯出工作者資料

bash
potato export-workers config.yaml --output workers.csv

最佳實踐

  1. 充分測試 - 先用小組進行試點
  2. 設定公平薪酬 - 計算預估時間並公平支付
  3. 清晰的說明 - 包含示例和邊界情況
  4. 使用注意力檢查 - 捕獲隨機點選行為
  5. 包含金標準問題 - 驗證理解程度
  6. 即時監控 - 儘早發現問題
  7. 計劃拒絕策略 - 預先設定明確的品質標準
  8. 溝通問題 - 就問題與工作者聯絡
  9. 根據反饋迭代 - 根據工作者意見進行改進
  10. 定期匯出資料 - 不要等到最後

延伸閱讀

有關實現細節，請參閱原始碼文件。