Skip to content

多階段工作流

使用調查、培訓和分支邏輯構建複雜的標註工作流。

Potato 2.0 支援結構化的標註工作流，包含多個順序階段：知情同意、研究前調查、說明、培訓、標註和研究後反饋。

調查流標註工作流 — 順序階段包裹核心標註任務；除標註外每個階段都可選調查流標註工作流

可用階段

階段描述
consent知情同意收集
prestudy標註前調查（人口統計、篩查）
instructions任務指南和資訊
training帶反饋的練習題
annotation主要標註任務（必需）
poststudy標註後調查和反饋

基本配置

在配置中使用 phases 部分：

yaml
phases:
  consent:
    enabled: true
    data_file: "data/consent.json"
 
  prestudy:
    enabled: true
    data_file: "data/demographics.json"
 
  instructions:
    enabled: true
    content: "data/instructions.html"
 
  training:
    enabled: true
    data_file: "data/training.json"
    schema_name: sentiment
    passing_criteria:
      min_correct: 8
 
  # annotation phase is always enabled
 
  poststudy:
    enabled: true
    data_file: "data/feedback.json"

調查問題類型

調查階段支援以下問題類型：

單選（Radio）

json
{
  "name": "experience",
  "type": "radio",
  "description": "How much annotation experience do you have?",
  "labels": ["None", "Some (< 10 hours)", "Moderate", "Extensive"],
  "required": true
}

核取方塊/多選

json
{
  "name": "languages",
  "type": "checkbox",
  "description": "What languages do you speak fluently?",
  "labels": ["English", "Spanish", "French", "German", "Chinese", "Other"]
}

文本輸入

json
{
  "name": "occupation",
  "type": "text",
  "description": "What is your occupation?",
  "required": true
}

數字輸入

json
{
  "name": "years_experience",
  "type": "number",
  "description": "Years of professional experience",
  "min": 0,
  "max": 50
}

Likert 量表

json
{
  "name": "familiarity",
  "type": "likert",
  "description": "How familiar are you with this topic?",
  "size": 5,
  "min_label": "Not familiar",
  "max_label": "Very familiar"
}

下拉選擇

json
{
  "name": "country",
  "type": "select",
  "description": "Select your country",
  "labels": ["USA", "Canada", "UK", "Germany", "France", "Other"]
}

知情同意階段

在開始前收集知情同意：

yaml
phases:
  consent:
    enabled: true
    data_file: "data/consent.json"

consent.json：

json
[
  {
    "name": "consent_agreement",
    "type": "radio",
    "description": "I have read and understood the research consent form and agree to participate.",
    "labels": ["I agree", "I do not agree"],
    "right_label": "I agree",
    "required": true
  }
]

right_label 欄位指定繼續所需的答案。

研究前調查

收集人口統計或篩查問題：

yaml
phases:
  prestudy:
    enabled: true
    data_file: "data/demographics.json"

demographics.json：

json
[
  {
    "name": "age_range",
    "type": "radio",
    "description": "What is your age range?",
    "labels": ["18-24", "25-34", "35-44", "45-54", "55+"],
    "required": true
  },
  {
    "name": "education",
    "type": "radio",
    "description": "Highest level of education completed",
    "labels": ["High school", "Bachelor's degree", "Master's degree", "Doctoral degree", "Other"],
    "required": true
  },
  {
    "name": "english_native",
    "type": "radio",
    "description": "Is English your native language?",
    "labels": ["Yes", "No"],
    "required": true
  }
]

說明階段

顯示任務說明：

yaml
phases:
  instructions:
    enabled: true
    content: "data/instructions.html"

或使用內聯內容：

yaml
phases:
  instructions:
    enabled: true
    inline_content: |
      <h2>Task Instructions</h2>
      <p>In this task, you will classify the sentiment of product reviews.</p>
      <ul>
        <li><strong>Positive:</strong> Expresses satisfaction or praise</li>
        <li><strong>Negative:</strong> Expresses dissatisfaction or criticism</li>
        <li><strong>Neutral:</strong> Factual or mixed sentiment</li>
      </ul>

培訓階段

帶反饋的練習題（詳見培訓階段）：

yaml
phases:
  training:
    enabled: true
    data_file: "data/training.json"
    schema_name: sentiment
    passing_criteria:
      min_correct: 8
      total_questions: 10
    show_explanations: true

研究後調查

標註後收集反饋：

yaml
phases:
  poststudy:
    enabled: true
    data_file: "data/feedback.json"

feedback.json：

json
[
  {
    "name": "difficulty",
    "type": "likert",
    "description": "How difficult was this task?",
    "size": 5,
    "min_label": "Very easy",
    "max_label": "Very difficult"
  },
  {
    "name": "clarity",
    "type": "likert",
    "description": "How clear were the instructions?",
    "size": 5,
    "min_label": "Very unclear",
    "max_label": "Very clear"
  },
  {
    "name": "suggestions",
    "type": "text",
    "description": "Any suggestions for improvement?",
    "textarea": true,
    "required": false
  }
]

內建模板

Potato 包含常見調查問題的預定義標籤集：

模板標籤
countries國家列表
languages常見語言
ethnicity民族選項
religion宗教選項

在問題中使用模板：

json
{
  "name": "country",
  "type": "select",
  "description": "Select your country",
  "template": "countries"
}

自由文本欄位

在結構化問題旁新增可選的文本輸入：

json
{
  "name": "topics",
  "type": "checkbox",
  "description": "Which topics interest you?",
  "labels": ["Technology", "Sports", "Politics", "Entertainment"],
  "free_response": true,
  "free_response_label": "Other (please specify)"
}

頁面標題

自定義調查部分標題：

json
{
  "page_header": "Demographics Survey",
  "questions": [
    {"name": "age", "type": "radio", ...},
    {"name": "gender", "type": "radio", ...}
  ]
}

完整示例

yaml
task_name: "Sentiment Analysis Study"
task_dir: "."
port: 8000
 
# Data configuration
data_files:
  - "data/reviews.json"
 
item_properties:
  id_key: id
  text_key: text
 
# Annotation scheme
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What is the sentiment of this review?"
    labels:
      - Positive
      - Negative
      - Neutral
    sequential_key_binding: true
 
# Multi-phase workflow
phases:
  consent:
 
  prestudy:
 
  instructions:
 
  training:
 
  # annotation phase is always enabled
 
  poststudy:
 
# Output
output_annotation_dir: "output/"
output_annotation_format: "json"
 
# User access
allow_all_users: true

舊版配置

舊的 surveyflow 配置格式仍然支援向後相容：

yaml
surveyflow:
  enabled: true
  phases:
    - name: pre_survey
      type: survey
      questions: survey_questions.json
    - name: main_annotation
      type: annotation

但是，我們建議新項目遷移到新的 phases 格式。

最佳實踐

1. 保持調查簡潔

過長的調查會降低完成率。只關注必要的問題。

2. 複雜任務使用培訓

培訓階段提高標註品質，特別是對於細微差別的任務。

3. 設定合理的通過標準

yaml
# Too strict - may exclude good annotators
passing_criteria:
  require_all_correct: true
 
# Better - allows for learning
passing_criteria:
  min_correct: 8
  total_questions: 10

4. 提供清晰的說明

在說明階段包含示例以明確預期。

5. 測試完整流程

部署前自己完成整個工作流以發現問題。

6. 明智使用必填欄位

只在必要時將問題標記為必填——可選問題能獲得更好的回答品質。

眾包整合

對於 Prolific 或 MTurk，配置完成程式碼：

yaml
phases:
  poststudy:
    enabled: true
    data_file: "data/feedback.json"
    show_completion_code: true
    completion_code_format: "POTATO-{user_id}-{timestamp}"

詳見眾包。