Skip to content
Tutorials4 min read

建立情感分析任務

構建一個完整的情感分類任務，包含單選按鈕、工具提示和鍵盤快捷鍵，實現高效標註。

Potato Team

情感分析是一項基礎的 NLP 任務，Potato 讓收集高品質情感標籤變得簡單。在本教程中，我們將構建一個功能齊全的生產級情感標註介面。

項目概述

我們將建立一個標註社交媒體帖子的介面，包含：

  • 三分類情感分類（正面、負面、中性）
  • 每個標註的置信度評分
  • 可選的文字說明
  • 鍵盤快捷鍵以提高速度
  • 品質控制措施

完整配置

以下是完整的 config.yaml：

yaml
annotation_task_name: "Social Media Sentiment Analysis"
 
# Data configuration
data_files:
  - "data/tweets.json"
 
item_properties:
  id_key: id
  text_key: text
 
# Annotation interface
annotation_schemes:
  # Primary sentiment label
  - annotation_type: radio
    name: sentiment
    description: "What is the overall sentiment of this post?"
    labels:
      - name: Positive
        tooltip: "Expresses happiness, satisfaction, or approval"
        keyboard_shortcut: "1"
      - name: Negative
        tooltip: "Expresses sadness, frustration, or disapproval"
        keyboard_shortcut: "2"
      - name: Neutral
        tooltip: "Factual, objective, or lacks emotional content"
        keyboard_shortcut: "3"
    label_requirement:
      required: true
 
  # Confidence rating
  - annotation_type: likert
    name: confidence
    description: "How confident are you in your sentiment label?"
    size: 5
    min_label: "Not confident"
    max_label: "Very confident"
    label_requirement:
      required: true
 
  # Optional explanation
  - annotation_type: text
    name: explanation
    description: "Why did you choose this label? (Optional)"
    rows: 4
    label_requirement:
      required: false
    placeholder: "Explain your reasoning..."
 
# Guidelines
annotation_guidelines:
  title: "Sentiment Annotation Guidelines"
  content: |
    ## Your Task
    Classify the sentiment expressed in each social media post.
 
    ## Labels
 
    **Positive**: The author expresses positive emotions or opinions
    - Happiness, excitement, gratitude
    - Praise, recommendations, approval
    - Examples: "Love this!", "Best day ever!", "Highly recommend"
 
    **Negative**: The author expresses negative emotions or opinions
    - Anger, frustration, sadness
    - Complaints, criticism, disapproval
    - Examples: "Terrible service", "So disappointed", "Worst experience"
 
    **Neutral**: Factual or lacking clear sentiment
    - News, announcements, questions
    - Mixed or balanced opinions
    - Examples: "The store opens at 9am", "Has anyone tried this?"
 
    ## Tips
    - Focus on the author's sentiment, not the topic
    - Sarcasm should be labeled based on intended meaning
    - When unsure, lower your confidence rating
 
# User management
automatic_assignment:
  on: true
  sampling_strategy: random
  labels_per_instance: 1
  instance_per_annotator: 100

示例資料格式

建立 data/tweets.json：

json
{"id": "t001", "text": "Just got my new laptop and I'm absolutely loving it! Best purchase of the year! #happy"}
{"id": "t002", "text": "Waited 2 hours for customer service and they still couldn't help me. Never shopping here again."}
{"id": "t003", "text": "The new coffee shop on Main Street opens tomorrow at 7am."}
{"id": "t004", "text": "This movie was okay I guess. Some good parts, some boring parts."}
{"id": "t005", "text": "Can't believe how beautiful the sunset was tonight! Nature is amazing."}

執行任務

啟動標註伺服器：

bash
potato start config.yaml

導航到 http://localhost:8000 並登入開始標註。

理解介面

主標註區域

介面顯示：

  1. 要標註的文本（高亮顯示 URL、提及、話題標籤）
  2. 帶有工具提示的情感單選按鈕
  3. 置信度 Likert 量表
  4. 可選的說明文本框

鍵盤工作流

為了最高效率：

  1. 閱讀文本
  2. 按 1、2 或 3 選擇情感
  3. 點選置信度級別（或使用滑鼠）
  4. 按 Enter 提交

進度跟蹤

介面顯示：

  • 當前進度（例如 "15 / 100"）
  • 預計剩餘時間
  • 會話統計

輸出格式

標註儲存到 annotations/username.jsonl：

json
{
  "id": "t001",
  "text": "Just got my new laptop and I'm absolutely loving it!...",
  "annotations": {
    "sentiment": "Positive",
    "confidence": 5,
    "explanation": "Clear expression of happiness with the purchase"
  },
  "annotator": "john_doe",
  "timestamp": "2026-01-15T14:30:00Z"
}

新增品質控制

注意力檢查

新增黃金標準項目以驗證標註者注意力：

yaml
quality_control:
  attention_checks:
    enabled: true
    frequency: 10  # Every 10th item
    items:
      - text: "I am extremely happy and satisfied! This is the best!"
        expected:
          sentiment: "Positive"
      - text: "This is absolutely terrible and I hate it completely."
        expected:
          sentiment: "Negative"

標註者間一致性

對於研究項目，啟用多重標註：

yaml
automatic_assignment:
  on: true
  sampling_strategy: random
  labels_per_instance: 3  # Each item annotated by 3 people
  instance_per_annotator: 50

分析結果

匯出並分析您的標註：

python
import json
from collections import Counter
 
# Load annotations
annotations = []
with open('annotations/annotator1.jsonl') as f:
    for line in f:
        annotations.append(json.loads(line))
 
# Sentiment distribution
sentiments = Counter(a['annotations']['sentiment'] for a in annotations)
print(f"Sentiment distribution: {dict(sentiments)}")
 
# Average confidence
confidences = [a['annotations']['confidence'] for a in annotations]
print(f"Average confidence: {sum(confidences)/len(confidences):.2f}")

下一步

在我們的文件中探索更多標註類型。