教程阅读约需 5 分钟

创建情感分析任务

构建一个完整的情感分类任务，用单选按钮、工具提示和键盘快捷键实现高效标注。

Potato Team2026年1月15日

情感分析是最常见的 NLP 任务之一，在 Potato 里为它收集干净的标签并不费事。本教程构建的情感标注界面可以直接交到标注者手里，带上让标注更快的键盘快捷键和质量检查。

项目概述

我们要标注的是社交媒体帖子。这个界面包含：

三分类情感标注（正面、负面、中性）
每条标注的置信度评分
可选的文字说明
提速用的键盘快捷键
质量控制措施

完整配置

以下是完整的 config.yaml：

yaml

annotation_task_name: "Social Media Sentiment Analysis"
 
# Data configuration
data_files:
  - "data/tweets.json"
 
item_properties:
  id_key: id
  text_key: text
 
# Annotation interface
annotation_schemes:
  # Primary sentiment label
  - annotation_type: radio
    name: sentiment
    description: "What is the overall sentiment of this post?"
    labels:
      - name: Positive
        tooltip: "Expresses happiness, satisfaction, or approval"
        key_value: "1"
      - name: Negative
        tooltip: "Expresses sadness, frustration, or disapproval"
        key_value: "2"
      - name: Neutral
        tooltip: "Factual, objective, or lacks emotional content"
        key_value: "3"
    label_requirement:
      required: true
 
  # Confidence rating
  - annotation_type: likert
    name: confidence
    description: "How confident are you in your sentiment label?"
    size: 5
    min_label: "Not confident"
    max_label: "Very confident"
    label_requirement:
      required: true
 
  # Optional explanation
  - annotation_type: text
    name: explanation
    description: "Why did you choose this label? (Optional)"
    rows: 4
    label_requirement:
      required: false
    placeholder: "Explain your reasoning..."
 
# Guidelines
annotation_instructions: |
  ## Sentiment Annotation Guidelines
 
  ### Your Task
  Classify the sentiment expressed in each social media post.
 
  ### Labels
 
  **Positive**: The author expresses positive emotions or opinions
  - Happiness, excitement, gratitude
  - Praise, recommendations, approval
  - Examples: "Love this!", "Best day ever!", "Highly recommend"
 
  **Negative**: The author expresses negative emotions or opinions
  - Anger, frustration, sadness
  - Complaints, criticism, disapproval
  - Examples: "Terrible service", "So disappointed", "Worst experience"
 
  **Neutral**: Factual or lacking clear sentiment
  - News, announcements, questions
  - Mixed or balanced opinions
  - Examples: "The store opens at 9am", "Has anyone tried this?"
 
  ### Tips
  - Focus on the author's sentiment, not the topic
  - Sarcasm should be labeled based on intended meaning
  - When unsure, lower your confidence rating
 
# User management
automatic_assignment:
  on: true
  sampling_strategy: random
  labels_per_instance: 1
  instance_per_annotator: 100

示例数据格式

创建 data/tweets.json：

json

{"id": "t001", "text": "Just got my new laptop and I'm absolutely loving it! Best purchase of the year! #happy"}
{"id": "t002", "text": "Waited 2 hours for customer service and they still couldn't help me. Never shopping here again."}
{"id": "t003", "text": "The new coffee shop on Main Street opens tomorrow at 7am."}
{"id": "t004", "text": "This movie was okay I guess. Some good parts, some boring parts."}
{"id": "t005", "text": "Can't believe how beautiful the sunset was tonight! Nature is amazing."}

运行任务

启动标注服务器：

bash

potato start config.yaml

打开 http://localhost:8000 并登录，就可以开始标注了。

服务器起来之后，标注者看到的是这样的界面：

带有正面、负面、中性单选按钮的情感分析标注界面情感分类界面，显示单选标签、置信度量表和可选的说明字段

理解界面

主标注区域

界面显示：

待标注的文本（其中的 URL、提及和话题标签会高亮）
带工具提示的情感单选按钮
置信度 Likert 量表
可选的说明文本框

键盘工作流

标注者找到节奏之后，最快的一轮循环是：

读文本
按 1、2 或 3 选情感
点击置信度级别（或者用鼠标）
按 Enter 提交

进度跟踪

界面显示：

当前进度（例如 "15 / 100"）
预计剩余时间
会话统计

输出格式

不论 export_annotation_format 写的是什么，每位标注者的成果都会保存到 <output_annotation_dir>/<user>/user_state.json。这个设置只是在旁边额外加一份定期导出，所以取 jsonl 时，合并后的文件是 <output_annotation_dir>/exports/jsonl/annotations.jsonl，每位标注者对每条数据一条记录：

json

{
  "instance_id": "t001",
  "user_id": "john_doe",
  "labels": {
    "sentiment": {"Positive": "true"},
    "confidence": {"5": "true"},
    "explanation": {"text_box": "Clear expression of happiness with the purchase"}
  },
  "spans": {},
  "links": {}
}

添加质量控制

注意力检查

加入黄金标准条目来检验标注者是否在认真作答。这里用到的 radio 和 likert 方案的完整选项，见源文档。

yaml

attention_checks:
  enabled: true
  frequency: 10          # Every 10th item
  items_file: data/attention_checks.json
  failure_handling:
    warn_threshold: 2
    block_threshold: 5

这些条目放在 items_file 里，而不是直接写进配置，每条都带着它应得的答案：

json

[
  {
    "id": "chk1",
    "text": "I am extremely happy and satisfied! This is the best!",
    "expected_answer": { "sentiment": "Positive" }
  },
  {
    "id": "chk2",
    "text": "This is absolutely terrible and I hate it completely.",
    "expected_answer": { "sentiment": "Negative" }
  }
]

黄金标准条目用 gold_label 而不是 expected_answer，放在 gold_standards.items_file 中。注意这两个块都属于顶层：Potato 从配置根部读取 attention_checks，把它塞进 quality_control: 块里会让它根本没被读到，而且不报任何错。

标注者间一致性

研究项目可以启用多重标注：

yaml

automatic_assignment:
  on: true
  sampling_strategy: random
  labels_per_instance: 3  # Each item annotated by 3 people
  instance_per_annotator: 50

分析结果

导出并分析你的标注：

python

import json
from collections import Counter
 
records = []
with open('annotation_output/exports/jsonl/annotations.jsonl') as f:
    for line in f:
        records.append(json.loads(line))
 
def chosen(record, scheme):
    """The label picked for a single-choice scheme: the key whose value is "true"."""
    for name, value in record["labels"].get(scheme, {}).items():
        if value == "true":
            return name
    return None
 
sentiments = Counter(chosen(r, "sentiment") for r in records)
print(f"Sentiment distribution: {dict(sentiments)}")
 
# A likert stores its scale position the same way, as the label name
points = [int(p) for r in records if (p := chosen(r, "confidence"))]
print(f"Average confidence: {sum(points)/len(points):.2f}")

下一步

设置众包进行大规模标注
添加 AI 建议加速标注
实现主动学习优先处理困难案例

在我们的文档中探索更多标注类型。