Skip to content

置信度标注

在 Potato 中给其他标注配上置信度评分，用李克特量表或滑块记录标注者对自己判断的确信程度。

置信度标注方案让标注者对自己刚做出的另一项标注给出确信程度。它把一个置信度量表（李克特或滑块）和目标标注方案关联起来，这样研究者不仅能拿到标注者选了什么，还能知道他们有多确定。

Potato confidence annotation pairing a sentiment label with a 5-point confidence scaleConfidence annotation in Potato

概览

置信度标注对研究标注质量、找出有歧义的条目、在聚合时给标签加权都很关键。配置之后，目标标注旁边会出现一个置信度控件，提示标注者说明自己对这个判断有多确定。

快速开始

yaml
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: What is the sentiment of this text?
    labels: ["Positive", "Negative", "Neutral"]
 
  - annotation_type: confidence
    name: sentiment_confidence
    description: How confident are you in your sentiment label?
    target_schema: sentiment
    scale_type: likert
    scale_points: 5

配置项

字段类型默认值说明
annotation_typestring必填必须为 "confidence"
namestring必填本方案的唯一标识符
descriptionstring必填展示给标注者的说明文字
target_schemastring可选这项置信度对应的标注方案名称
scale_typestring"likert"量表类型："likert" 为离散档位，"slider" 为连续取值
scale_pointsinteger5李克特量表的档位数（滑块模式下忽略）
labelsarray可选各档位的自定义标签（例如 ["Not confident", "Very confident"]）
slider_mininteger—滑块最小值（仅当 scale_type 为 "slider" 时使用）
slider_maxinteger—滑块最大值（仅当 scale_type 为 "slider" 时使用）
label_requirement.requiredbooleanfalse是否必须填写置信度才能继续

示例

李克特置信度量表

yaml
annotation_schemes:
  - annotation_type: radio
    name: toxicity
    description: Is this comment toxic?
    labels: ["Toxic", "Not Toxic"]
 
  - annotation_type: confidence
    name: toxicity_confidence
    description: How confident are you in your toxicity judgment?
    target_schema: toxicity
    scale_type: likert
    scale_points: 5
    labels: ["Not at all confident", "Slightly confident", "Moderately confident", "Very confident", "Extremely confident"]

滑块置信度量表

yaml
annotation_schemes:
  - annotation_type: radio
    name: stance
    description: What stance does the author take?
    labels: ["Support", "Oppose", "Neutral"]
 
  - annotation_type: confidence
    name: stance_confidence
    description: Rate your confidence from 0 (guessing) to 100 (certain).
    target_schema: stance
    scale_type: slider

必填的置信度评分

yaml
annotation_schemes:
  - annotation_type: multiselect
    name: topics
    description: Select all topics that apply.
    labels: ["Politics", "Economy", "Health", "Education"]
 
  - annotation_type: confidence
    name: topics_confidence
    description: How confident are you in your topic selections?
    target_schema: topics
    scale_type: likert
    scale_points: 3
    labels: ["Low", "Medium", "High"]

独立使用的置信度（不关联目标）

置信度标注也可以不指定目标方案，单独用于一般性的自我评估：

yaml
annotation_schemes:
  - annotation_type: confidence
    name: task_familiarity
    description: How familiar are you with this topic area?
    scale_type: likert
    scale_points: 5
    labels: ["Not familiar", "Slightly familiar", "Somewhat familiar", "Very familiar", "Expert"]

输出格式

json
{
  "toxicity_confidence": {
    "labels": {
      "confidence": 4
    }
  }
}

李克特量表的取值范围是 1 到 scale_points；滑块的取值范围是 slider_min 到 slider_max。

实践建议

  1. 尽量关联目标方案 —— 置信度和具体的标注决策挂钩时才最有价值
  2. 求简单就用李克特 —— 离散量表对标注者来说更快也更好上手
  3. 求精细就用滑块 —— 后续分析需要精确的置信度数值时用它
  4. 把置信度设为必填 —— 选填的置信度经常被跳过，数据可用性会下降
  5. 分析置信度分布 —— 低置信度的条目适合拿去仲裁或追加标注

延伸阅读

有关实现详情，请参阅源文档。