Skip to content

置信度標註

在 Potato 中給其他標註配上置信度評分，用李克特量表或滑塊記錄標註者對自己判斷的確信程度。

置信度標註方案讓標註者對自己剛做出的另一項標註給出確信程度。它把一個置信度量表（李克特或滑塊）和目標標註方案關聯起來，這樣研究者不僅能拿到標註者選了什麼，還能知道他們有多確定。

Potato confidence annotation pairing a sentiment label with a 5-point confidence scaleConfidence annotation in Potato

概覽

置信度標註對研究標註品質、找出有歧義的條目、在聚合時給標籤加權都很關鍵。配置之後，目標標註旁邊會出現一個置信度控制元件，提示標註者說明自己對這個判斷有多確定。

快速開始

yaml
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: What is the sentiment of this text?
    labels: ["Positive", "Negative", "Neutral"]
 
  - annotation_type: confidence
    name: sentiment_confidence
    description: How confident are you in your sentiment label?
    target_schema: sentiment
    scale_type: likert
    scale_points: 5

配置項

欄位類型預設值說明
annotation_typestring必填必須為 "confidence"
namestring必填本方案的唯一識別符號
descriptionstring必填展示給標註者的說明文字
target_schemastring可選這項置信度對應的標註方案名稱
scale_typestring"likert"量表類型："likert" 為離散檔位，"slider" 為連續取值
scale_pointsinteger5李克特量表的檔位數（滑塊模式下忽略）
labelsarray可選各檔位的自定義標籤（例如 ["Not confident", "Very confident"]）
slider_mininteger—滑塊最小值（僅當 scale_type 為 "slider" 時使用）
slider_maxinteger—滑塊最大值（僅當 scale_type 為 "slider" 時使用）
label_requirement.requiredbooleanfalse是否必須填寫置信度才能繼續

示例

李克特置信度量表

yaml
annotation_schemes:
  - annotation_type: radio
    name: toxicity
    description: Is this comment toxic?
    labels: ["Toxic", "Not Toxic"]
 
  - annotation_type: confidence
    name: toxicity_confidence
    description: How confident are you in your toxicity judgment?
    target_schema: toxicity
    scale_type: likert
    scale_points: 5
    labels: ["Not at all confident", "Slightly confident", "Moderately confident", "Very confident", "Extremely confident"]

滑塊置信度量表

yaml
annotation_schemes:
  - annotation_type: radio
    name: stance
    description: What stance does the author take?
    labels: ["Support", "Oppose", "Neutral"]
 
  - annotation_type: confidence
    name: stance_confidence
    description: Rate your confidence from 0 (guessing) to 100 (certain).
    target_schema: stance
    scale_type: slider

必填的置信度評分

yaml
annotation_schemes:
  - annotation_type: multiselect
    name: topics
    description: Select all topics that apply.
    labels: ["Politics", "Economy", "Health", "Education"]
 
  - annotation_type: confidence
    name: topics_confidence
    description: How confident are you in your topic selections?
    target_schema: topics
    scale_type: likert
    scale_points: 3
    labels: ["Low", "Medium", "High"]

獨立使用的置信度（不關聯目標）

置信度標註也可以不指定目標方案，單獨用於一般性的自我評估：

yaml
annotation_schemes:
  - annotation_type: confidence
    name: task_familiarity
    description: How familiar are you with this topic area?
    scale_type: likert
    scale_points: 5
    labels: ["Not familiar", "Slightly familiar", "Somewhat familiar", "Very familiar", "Expert"]

輸出格式

json
{
  "toxicity_confidence": {
    "labels": {
      "confidence": 4
    }
  }
}

李克特量表的取值範圍是 1 到 scale_points；滑塊的取值範圍是 slider_min 到 slider_max。

實踐建議

  1. 儘量關聯目標方案 —— 置信度和具體的標註決策掛鉤時才最有價值
  2. 求簡單就用李克特 —— 離散量表對標註者來說更快也更好上手
  3. 求精細就用滑塊 —— 後續分析需要精確的置信度數值時用它
  4. 把置信度設為必填 —— 選填的置信度經常被跳過，資料可用性會下降
  5. 分析置信度分佈 —— 低置信度的條目適合拿去仲裁或追加標註

延伸閱讀

有關實現詳情，請參閱源文件。