Skip to content
Showcase/Image Captioning Evaluation
beginnerimage

Image Captioning Evaluation

Rate AI-generated image captions for accuracy, fluency, and detail.

Labels:outdoornatureurbanpeopleanimal+−

配置文件config.yaml

annotation_task_name: "Image Captioning Evaluation"
task_name: "Image Captioning Evaluation"
task_description: "Rate the quality of the generated caption for the image."
task_dir: "."
port: 8000

data_files:
  - "sample-data.json"

item_properties:
  id_key: "id"
  text_key: "image_url"
  image_key: "image_url"
  context_key: "caption"

annotation_schemes:
  - annotation_type: likert
    name: accuracy
    description: "Does the caption accurately describe what's in the image?"
    size: 5
    min_label: "Inaccurate"
    max_label: "Accurate"
    required: true

  - annotation_type: likert
    name: detail
    description: "How detailed is the caption?"
    size: 5
    min_label: "Too vague"
    max_label: "Appropriate detail"
    required: true

  - annotation_type: radio
    name: hallucination
    description: "Does the caption mention things not in the image?"
    labels:
      - "Yes, hallucinations present"
      - "No hallucinations"
    required: true

output_annotation_dir: "output/"
output_annotation_format: "json"

示例数据sample-data.json

[
  {
    "id": "1",
    "image_url": "https://images.unsplash.com/photo-1543466835-00a7907e9de1?w=640",
    "caption": "A brown dog sitting on grass looking at the camera with its tongue out."
  },
  {
    "id": "2",
    "image_url": "https://images.unsplash.com/photo-1504208434309-cb69f4fe52b0?w=640",
    "caption": "A sunset over mountains with orange and purple clouds."
  }
]

获取此设计

View on GitHub

Clone or download from the repository

快速开始：

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/evaluation/image-captioning-eval
potato start config.yaml

详情

标注类型

likertradio

领域

Computer VisionNLP

应用场景

Image CaptioningVLM Evaluation

标签

captioningimageevaluationvlm

发现问题或想改进此设计？

提交 Issue

相关设计

Image Classification

Multi-class image classification with thumbnail preview and zoom controls.

likertmultiselect

T2I-CompBench Text-to-Image Evaluation

Compositional text-to-image generation evaluation based on T2I-CompBench (Huang et al., NeurIPS 2023). Annotators rate image quality on a Likert scale, classify the compositional challenge type, and compare pairs of generated images via pairwise preference.

likertradio

AnnoMI Counselling Dialogue Annotation

Annotation of motivational interviewing counselling dialogues based on the AnnoMI dataset. Annotators label therapist and client utterances for MI techniques (open questions, reflections, affirmations) and client change talk (sustain talk, change talk), with quality ratings for therapeutic interactions.

radiomultiselect