Image Captioning Evaluation

Rate AI-generated image captions for accuracy, level of detail, and hallucination.

Configuration Fileconfig.yaml

yaml

annotation_task_name: "Image Captioning Evaluation"
task_name: "Image Captioning Evaluation"
task_description: "Rate the quality of the generated caption for the image."
task_dir: "."
port: 8000

data_files:
  - "sample-data.json"

item_properties:
  id_key: "id"
  text_key: "image_url"
  image_key: "image_url"
  context_key: "caption"

annotation_schemes:
  - annotation_type: likert
    name: accuracy
    description: "Does the caption accurately describe what's in the image?"
    size: 5
    min_label: "Inaccurate"
    max_label: "Accurate"
    required: true

  - annotation_type: likert
    name: detail
    description: "How detailed is the caption?"
    size: 5
    min_label: "Too vague"
    max_label: "Appropriate detail"
    required: true

  - annotation_type: radio
    name: hallucination
    description: "Does the caption mention things not in the image?"
    labels:
      - "Yes, hallucinations present"
      - "No hallucinations"
    required: true

output_annotation_dir: "output/"
output_annotation_format: "json"

Sample Datasample-data.json

json

[
  {
    "id": "1",
    "image_url": "https://images.unsplash.com/photo-1543466835-00a7907e9de1?w=640",
    "caption": "A brown dog sitting on grass looking at the camera with its tongue out."
  },
  {
    "id": "2",
    "image_url": "https://images.unsplash.com/photo-1504208434309-cb69f4fe52b0?w=640",
    "caption": "A sunset over mountains with orange and purple clouds."
  }
]

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/evaluation/image-captioning-eval
potato start config.yaml

Details

Annotation Types

likertradio

Domain

Computer VisionNLP

Use Cases

Image CaptioningVLM Evaluation

Related Designs

Image Classification

Multi-class image classification with thumbnail preview and zoom controls.

likertmultiselect

T2I-CompBench: Compositional Text-to-Image Benchmark

T2I-CompBench is a 6,000-prompt benchmark for compositional text-to-image generation (Huang et al., NeurIPS 2023). This Potato config reproduces its human eval: Likert ratings, challenge-type tagging, and pairwise comparison.