Visual Genome Region Descriptions for Images

# yaml-language-server: $schema=https://potatoannotator.readthedocs.io/en/latest/schemas/potato-config.schema.json
#
# Visual Genome: Region Descriptions
#
# Paper:   Krishna, Zhu, Groth, Johnson, Hata, Kravitz, Chen, Kalantidis, Li,
#          Shamma, Bernstein & Fei-Fei, "Visual Genome: Connecting Language and
#          Vision Using Crowdsourced Dense Image Annotations", International
#          Journal of Computer Vision 123(1):32-73, 2017.
#          @article{Krishna_2017, title={Visual Genome: Connecting Language and
#          Vision Using Crowdsourced Dense Image Annotations}, volume={123},
#          number={1}, journal={International Journal of Computer Vision},
#          year={2017}, pages={32-73}, DOI={10.1007/s11263-016-0981-7}}
# DOI:     10.1007/s11263-016-0981-7
# arXiv:   https://arxiv.org/abs/1602.07332
# Dataset: https://homes.cs.washington.edu/~ranjay/visualgenome/index.html
# Licence: Visual Genome annotations are CC BY 4.0 (Hugging Face dataset card,
#          ranjaykrishna/visual_genome). Its images are COCO and YFCC100M photos
#          with their own licences, so this design does not use them.
#
# Protocol, quoted from the paper (arXiv 1602.07332, Sections 2.1 and 4.2)
# ------------------------------------------------------------------------
#   "we collect human-generated image region descriptions, with each region
#    localized by a bounding box."
#   "When a new image is added to the crowdsourcing pipeline with no
#    annotations, it is sent to a worker who is asked to draw three bounding
#    boxes and write three descriptions for the region enclosed by each box."
#   "Each description is a phrase ranging from 1 to 16 words in length
#    describing that region."
#   "Regions are allowed to have a high degree of overlap with each other when
#    the descriptions differ."
#   "The bounding box must cover all objects mentioned in the description."
#
# The task has no label set. Each drawn box carries one free-text description.
#
# Adaptations
# -----------
# - Sample images are ten photographs from Wikimedia Commons, each under CC0 or
#   CC BY and credited in the item's `credit` field. They are not Visual Genome
#   images, because those carry mixed licences. Hotlinked, not copied.
# - The original asked each worker for three boxes and kept going, with the
#   earlier descriptions shown, until an image had 50 (BLEU-filtered so a new
#   description differed from the old ones). Here annotators draw as many regions
#   as they like and never see anyone else's descriptions, which is what
#   agreement scoring needs.
# - Two annotators per item so /admin/iaa reports caption agreement.
# - The 1 to 16 word limit and the box coverage rule are in the instructions
#   and not enforced by the form.
# - Visual Genome was collected on Amazon Mechanical Turk. Potato is not tied to
#   that platform.
#
# Model training
# --------------
# The `vlm-sft` trainer (kind `caption`) fine-tunes a vision-language model on
# these regions: one example per captioned region, with the region drawn on the
# image as a red box and named by its coordinates, and the caption as the
# answer. Pick it on /admin/training (key below). It needs
# `pip install 'potato-annotation[train-vision]'` and a GPU for real models, and
# it checks the chosen model's own licence. Its caption predictions are scored
# on held-out items and never written back as prelabels, because the regions
# are the annotator's answer. It trains a model and does not serve one.

port: 8000
annotation_task_name: "Visual Genome - Region Descriptions"
task_dir: "."
output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

data_files:
  - sample-data.json

# The photo credit is the item text. The image itself is drawn by the canvas
# below, from source_field.
item_properties:
  id_key: "id"
  text_key: "credit"

require_password: false
admin_api_key: "demo-admin-key"

user_config:
  allow_all_users: true
  users: []

# Caption agreement is undefined with one annotator.
num_annotators_per_item:
  default: 2

annotation_schemes:
  # The canvas. region_caption owns the description of each region, and this
  # schema owns the drawing.
  - annotation_type: image_annotation
    name: region
    description: "Draw a bounding box around each region you want to describe."
    source_field: image_url
    tools:
      - bbox
    labels:
      - name: region
        color: "#6e56cf"

  - annotation_type: region_caption
    name: captions
    description: "Describe the region enclosed by each box."
    placeholder: "e.g. yellow fire hydrant"
    # Warn once before advancing with a region still undescribed.
    require_all: true
    # Token overlap is a lower bound. "man in red shirt" against "person
    # wearing a crimson top" scores as complete disagreement.
    agreement_distance: token

annotation_instructions: |
  Describe what is in the picture, one region at a time.

  1. Draw a bounding box around a part of the scene.
  2. Write a description of the region enclosed by the box.
  3. Repeat for other regions, three or more per image.

  A description is a short phrase of 1 to 16 words, such as "yellow fire
  hydrant" or "woman in shorts is standing behind the man". The box must cover
  every object the description mentions. Regions may overlap heavily as long
  as the descriptions differ, so a box around a whole scene and a box around
  one object in it can both be right.

  Write about different things in the image, not the same thing twice.

model_training:
  enabled: true