How to annotate RefCOCO

# yaml-language-server: $schema=https://potatoannotator.readthedocs.io/en/latest/schemas/potato-config.schema.json
#
# RefCOCO — Referring Expression Grounding
#
# Paper:   Yu, Poirson, Yang, Berg & Berg, "Modeling Context in Referring
#          Expressions", ECCV 2016 — https://arxiv.org/abs/1608.00272
# Dataset: https://github.com/lichengunc/refer
#
# Referring expression comprehension: given an image and a phrase naming one
# specific object, locate that object. RefCOCO's expressions were collected
# with the ReferItGame two-player interface (Kazemzadeh et al., EMNLP 2014),
# where a second player had to click the region the phrase described, so
# expressions that did not identify their target unambiguously were filtered
# out during play.
#
# RefCOCO:  142,209 expressions for 50,000 objects across 19,994 images.
# RefCOCO+: 141,564 expressions for 49,856 objects across 19,992 images.
#           Same setup, but absolute spatial words ("left", "right") are
#           forbidden, so expressions rely on appearance and are harder.
# Images come from MS COCO.
#
# Reproduction notes
# ------------------
# `grounding_eval` owns the phrase-to-region binding; it does not draw, so an
# `image_annotation` schema sits beside it to own the canvas. Each item carries
# every expression for its image in `expressions`, which matches RefCOCO's
# structure (several expressions per object) and means the annotator works
# through them one at a time against a single image.
#
# `region_type: box` is the RefCOCO convention. Set `point` instead to collect
# Molmo-style pointing, which Potato scores as a point-in-region hit rate
# rather than IoU — a point has no area, so IoU against one is always zero.
#
# Two annotators per item, so grounding agreement is defined. Potato scores
# grounding by IoU at four thresholds and counts ungroundedness separately.

port: 8000
annotation_task_name: "RefCOCO - Referring Expression Grounding"
task_dir: "."
output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "image_url"

instance_display:
  fields:
    - key: image_url
      type: image
      label: "Image"

user_config:
  allow_all_users: true
  users: []

# Grounding agreement needs at least two independent annotations per image.
num_annotators_per_item:
  default: 2

annotation_schemes:
  # The canvas. grounding_eval binds phrases to regions but does not draw them,
  # so it needs an image schema alongside it.
  - annotation_type: image_annotation
    name: region
    description: "Draw a tight box around the object the selected phrase names."
    source_field: image_url
    tools:
      - bbox
    labels:
      - name: referent
        color: "#6e56cf"

  - annotation_type: grounding_eval
    name: grounding
    description: "What does each referring expression refer to?"
    region_type: box
    expressions_field: expressions
    # Warn once before advancing with phrases still unanswered. An expression
    # the annotator never answered is excluded from scoring rather than counted
    # as a miss, so leaving one blank is not the same as saying "not present".
    require_all: false

  - annotation_type: text
    name: clarification
    description: "If an expression is ambiguous, write a clearer one. (optional)"
    placeholder: "Optional: a more precise referring expression"

annotation_instructions: |
  Each image comes with one or more referring expressions. Every expression
  names exactly one object.

  For each expression:
  1. Select it from the list.
  2. Find the object it names.
  3. Draw a tight bounding box around that object.

  Draw the box as tight as the object allows — include the whole object and as
  little else as possible.

  Expressions often disambiguate by comparison to other objects in the image
  ("the dog on the left", "the taller man"), so read the whole scene before
  deciding. RefCOCO+ expressions avoid absolute spatial words, so they lean on
  colour, size and other attributes instead.

  If an expression genuinely does not pick out anything in the image, mark it
  as not present rather than boxing your best guess.