nuScenes 3D Cuboid Annotation on Lidar Point Clouds

# yaml-language-server: $schema=https://potatoannotator.readthedocs.io/en/latest/schemas/potato-config.schema.json
#
# nuScenes: 3D cuboids with class and attributes on a lidar point cloud
#
# Dataset:  nuScenes, Caesar et al., Motional (nuTonomy)
# Paper:    Caesar, Bankiti, Lang, Vora, Liong, Xu, Krishnan, Pan, Baldan and
#           Beijbom, "nuScenes: A Multimodal Dataset for Autonomous Driving",
#           CVPR 2020, pp. 11618-11628
#           https://arxiv.org/abs/1903.11027
# DOI:      10.1109/CVPR42600.2020.01164
# Dataset:  https://www.nuscenes.org/nuscenes
# Licence:  CC BY-NC-SA 4.0 (paper, section 1: "anyone can use this dataset for
#           non-commercial research purposes"; terms at
#           https://www.nuscenes.org/terms-of-use). nuScenes data is NOT
#           redistributed here.
# Citation (Crossref BibTeX):
#   @inproceedings{Caesar_2020, title={nuScenes: A Multimodal Dataset for
#   Autonomous Driving}, DOI={10.1109/cvpr42600.2020.01164},
#   booktitle={2020 IEEE/CVF Conference on Computer Vision and Pattern
#   Recognition (CVPR)}, publisher={IEEE}, author={Caesar, Holger and Bankiti,
#   Varun and Lang, Alex H. and Vora, Sourabh and Liong, Venice Erin and Xu,
#   Qiang and Krishnan, Anush and Pan, Yu and Baldan, Giancarlo and Beijbom,
#   Oscar}, year={2020}, month=June, pages={11618-11628}}
#
# The ten point clouds in media/clouds/ are SYNTHETIC. They were generated for
# this showcase from a handful of box-shaped objects on a flat road, so they are
# illustrative and are not nuScenes scans. nuScenes is non-commercial and
# share-alike (CC BY-NC-SA 4.0), and this showcase carries no data under those
# terms, so no nuScenes lidar sweep is bundled. See media/LICENSE-media.txt.
#
# Protocol (paper, section 2, "Data annotation"):
#   "we sample keyframes (image, lidar, radar) at 2Hz. We annotate each of the
#   23 object classes in every keyframe with a semantic category, attributes
#   (visibility, activity, and pose) and a cuboid modeled as x, y, z, width,
#   length, height and yaw angle. We annotate objects continuously throughout
#   each scene if they are covered by at least one lidar or radar point."
#
# Label glossary: the 23 labels of the nuScenes annotator instructions (the list
# numbers the two sub-entries of Truck too, 25 numbers in all)
# (https://github.com/nutonomy/nuscenes-devkit/blob/master/docs/instructions_nuscenes.md),
# quoted by name. Truck's two sub-entries, Pickup Truck and Front Of Semi Truck,
# are described under Truck in the instructions and are not separate labels here.
#   Car or Van or SUV: "Vehicle designed primarily for personal use, e.g.
#     sedans, hatch-backs, wagons, vans, mini-vans, SUVs and jeeps."
#   Truck: "Vehicles primarily designed to haul cargo including pick-ups,
#     lorrys, trucks and semi-tractors."
#   Bendy Bus: "Buses and shuttles designed to carry more than 10 people and
#     comprises two or more rigid sections linked by a pivoting joint.
#     Annotate each section of the bendy bus individually."
#   Rigid Bus: "Rigid buses and shuttles designed to carry more than 10 people."
#   Trailer: "Any vehicle trailer, both for trucks, cars and motorcycles
#     (regardless of whether currently being towed or not)."
#   Traffic Cone: "All types of traffic cones."
#   Temporary Traffic Barrier: "If there are multiple barriers either connected
#     or just placed next to each other, they should be annotated separately."
#   Debris: "Debris or movable object that is too large to be driven over
#     safely."
#   The remaining labels follow the instructions' list verbatim by name.
#
# Attributes (instructions, "Attributes"):
#   Visibility, for every object: 0%-40%, 41%-60%, 61%-80%, 81%-100% of the
#     object "visible in panoramic view of all cameras".
#   Vehicle Activity, "for each vehicle with four or more wheels": Parked,
#     Stopped, Moving.
#   Has Rider, "for each bicycle, motorcycle and portable personal mobility
#     vehicle": Yes, No.
#   Human Activity, "for each human in the scene": Sitting or Lying Down,
#     Standing, Moving.
#
# Special rules quoted for the annotator (instructions, "Special Rules"):
#   "Cuboids must be very tight. Draw the cuboid as close as possible to the
#   edge of the object without excluding any LIDAR points."
#   "Label any target object containing at least 1 LIDAR or RADAR point, as
#   long as you can be reasonably sure you know the location and shape of the
#   object."
#
# Adaptations:
#   - Lidar only. nuScenes annotators also see six camera images and radar;
#     this design supplies no calibration, so the camera panels do not appear.
#   - Single synthetic sweeps, not 2 Hz keyframe sequences in 20 s scenes, and
#     no instance tracking across frames.
#   - Cuboids are stored in the sensor frame in metres, in Potato's own cuboid
#     format, not as nuScenes translation, size and rotation in the global frame.
#   - Visibility is judged from cameras in nuScenes. With no images here, the
#     annotator estimates it from the point cloud, so the attribute is for
#     practice and does not reproduce the protocol.
#   - Which labels carry which attribute is read from the instruction sentences
#     above. The instructions say "vehicle with four or more wheels" and "each
#     human", and the label lists below are our reading of those phrases.
#   - The paper reports 8 attributes and names visibility, activity and pose.
#     The annotator instructions describe four attribute groups (Visibility,
#     Vehicle Activity, Has Rider, Human Activity), and those four are the
#     fields here. The paper's pose attribute is not in the instructions and
#     is left out.
#
# Model training (trainer: external-3d):
#   Potato does not ship a 3D detector. The external-3d trainer writes the
#   annotated scans in OpenPCDet's custom dataset layout (points/<id>.npy and
#   labels/<id>.txt, one line "x y z dx dy dz heading label" per cuboid), runs
#   the train_command and predict_command you supply, and reads the boxes your
#   detector writes (x y z dx dy dz heading label score) back as cuboid_3d
#   objects marked as model output. It does not install, serve or export a
#   detector, handles only single scans (not sequences), and keeps heading
#   about +z only (no pitch or roll). Set the commands in the trainer
#   parameters on /admin/training. This config enables the loop and leaves the
#   commands empty, because they depend on a detector installed on your own host.
#
# Run from the folder holding this file:
#   python -m potato.flask_server start config.yaml -p 8000
#
# Controls: press `c` for the box tool, pick a class, and drag on the ground
# plane to place a box. Left mouse button orbits, right button pans, scroll
# zooms.

port: 8000
annotation_task_name: "nuScenes 3D cuboids (synthetic clouds)"
task_dir: "."
media_directory: media

data_files:
  - sample-data.json

item_properties:
  id_key: id
  text_key: point_cloud

output_annotation_dir: "annotation_output/"
export_annotation_format: jsonl

require_password: false
admin_api_key: "demo-admin-key"

user_config:
  allow_all_users: true
  users: []

annotation_instructions: |
  Draw a 3D box around every object from the label list and give it one of the
  23 nuScenes classes. Draw one box per object. Cuboids must be very tight:
  as close to the edge of the object as possible without excluding any lidar
  points. Label any object with at least one lidar point when you can be
  reasonably sure of its location and shape. Then fill in the attributes in the
  panel under the viewer. The clouds are synthetic and illustrative, not
  nuScenes scans.

annotation_schemes:
  - annotation_type: spatial_annotation
    name: objects
    description: "Draw a tight 3D box around every object from the nuScenes label list and give it a class and attributes."
    source_field: point_cloud
    tools:
      - cuboid_3d
    labels:
      - "Car or Van or SUV"
      - "Truck"
      - "Bendy Bus"
      - "Rigid Bus"
      - "Construction Vehicle"
      - "Motorcycle"
      - "Bicycle"
      - "Bicycle Rack"
      - "Trailer"
      - "Police Vehicle"
      - "Ambulance"
      - "Adult Pedestrian"
      - "Child Pedestrian"
      - "Construction Worker"
      - "Stroller"
      - "Wheelchair"
      - "Portable Personal Mobility Vehicle"
      - "Police Officer"
      - "Animal"
      - "Traffic Cone"
      - "Temporary Traffic Barrier"
      - "Pushable Pullable Object"
      - "Debris"
    attributes:
      - name: visibility
        type: select
        description: "Percentage of the object visible in the panoramic view of all cameras."
        options: ["0%-40%", "41%-60%", "61%-80%", "81%-100%"]
        required: true
      - name: vehicle_activity
        type: select
        description: "For each vehicle with four or more wheels."
        options: ["Parked", "Stopped", "Moving"]
        labels: ["Car or Van or SUV", "Truck", "Bendy Bus", "Rigid Bus", "Construction Vehicle", "Police Vehicle", "Ambulance"]
        required: true
      - name: has_rider
        type: select
        description: "For each bicycle, motorcycle and portable personal mobility vehicle."
        options: ["Yes", "No"]
        labels: ["Bicycle", "Motorcycle", "Portable Personal Mobility Vehicle"]
        required: true
      - name: human_activity
        type: select
        description: "For each human in the scene."
        options: ["Sitting or Lying Down", "Standing", "Moving"]
        labels: ["Adult Pedestrian", "Child Pedestrian", "Construction Worker", "Police Officer"]
        required: true
    color_mode: height
    point_size: 2.0
    lod: true
    slab_thickness: 2.0
    default_box_height: 1.7
    fit_box_height: true

# external-3d consumes spatial_annotation cuboids. Pick it on /admin/training
# and set train_command and predict_command for your own detector there.
model_training:
  enabled: true
  writeback:
    schemas: ["objects"]
    include_unlabeled: true
    min_confidence: 0.5