How to Annotate 3D Point Clouds

Annotating a point cloud means placing oriented 3D boxes on sparse, unevenly sampled data where a distant object may be a few dozen returns. The hard parts are not the drawing: they are keeping the viewer interactive, giving the annotator a way to check their work, and choosing a rotation representation that survives a round trip.

Formats

Parse server-side rather than in the browser. Four formats means four sets of endianness and record-layout bugs, and re-parsing a two-million-point scan on every page load.

Keeping it interactive

A raw scan is too large to render naively. Two things make it workable:

• Octree level of detail. Points load by region and by distance to the camera. This is on by default in Potato (lod: true).
• A decimation cap (max_points). Treat it as a viewer setting, not a data setting.

That distinction bit once and is worth repeating: per-point segment labels used to be stored as indices into the decimated cloud, so lowering max_points silently re-pointed existing labels at different points. Indices should always refer to the source file.

Give annotators a 2D check

A car at 40 metres is a few dozen returns. Whether the box is tight, the right length, and correctly rotated are close to unanswerable from the cloud alone, and obvious in a camera image.

So: edit in 3D, verify in 2D. Supply calibration per item and project every box into each camera view. Without that, 3D labelling has no feedback loop — a box that looks correct from the orbit camera can be metres off along the view axis with nothing to say so.

Orthographic slab panels give the other half: three axis-aligned views for precise adjustment, fully keyboard-driven.

Store rotation as a quaternion

Most formats store a single yaw angle. Storing a full quaternion internally looks like over-engineering until a round trip loses something.

It makes KITTI import lossless, including the roughly 0.85° camera-to-lidar mounting tilt a yaw-only field discards without comment. Export back to KITTI still has to drop pitch and roll — the right behaviour is to report how much orientation was discarded rather than flattening the box silently.

Cross that gap in the format-specific code, loudly, rather than in the storage layer, silently.

Frames are where this goes wrong

Three conventions that are easy to get wrong and hard to notice:

• Reference frame versus camera frame. Using a camera's transform where the rectified reference frame is wanted shifts every box by that camera's stereo baseline. Systematic, a few centimetres, and easily mistaken for annotator sloppiness.
• Location is the bottom face in KITTI, not the centre. Reading it as the centre puts every object half its own height underground.
• Dimension order and axis assignment. A round-trip test cannot catch a 90° rotation error here, because the inverse conversion makes the same wrong assumption and the two agree with each other. Only a comparison against the reference implementation's own corner formula catches it.

Measuring agreement

Use exact rotated 3D IoU. An axis-aligned approximation is fine for level automotive data and wrong for drone, handheld and indoor scans, where it reports disagreement that is an artefact of the measure rather than of the annotators.

As with 2D, split the question: did annotators find the same objects, label them the same, and place them the same. See measuring agreement on bounding boxes.

Config

annotation_schemes:
  - annotation_type: spatial_annotation
    name: objects
    description: "Put a 3D box around every vehicle and pedestrian."
    source_field: point_cloud
    calibration_field: calibration
    tools: [cuboid_3d, point_3d]
    labels:
      - {name: car, color: "#FF6B6B", key_value: "1"}
      - {name: pedestrian, color: "#FFD93D", key_value: "2"}
    color_mode: height
    lod: true
    max_points: 400000
    fit_box_height: true

Working example: the KITTI showcase design.

When another tool is the right answer

Segments.ai, Kognic, Deepen AI, Supervisely and Xtreme1/BasicAI are specialists, and for production autonomous-driving pipelines they are ahead: sequence workflows with track propagation, radar and multi-LiDAR, auto-fit cuboid models, and in Deepen's case a full targetless calibration product. Supervisely and Xtreme1 are self-hostable and Supervisely handles far larger clouds.

Potato does not do point-cloud sequences: annotation is per frame, and track propagation across a sweep is not implemented. Choose it for 3D when you want reliability statistics over spatial labels, or when 3D is one part of a multimodal study.

Further reading

• Point clouds reference
• Calibration
• Depth map annotation