Camera Calibration and 2D Verification

Draw a 3D box in the point cloud and see it land on the object in the photograph that saw it. Camera calibration supplied per item projects every cuboid into every camera view, redrawn as the box moves.

annotation_schemes:
  - annotation_type: spatial_annotation
    name: objects
    source_field: point_cloud
    calibration_field: calibration    # the item field holding the rig
    tools: [cuboid_3d]
    labels:
      - {name: car, color: "#FF6B6B"}

Runnable example: examples/spatial/kitti-cuboids/.

Why this matters more than it sounds

A car at 40 metres is a few dozen lidar returns. Whether the box is tight, whether it is the right length, and whether it is rotated correctly are all close to unanswerable from the cloud alone, and obvious in the camera image.

So the annotator edits in 3D and verifies in 2D. The panels under the viewport show each camera's view with the boxes projected onto it.

Without this, 3D labelling has no feedback loop. A box that looks correct from the orbit camera can be metres off along the view axis, and nothing in the 3D view says so.

Frames are where this goes wrong

Calibration is a chain of transforms, and every link is a chance to be quietly wrong:

• Reference frame versus camera frame. Using a particular camera's transform where the rectified reference frame is wanted shifts every box by that camera's stereo baseline. A few centimetres, systematic, and easy to mistake for annotator sloppiness.
• Location conventions. KITTI's 3D location is the midpoint of the box's bottom face. Reading it as the centre puts every object half its own height underground.
• Dimension order and axis assignment. KITTI's dimensions are h w l, and in the box's own frame length runs along X, height along Y downward, and width along Z. Assuming length runs along the camera's forward axis, which reads as the natural choice, produces a box rotated exactly 90°.

That last one is the instructive case: a round-trip test cannot catch it, because the inverse conversion makes the same assumption and the two agree with each other. Only a comparison against the devkit's own corner formula catches it, and there is a test for exactly that.

Related

• Point clouds
• Depth maps
• CV formats
• Source documentation