How to Annotate Depth Maps
Depth files do not carry their own unit, zero means no-return rather than close, and the interesting range is never the full range. What to get right before annotating depth.
A depth map opened as an image is a black rectangle, and if it were not, you would still be looking at colours with no idea what distance any of them means. Three properties separate depth from an image, and all three are silent when you get them wrong.
The file does not know its own unit
Millimetres for RealSense, Azure Kinect and NYU-Depth. 1/256 m for the KITTI completion benchmark. Metres for anything written from a float array.
Nothing in the file records which. depth_scale is metres per stored unit, and getting it wrong is invisible: the window rescales and the picture looks the same. The only way to catch it is a readout in metres checked against something whose distance you know.
Zero is not a distance
Zero is the near-universal "no return" code — the sensor did not measure this pixel. Read as depth, it paints a bright wall across every hole in coverage.
Carry non-measurements as NaN and paint them a colour that appears in no colormap. Potato uses magenta, so a hole cannot be misread as near or far depth, and reports what fraction of the map is holes.
That fraction matters more than it sounds. A stereo rig facing a textureless wall really does return 80% holes, and an annotator who cannot see which pixels are missing will read them as geometry and label them.
The interesting range is never the full range
A 16-bit depth map whose content sits between 1200 and 1800 renders with 4 grey levels of separation out of 255 under a naive 8-bit cast. Under a percentile window it renders with 254.
Default the near/far window to the 2nd and 98th percentile of the valid pixels — valid meaning after the no-return pixels are excluded, or the holes will dominate the percentiles.
Configuration
instance_display:
fields:
- key: depth
type: depth_map
label: "Depth"Formats read directly: 16-bit PNG, 16-bit TIFF, NPY and NPZ, PFM, EXR.
Unprojection
Given camera intrinsics, a depth map becomes a point cloud, and the same 3D cuboid tools apply. For RGB-D datasets this is usually the more productive way to annotate: judging a 3D extent is easier in 3D than by reading a colormap.
What to annotate on depth
- Quality judgements on estimated depth, against the RGB frame.
- Region labels where depth is wrong, which is the supervision signal for depth completion.
- 3D boxes, after unprojection.
For a quality task, show the RGB frame alongside. Depth errors are usually obvious only in comparison.