# How to Annotate Depth Maps

Source: https://www.potatoannotator.com/docs/guides/depth-map-annotation

**A depth map opened as an image is a black rectangle, and if it were not, you would still be looking at colours with no idea what distance any of them means.** Three properties separate depth from an image, and all three are silent when you get them wrong.

## The file does not know its own unit

Millimetres for RealSense, Azure Kinect and NYU-Depth. 1/256 m for the KITTI completion benchmark. Metres for anything written from a float array.

Nothing in the file records which. `depth_scale` is metres per stored unit, and getting it wrong is invisible: the window rescales and the picture looks the same. The only way to catch it is a **readout in metres** checked against something whose distance you know.

## Zero is not a distance

Zero is the near-universal "no return" code — the sensor did not measure this pixel. Read as depth, it paints a bright wall across every hole in coverage.

Carry non-measurements as NaN and paint them a colour that appears in no colormap. Potato uses magenta, so a hole cannot be misread as near or far depth, and reports what fraction of the map is holes.

That fraction matters more than it sounds. A stereo rig facing a textureless wall really does return 80% holes, and an annotator who cannot see which pixels are missing will read them as geometry and label them.

## The interesting range is never the full range

A 16-bit depth map whose content sits between 1200 and 1800 renders with **4 grey levels of separation out of 255** under a naive 8-bit cast. Under a percentile window it renders with **254**.

Default the near/far window to the 2nd and 98th percentile of the *valid* pixels — valid meaning after the no-return pixels are excluded, or the holes will dominate the percentiles.

## Configuration

```yaml
instance_display:
  fields:
    - key: depth
      type: depth_map
      label: "Depth"
```

Formats read directly: 16-bit PNG, 16-bit TIFF, NPY and NPZ, PFM, EXR.

## Unprojection

Given camera intrinsics, a depth map becomes a point cloud, and the same [3D cuboid tools](/docs/guides/point-cloud-annotation) apply. For RGB-D datasets this is usually the more productive way to annotate: judging a 3D extent is easier in 3D than by reading a colormap.

## What to annotate on depth

- **Quality judgements** on estimated depth, against the RGB frame.
- **Region labels** where depth is wrong, which is the supervision signal for depth completion.
- **3D boxes**, after unprojection.

For a quality task, show the RGB frame alongside. Depth errors are usually obvious only in comparison.

## Further reading

- [Depth maps reference](/docs/vision-spatial/depth-maps)
- [Point cloud annotation](/docs/guides/point-cloud-annotation)
- [Media ingest](/docs/vision-spatial/media-ingest)