Label LiDAR Point Clouds With 3D Cuboids From KITTI

Autonomous-driving and robotics teams can follow this tutorial to draw 3D boxes on KITTI LiDAR scans in Potato 2.13.1, check every box against the camera image, measure agreement between two annotators, import KITTI tracking labels as editable tracks, and export KITTI label files. A point cloud from a LiDAR sensor is a set of 3D points, each with a position in meters and a return intensity. A cuboid is an oriented 3D bounding box with a center, a length, a width, a height, and a heading.

The worked example uses two scans from the KITTI 3D object benchmark and the first 10 frames of one sequence from the KITTI tracking benchmark. The numbers on this page come from a run of Potato 2.12.1 on those files, with two scripted annotators and a headless browser. The imports and exports were run again on Potato 2.13.1 on the saved annotations, and the import and export output shown is from 2.13.1.

The KITTI data

KITTI was recorded from a car driving around Karlsruhe, with a Velodyne LiDAR and four cameras (Geiger et al., 2013). The 3D object benchmark (Geiger et al., 2012) has 7,481 training scans and 7,518 test scans, each with a left color image and a calibration file. The tracking benchmark has 21 labeled training sequences. The KITTI site publishes all KITTI datasets under CC BY-NC-SA 3.0 and says the data is "made available for academic use only". You can't use KITTI scans or labels for commercial work, and CC BY-NC-SA asks that adaptations you share use the same license. This tutorial does not bundle any KITTI file, so download the files yourself after logging in on the KITTI site.

From the object benchmark page, download the Velodyne point clouds (29 GB), the left color images (12 GB), the camera calibration matrices (16 MB), and the training labels (5 MB). From the tracking benchmark page, download the Velodyne point clouds, the camera calibration, and the training labels. This tutorial uses only training scans 000063 and 000151 and frames 000000 to 000009 of tracking sequence 0000, laid out as below:

kitti-lidar/
├── kitti_object/training/
│   ├── velodyne/000063.bin, 000151.bin
│   ├── image_2/000063.png, 000151.png
│   ├── calib/000063.txt, 000151.txt
│   └── label_2/000063.txt, 000151.txt
└── kitti_tracking/training/
    ├── velodyne/0000/000000.bin ... 000009.bin
    ├── calib/0000.txt
    └── label_02/0000.txt

A KITTI .bin scan is a headerless list of 32-bit floats, four per point (x, y, z, and intensity). Scan 000063 has 119,796 points, and scan 000151 has 115,174. KITTI's labels for the two scans list eight objects:

KITTI's labels are the answer key in this tutorial. Annotators never see KITTI's labels, and the test run below compares each annotator's boxes with KITTI's.

Installing Potato

The point-cloud viewer needs nothing beyond the base package:

pip install potato-annotation==2.13.1

Potato reads the .bin files on the server, and the viewer's 3D library ships inside the package. In the headless-browser test, the annotation page made no request to any host other than the Potato server.

The project folder

The first project, kitti-boxes, gives the two object-benchmark scans to two annotators. Copy each scan's .bin, .png, and calibration file into a media/ folder, and list one scan per line in a data file:

kitti-boxes/
├── config.yaml
├── data/
│   └── scans.jsonl
└── media/
    ├── velodyne/000063.bin, 000151.bin
    ├── image_2/000063.png, 000151.png
    └── calib/000063.txt, 000151.txt

point_cloud names the scan relative to media/, and calibration names KITTI's calibration file together with the left color image (KITTI's camera P2):

{"id": "000063", "text": "KITTI 3D object training scan 000063", "point_cloud": "velodyne/000063.bin", "calibration": {"file": "calib/000063.txt", "images": {"P2": "image_2/000063.png"}}}
{"id": "000151", "text": "KITTI 3D object training scan 000151", "point_cloud": "velodyne/000151.bin", "calibration": {"file": "calib/000151.txt", "images": {"P2": "image_2/000151.png"}}}

The label_2 files stay out of the project, because the annotators never need KITTI's answers.

The config

The config below is complete and passes python -m potato.validate_cli config.yaml --strict on 2.12.1 and on 2.13.1:

annotation_task_name: KITTI 3D boxes, two annotators
task_dir: .
output_annotation_dir: annotation_output/
media_directory: media
data_files:
  - data/scans.jsonl
item_properties:
  id_key: id
  text_key: text
user_config:
  allow_all_users: true
  users: []
 
# Both annotators box every scan
num_annotators_per_item: 2
 
annotation_schemes:
  - annotation_type: spatial_annotation
    name: objects
    description: Put a 3D box around every car, van, pedestrian, and cyclist
    source_field: point_cloud
    calibration_field: calibration
    tools: [cuboid_3d]
    # KITTI's own class names, so the export writes the names unchanged
    labels:
      - {name: Car, color: "#FF6B6B", key_value: "1"}
      - {name: Van, color: "#4ECDC4", key_value: "2"}
      - {name: Pedestrian, color: "#FFD93D", key_value: "3"}
      - {name: Cyclist, color: "#6C8AE4", key_value: "4"}
    attributes:
      - name: occluded
        type: select
        options: [none, partly, largely, unknown]
        default: none
    default_box_height: 1.5
    fit_box_height: true

Four settings need a reason:

• The class names. The KITTI export writes each box's class name as the label file's type column. Naming the classes Car, Van, Pedestrian, and Cyclist produces lines that KITTI's tools read without a mapping step.
• calibration_field. The calibration turns on the camera panel under the viewer, and the export needs the calibration to write KITTI's camera coordinates. The export skips an item with no calibration and names the item in a warning.
• The occluded attribute. The four options map to KITTI's occlusion codes 0 to 3 in the export. The config has no truncation attribute, because the export measures truncation from the camera image, as described under the export step below.
• default_box_height and fit_box_height. A new box starts 1.5 m tall, close to the 1.45 to 1.59 m of the four cars in the two scans. With fit_box_height: true, the viewer then fits the box's top and bottom to the LiDAR points inside the box's footprint whenever the footprint contains at least five points.

Drawing boxes

Start the server in the project folder:

potato start config.yaml -p 8000

Each annotator opens the server's address, registers with a username and a password, and gets both scans. The viewer shows the cloud colored by height, and the status line under the cloud reads "119,796 points, all loaded" once the whole of scan 000063 has arrived. Under the cloud are three orthographic slab panels, Top (X-Y), Front (Y-Z), and Side (X-Z), each 2.0 m thick, and a camera panel with the left color image.

Press c for the 3D box tool and a number key for the class (1 for Car), then drag across the object's footprint on the ground. The status line confirms the box with the box's size and position, for example "Box added: 7.9 x 0.4 x 1.5 m at 34.3, -19.2" in the browser test. A new box is level and aligned with the LiDAR's axes. Press q and e to turn the selected box in 5-degree steps, or Shift+q and Shift+e for 1-degree steps. In a slab panel, the arrow keys move the box, Shift with an arrow moves the face on that side outward, and Alt with an arrow moves the face inward. The , and . keys step through the boxes on the scan, and Delete removes the selected box. The occluded menu for the selected box sits in the attributes panel at the bottom.

The camera panel draws every box onto the left color image through the scan's calibration. A box that looks right from above can be too short along the line of sight, and the camera image shows the error at once. Check every box in the camera panel before clicking Next.

A class button sets the class of the next box. From Potato 2.13.1, clicking a class button while a box is selected changes that box's class, and in a sequence the whole track's class. In the browser test on 2.12.1, the click changed only the armed class, and a box with the wrong class had to be deleted and drawn again.

The test run behind this page

Two scripted annotators boxed both scans through the running server and saved each scan with the same request the viewer sends. Each annotator started from KITTI's own label for every object. The script converted each label into the LiDAR frame with Potato's KITTI conversion and then changed the boxes in a fixed way:

• ana moved every center 0.10 m forward and 0.05 m right, made every side 0.05 m longer, turned every heading by 2 degrees, and copied KITTI's occlusion level.
• ben moved every center 0.25 m back and 0.15 m left, made every side 8% longer, turned every heading by -6 degrees, skipped the Van 34.8 m away on scan 000063, called the Cyclist on scan 000151 a Pedestrian, and left occlusion at none.

The fixed errors make the agreement numbers checkable against KITTI's labels. A person's errors vary more from box to box than the two scripted patterns.

Agreement between annotators

Open /admin in a browser, type the admin key from admin_api_key.txt in the project folder, and then open /admin/iaa. For the test run, the report listed both scans with both annotators and gave:

mean_agreement                          0.272
mean_object_count_diff                  0.500
attributes.occluded.alpha              -0.174
attributes.occluded.percent_agreement   0.400

Potato pairs the two annotators' boxes on each scan by 3D intersection over union (IoU), and Potato computes the IoU of two rotated boxes exactly. Two boxes count as the same object at a 3D IoU of 0.25 or more. A scan's score is the sum of the matched pairs' IoU divided by the number of matched pairs plus the number of unmatched boxes. A pair whose classes differ counts at half the pair's IoU. The two scans scored 0.239 (000063) and 0.305 (000151), and the report gives the mean of the two scores. The table below lists the pairs behind those scores and each box's 3D IoU against KITTI's label:

The two pedestrians show how strict 3D IoU is on a small object. The pedestrian boxes are 0.5 to 1.0 m across, and ana's and ben's centers are 0.40 m apart. The two boxes share too little volume to match, and each box counts as an unmatched object. The same 0.40 m offset on the cars, 3.2 to 4.5 m long, still matched at 0.53 to 0.66. The occlusion alpha of -0.174 covers the five matched boxes. Ben's blanket none agreed with ana's copied KITTI levels on two of the five.

The mean_agreement value is not corrected for chance, and the report gives no per-pair or per-annotator breakdown. Agreement for 3D Boxes, Time Segments, and Agent Traces explains the matching in more detail.

Exporting KITTI label files

The kitti_3d export writes one KITTI label file per scan, in a folder per annotator when more than one annotator has saved work:

python -m potato.export -c config.yaml -f kitti_3d -o export/

export/ana/label_2/000063.txt
export/ana/label_2/000151.txt
export/ben/label_2/000063.txt
export/ben/label_2/000151.txt

Each line has KITTI's 15 fields, and ana's file for scan 000151 reads:

Car 0.95 0 -0.87 0.00 199.35 312.43 369.00 1.55 1.83 3.74 -3.12 1.69 3.45 -1.60
Cyclist 0.00 1 -1.63 990.09 145.93 1031.72 217.23 1.77 0.83 1.76 10.57 0.91 18.52 -1.11
Pedestrian 0.84 0 -0.99 1210.27 127.60 1223.00 228.51 1.77 0.60 0.98 11.62 0.84 12.81 -0.25
Car 0.00 1 -0.43 845.53 161.27 1005.56 210.34 1.50 1.80 4.28 10.08 0.91 22.52 0.00

Potato stores a box in meters in the LiDAR frame, with the rotation as a quaternion. KITTI's 3D fields are in the rectified reference camera's frame. The export transforms each box through R0_rect and Tr_velo_to_cam from the scan's calibration file, moves the location from the box's center to the center of the box's bottom face, and writes the heading as rotation_y. KITTI stores only the heading, so the export drops any pitch and roll and reports in a warning how many boxes were tilted by more than 1 degree.

To check the conversion, a third scripted annotator saved KITTI's labels unchanged on both scans, and the exported files were compared with KITTI's label_2 files field by field:

The 2D box is the cuboid's projection rather than a box drawn on the image, and the projection runs wider than KITTI's hand-drawn box on the pedestrian on scan 000063 (5.7 px on the left edge). On an object cut off by the image edge, Potato 2.13.1 clips the box at the last pixel, so the pedestrian on scan 000151 ends at 1223.00, as in KITTI's file. Potato 2.12.1 clipped at the image width and wrote 1224.00. The export writes no DontCare lines, because Potato has no DontCare region type. KITTI's object labels and the export both use two decimals, so the 3D fields came back unchanged.

Importing KITTI labels

potato import -f kitti reads KITTI's label_2 files into a 2D image-box project. The import keeps each object's dimensions, location, and rotation_y as read-only attributes, and prints a warning that only the 2D box can be edited. Potato 2.13.1 has no importer that places KITTI's 3D object labels on a single scan as cuboids.

KITTI's tracking labels do come in as cuboids. potato import -f kitti_tracking reads a tracking folder with label_02/, velodyne/, and calib/, and makes one item per sequence, with one keyframe per labeled frame. Run the import from the kitti-lidar folder:

potato import -i kitti_tracking/training -f kitti_tracking -o tracking-0000

INFO: Marked 31 imported annotation(s) as source='import' (kitti_tracking)
WARNING: Left out 17 DontCare row(s): they mark regions KITTI excludes from evaluation, not objects.
Imported 1 sequence(s), 4 track(s), 31 keyframe(s), 3 categor(ies) into tracking-0000
 
Run it:
  potato start tracking-0000/config.yaml -p 8000

The test run kept the label rows for frames 0 to 9, to match the 10 downloaded scans. The 10 frames have four tracks, a Van, a Cyclist, and a Pedestrian from frame 0 and a second Van from frame 5. The importer converts every box from KITTI's camera frame into the LiDAR frame through the sequence's calibration file, and leaves out a sequence whose scans skip a frame number, because Potato numbers frames by the scans' order. The scheme is named objects. Potato 2.12.1 named the scheme image_annotation unless the command added --schema-name objects.

The importer writes tracking-0000/config.yaml, data/training.json, and a README. The generated config points media_directory back at the dataset folder and shows the imported tracks to every annotator as a starting point. The config below is the generated one with four changes, and the config is complete and passes the validator:

annotation_task_name: KITTI tracking 0000, two annotators
task_dir: .
output_annotation_dir: annotation_output/
media_directory: ../kitti_tracking/training
data_files:
  - data/training.json
item_properties:
  id_key: id
  text_key: notes
user_config:
  allow_all_users: true
  users: []
num_annotators_per_item: 2
pre_annotation:
  enabled: true
  field: predictions
  allow_modification: true
annotation_schemes:
  - annotation_type: spatial_annotation
    name: objects
    description: Correct the imported tracks
    source_field: point_cloud
    calibration_field: calibration
    tools: [cuboid_3d]
    labels:
      - {name: Van, color: "#4ECDC4", key_value: "1"}
      - {name: Pedestrian, color: "#FFD93D", key_value: "2"}
      - {name: Cyclist, color: "#6C8AE4", key_value: "3"}
    attributes:
      - name: occluded
        type: select
        options: [none, partly, largely, unknown]
        default: none
        per_keyframe: true
      - name: truncated
        type: select
        options: [none, partly, fully]
        default: none
        per_keyframe: true

The changes are a task name, num_annotators_per_item: 2, and a color and a key for each class. KITTI records occlusion and truncation for every frame, and the importer writes each frame's values, so the 2.13.1 importer also writes per_keyframe: true on both attributes. Without per_keyframe, a change to an attribute on one frame applies to every keyframe of the track, and on Potato 2.12.1 the setting has to be added by hand. Potato 2.12.1's validator also printed a note that pre_annotation was planned for removal in Potato 3.0, and 2.13.1 prints no note.

Correcting a track

Start the server in tracking-0000 as before. The frame bar under the cloud reads "Frame 1 of 10 (3 keyframes)", and [ and ] step through the frames. A track's box is stored only on the track's keyframes, and on the frames between two keyframes the viewer interpolates the box (center and size linearly, rotation along the shortest arc) and draws the box with dashed edges. Moving, resizing, or rotating an interpolated box makes a new keyframe there. f copies the selected box to the next frame as a keyframe, Delete removes the selected box's keyframe on the current frame, and the Delete track button removes the whole track. Potato does not move a box to follow an object between keyframes. Every keyframe is a box an annotator placed or imported.

The tracking calibration file names no image, so the camera panel shows the box projections without a photo. Potato's calibration names one image per camera for an item, so a sequence cannot show each frame's own photo.

Agreement and export on the sequence

Two more scripted annotators saved the imported tracks. eli saved all 31 imported keyframes unchanged. dana kept each track's keyframes on frames 0, 3, 6, and 9 and on the track's first and last frame, and deleted the other 17, so Potato interpolates the boxes on the frames between. /admin/iaa scores a sequence frame by frame, with interpolated boxes filled in, and adds track-identity measures:

mean_agreement                          0.991
n_frames                                10
identity.mota                           1.0
identity.idf1                           1.0
identity.id_switches                    0
attributes.occluded.alpha               1.0
attributes.truncated.alpha              0.745

The interpolated boxes stayed close to KITTI's labels on this stretch of road. Against KITTI's boxes on the same frames, dana's exported boxes had a mean center error of 0.005 m and a largest error of 0.051 m, and a 3D IoU of 0.876 at the lowest and 0.991 on average. A vehicle that turns or brakes between keyframes can leave a larger error, so add keyframes where the motion changes.

The tracking calibration file does not record the image size, so pass the size of KITTI's images to the export:

python -m potato.export -c config.yaml -f kitti_3d -o export/ --option image_width=1242 --option image_height=375

The export writes export/eli/label_02/0000.txt and export/dana/label_02/0000.txt in KITTI's tracking format, with a frame number and a track ID before the 15 object fields and one line for every frame of every track. Without the image size, the export writes the 2D boxes unclipped and says so in a warning. The first three lines of eli's file are:

0 0 Van 0 0 -1.788236 297.306825 163.588132 455.254746 294.245211 2.000000 1.823255 4.433886 -4.552284 1.858523 13.410495 -2.115488
0 1 Cyclist 0 0 -1.952163 739.947686 163.364720 931.778054 374.000000 1.739063 0.824591 1.785241 1.640400 1.675660 5.776261 -1.675458
0 2 Pedestrian 0 0 -2.540723 1088.811242 167.172676 1220.919879 324.237147 1.714062 0.767881 0.972283 6.301919 1.652419 8.455685 -1.900245

Compared with KITTI's label_02 file, eli's 31 lines matched on track, type, truncation, occlusion, dimensions, location, and rotation_y, and the unchanged boxes came back at a 3D IoU of 1.000 against KITTI's. Potato 2.13.1 writes six decimals, as KITTI's tracking labels do, and keeps an imported box's own truncation unless someone edits the box. Potato 2.12.1 wrote two decimals, which left the unchanged boxes at a 3D IoU of 0.979 or more. Potato 2.12.1 also measured truncation from the box's projection and wrote 1 where KITTI has 0 on 13 of the 31 lines, on the Cyclist, whose box projects below the image edge on all 10 frames, and on the Pedestrian on frames 1 to 3. The 2D boxes differed from KITTI's hand-drawn boxes by up to 28 px, on the Pedestrian 8.2 m away on frame 1.

Other tools for LiDAR cuboids

CVAT Community is MIT-licensed and self-hosted, draws cuboids on point clouds, and interpolates a cuboid track between frames. CVAT's 3D exports are Kitti Raw Format 1.0 and Sly Point Cloud Format 1.0, and CVAT's KITTI 1.0 format covers 2D images. CVAT Online's Solo and Team plans cost $33 per user per month billed monthly, or $23 billed yearly. Segments.ai reads point clouds in formats that include PCD, PLY, and KITTI binary, labels point cloud sequences with interpolation, and the Segments.ai Fusion plan adds sensor fusion interfaces. Segments.ai prices every plan by quote, caps Core-plan point clouds at 500,000 points, and offers free academic licenses. Supervisely has a free Community plan and a Pro plan from €199 per month. The pricing page lists cuboid tracking and automatic cuboid adjustment among Supervisely's point cloud features, and Supervisely's app catalog has Import KITTI 3D and Export to KITTI 3D apps. Xtreme1 is an Apache-2.0 tool hosted by the LF AI & Data Foundation, with 3D cuboids, LiDAR-camera fusion, and frame-series tracking. Version 0.9.3 came out on 2026-09-28, and BasicAI offers an enterprise version. Prices and features were read from each tool's own pages on 2026-10-29.

Several of these tools place or adjust cuboids with a model, such as Xtreme1's OpenPCDet integration, and Potato has no model for 3D boxes. Potato fits a study that needs several annotators on the same scans with agreement on the boxes, and a team that wants the KITTI conversion done by the tool rather than by a script. Point Cloud Annotation Tools Compared sets these tools beside Potato in more detail.

Limits of Potato for LiDAR labeling

• No model-assisted boxes or tracking. Boxes come from annotators or imports, and a track moves between keyframes by interpolation only.
• No 3D import for single KITTI scans. -f kitti reads KITTI's 2D boxes, and -f kitti_tracking reads sequences.
• Agreement without chance correction. /admin/iaa gives one mean 3D IoU score per scheme, with no per-pair or per-annotator breakdown.
• Projected 2D boxes. The export's 2D boxes come from the 3D box and run wider than KITTI's hand-drawn boxes.

Where to go next

• 3D Point Clouds and Cuboids, for every point-cloud format and option
• Calibration, for camera panels and the calibration formats Potato reads
• How to Annotate 3D Point Clouds, for rotation, frames, and level of detail
• Computer Vision Formats, for what each import and export format keeps
• Importing Projects, for the other potato import formats
• Computer vision and Robotics and embodied AI, for related tasks on this site

References

1. Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? The KITTI vision benchmark suite. 2012 IEEE Conference on Computer Vision and Pattern Recognition, 3354-3361. https://doi.org/10.1109/cvpr.2012.6248074
2. Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013). Vision meets robotics: The KITTI dataset. The International Journal of Robotics Research, 32(11), 1231-1237. https://doi.org/10.1177/0278364913491297