A point cloud is an unordered set of three-dimensional points, each with x, y and z coordinates and often extra attributes such as intensity, color or a surface normal, that samples the surfaces of a scene. In robotics perception, it is the most direct digital record of 3D geometry captured by LiDAR, depth cameras and stereo systems.
Point clouds sit between raw sensor data and higher-level understanding. A robot builds them from range measurements, then filters, segments and registers them to detect obstacles, recognize objects, estimate its own motion and build maps. Their value lies in metric accuracy: distances are measured, not inferred from appearance.
How a point cloud is formed
Every point in a point cloud originates from a range measurement along a known ray. A LiDAR measures the time of flight of a laser pulse at known azimuth and elevation angles; a depth camera measures distance per pixel; a stereo camera triangulates depth from the disparity between two images. Converting these measurements with the sensor’s intrinsic calibration yields Cartesian coordinates in the sensor frame.
To be useful in a robot, points are then transformed into a common coordinate frame using the extrinsic calibration and the robot’s pose. A cloud can be organized, keeping the row and column structure of the sensor image, or unorganized, a plain list of points. Organized clouds allow fast neighbor lookup through pixel adjacency, which simplifies normal estimation and segmentation.
The source sensor determines the character of the resulting cloud, as the following table shows.
| Source | Typical range | Density | Typical weakness |
|---|---|---|---|
| Spinning LiDAR | About 50 to 200 m | Sparse rings, dense near the sensor | Gaps between scan lines |
| Solid-state LiDAR | About 20 to 250 m | Dense in a limited field of view | Narrower coverage |
| Time-of-flight camera | About 0.2 to 6 m | Dense, one point per pixel | Multipath and sunlight interference |
| Stereo camera | About 0.3 to 20 m | Dense where texture exists | Fails on blank or repetitive surfaces |
| Structured light | About 0.2 to 3 m | Very dense at short range | Degrades in sunlight |
The point cloud processing pipeline
Raw point clouds are rarely used directly. A typical pipeline applies a sequence of steps, each with parameters that strongly influence speed and accuracy. Choosing these parameters is a core skill in robot perception.
| Step | Purpose | Key parameter | Typical value |
|---|---|---|---|
| Crop box filter | Remove points outside the region of interest | Box limits | Robot workspace in meters |
| Voxel grid downsampling | Reduce point count evenly | Voxel size | About 1 to 10 cm |
| Statistical outlier removal | Remove isolated noise points | Neighbors and standard deviation | About 20 to 50 neighbors |
| Ground removal | Separate floor from obstacles | Plane distance threshold | About 2 to 10 cm |
| Euclidean clustering | Group points into objects | Cluster tolerance | About 5 to 50 cm |
| Normal estimation | Compute surface orientation | Search radius | Several times the voxel size |
Ground removal is usually done with RANSAC plane fitting, which repeatedly fits a plane to random point triples and keeps the model with the most inliers. Voxel size is the main trade-off: a large voxel speeds up every later step but erases thin structures such as table legs, cables and chair edges.
Registration and the role in mapping
Registration finds the rigid transformation that aligns one point cloud with another. The classic algorithm is ICP (iterative closest point), which alternately matches each point to its nearest neighbor and solves for the best rotation and translation. ICP converges well only from a good initial guess, so it is usually seeded by odometry or by a coarse feature-based alignment using descriptors such as FPFH.
Registration is the engine behind LiDAR odometry and many SLAM systems: consecutive scans are aligned to estimate motion, and the aligned clouds are accumulated into a map. Variants such as point-to-plane ICP and generalized ICP converge faster on structured scenes like corridors and buildings.
Point clouds compared with other 3D representations
A point cloud is only one way to describe geometry. It is the most faithful to the raw data but carries no explicit connectivity, free space or surface. Robots often convert it into another representation suited to a specific task.
| Representation | What it stores | Strength | Limitation |
|---|---|---|---|
| Point cloud | Sampled surface points | Exact measurements, simple | No surfaces, no free space |
| Voxel or octree map | Occupied and free cells | Encodes free space for planning | Resolution limits detail |
| Mesh | Connected triangles | Continuous surfaces, rendering | Costly to build and update |
| Signed distance field | Distance to nearest surface | Smooth collision checking | Higher memory use |
| Depth image | Distance per pixel | Compact, fast for neural networks | Single viewpoint only |
For object recognition, deep networks such as PointNet-style architectures consume points directly, while others project the cloud into a bird’s-eye-view image or voxel grid. In sensor fusion, points are projected into camera images to attach color and semantic labels to geometry.
Limitations and common pitfalls
Point clouds look convincing on screen, which hides several systematic problems that degrade robot behavior.
- Motion distortion: a spinning LiDAR captures one sweep over about 50 to 100 ms, so a moving robot skews the cloud unless it is deskewed with IMU or odometry data.
- Problem surfaces: glass, mirrors, black rubber and shiny metal return few or false points.
- Uneven density: point spacing grows with distance, so a fixed clustering tolerance merges near objects and splits far ones.
- Data volume: a modern LiDAR produces hundreds of thousands to millions of points per second, which demands downsampling or GPU processing.
- Calibration errors: a small extrinsic rotation error displaces distant points by many centimeters.
The simple explanation
Imagine touching a dark room with thousands of fingertips at once and writing down exactly where each fingertip hit something. The list of all those touch points is a point cloud. It shows the shape of walls, boxes and people as a cloud of dots in 3D space.
A robot cannot recognize anything from the dots alone, so it cleans them, groups them and compares them. The key benefit is real measured distance, which lets the robot know exactly how far away an obstacle is.
How it is used in practice
A bin-picking cell in a parts factory uses a structured-light 3D camera mounted above a crate of randomly piled metal brackets. Each capture produces about one million points. The software crops the cloud to the crate, applies a 5 mm voxel grid, removes the crate floor with RANSAC and clusters the remaining points into individual parts.
Each cluster is registered against a CAD-derived model with coarse feature matching followed by ICP, giving a 6D pose for every visible bracket. The robot then chooses the least obstructed part and picks it. Because shiny edges produce missing points, the engineers add a diffuse lighting setup and a matte crate liner, which noticeably improves the pick success rate.