An occupancy network is a neural network for 3D scene perception that predicts, for each small volume of space around a vehicle, whether it is occupied or free, often together with its semantic class and motion. In autonomous driving, it replaces or complements classic object detection by describing the world as geometry rather than a list of boxes, so the vehicle can avoid obstacles it was never explicitly trained to recognize.
The idea originated in computer vision research on implicit 3D shape representation and was adapted to driving around the early 2020s. Today it appears in several camera-centric perception stacks of electric and autonomous vehicles, where it feeds the planner with a dense map of drivable and blocked space.
How an occupancy network represents space
An occupancy network divides the surroundings into a 3D grid of voxels, small cubes typically between 0.1 and 0.5 meters on a side, covering a region of perhaps 50 to 100 meters around the car and several meters in height. For each voxel, the network outputs a probability of occupancy, and many designs add a semantic label such as road, vehicle, pedestrian or vegetation, plus a flow vector describing how that voxel moves.
Some variants use an implicit representation instead of a fixed grid: the network can be queried at any continuous 3D coordinate and returns an occupancy value. This allows finer detail where needed without storing every voxel, at the cost of more computation per query.
| Output | Format | Use in the driving stack |
|---|---|---|
| Occupancy probability | Value from 0 to 1 per voxel | Collision checking and free-space planning |
| Semantic class | Category label per voxel | Distinguishing curbs, vehicles and people |
| Occupancy flow | 3D velocity vector per voxel | Predicting motion of unknown objects |
| Surface geometry | Sub-voxel offset or signed distance | Precise edges for tight maneuvers |
| Uncertainty | Variance or confidence score | Cautious behavior in ambiguous areas |
From camera images to a 3D occupancy grid
Most production occupancy networks are vision-based, taking images from six to eight surround cameras. Each image passes through a shared backbone network, usually a convolutional network or vision transformer, which extracts multi-scale feature maps. The difficult step is lifting 2D features into 3D, because a single camera pixel does not reveal depth.
Two lifting strategies dominate. Depth-based lifting predicts a depth distribution for each pixel and projects features along camera rays into 3D. Attention-based lifting starts from learnable 3D queries and uses cross-attention to sample features from the camera images where each query projects. Many networks first build a bird’s-eye view feature plane and then extend it vertically, which keeps memory usage manageable.
Temporal fusion is the next stage. Features from previous frames are warped using the vehicle’s measured ego-motion and combined with the current frame, which helps recover occluded regions and estimate motion. A decoder then upsamples the volume to the final voxel resolution.
| Stage | Common technique | Key trade-off |
|---|---|---|
| Image encoding | ResNet, RegNet or transformer backbone | Accuracy versus inference latency |
| 2D-to-3D lifting | Depth distributions or cross-attention | Geometric precision versus compute |
| Temporal fusion | Ego-motion warping of past features | Stability versus lag on fast objects |
| Volume decoding | 3D convolutions or sparse upsampling | Resolution versus memory footprint |
| Training supervision | Lidar-derived voxel labels | Label quality versus annotation cost |
Training usually relies on lidar-derived ground truth: fleets of data-collection vehicles record lidar point clouds, which are accumulated over time and voxelized into dense labels. The deployed network can then run without lidar on the production car, a central reason the approach appeals to camera-only designs.
Occupancy networks compared with other perception methods
Occupancy networks differ from 3D bounding box detection in what they assume about the world. A detector reports a fixed set of trained classes as boxes; anything outside those classes, such as a fallen ladder or an unusual trailer, may simply be missed. An occupancy network asks a more general question, whether space is physically blocked, which provides a safety net for long-tail objects.
| Representation | Describes | Strength | Weakness |
|---|---|---|---|
| 3D bounding boxes | Known object classes | Compact, easy to track | Misses unknown shapes |
| Bird’s-eye view map | 2D layout from above | Efficient for planning | Loses height information |
| Occupancy network | 3D volume, occupied or free | Handles arbitrary geometry | High compute and memory |
| Raw lidar point cloud | Measured surface points | Direct range accuracy | Sparse, needs interpretation |
| HD map | Static road prior | Precise lane geometry | Becomes outdated |
In practice, the methods are complementary rather than exclusive. Many stacks run a detector for agents that require trajectory prediction, an occupancy network for general obstacles, and lane or map networks for road structure, then fuse them before the driving policy acts.
Limitations and open challenges
The biggest constraint is computational cost. A dense grid of 200 by 200 by 16 voxels already holds 640,000 cells, and finer grids grow cubically. Engineers use sparse representations, coarse-to-fine decoding and reduced height ranges to fit within automotive chips running at 10 to 36 frames per second.
- Depth ambiguity from cameras causes thickened or blurred obstacles at long range.
- Thin structures such as poles, cables and barrier gates are easily missed at coarse resolution.
- Occluded regions must be inferred, so predictions behind large vehicles are uncertain.
- Label noise from lidar ground truth, including reflections and moving objects, propagates into the network.
- Validation is difficult, because standard metrics such as mean intersection over union do not directly measure safety relevance.
These issues matter for safety arguments. Under frameworks like ISO 21448, developers must show that perception limitations are understood and mitigated, which for occupancy networks means testing rare geometry, adverse weather and edge-of-range performance with dedicated scenario sets.
Understanding it simply
Imagine the space around a car filled with millions of tiny invisible building blocks. An occupancy network looks through the car’s cameras. Block by block, it decides whether something solid is there or the space is empty. It also notes whether that something is moving.
This matters because the car does not need to know what an object is to avoid it. A mattress, a strange trailer or a pile of debris all look like filled blocks. So the car can steer around anything that takes up space, even things it has never seen.
Applied in practice
An engineering team developing a camera-based driving system for an electric delivery van tests it in a city depot. During a run, a worker leaves a wheeled cart loaded with tilted boxes partly in the lane. The object detector does not recognize the cart as any trained class, but the occupancy network marks a cluster of voxels as occupied up to 1.4 meters high.
The planner treats those voxels as blocked space and plans a path that keeps about 0.8 meters of lateral clearance, slowing the van from 25 to 12 km/h as it passes. Reviewing the logs afterward, the team uses the event to add similar carts to its scenario library, confirming that geometry-based perception covered a gap the class-based detector left open.