Semantic mapping

On this page 7

Semantic mapping is the process by which a robot builds a map of its environment enriched with meaning, attaching category labels, object identities and relations to the geometric structure it perceives. In robotics perception, it extends purely metric maps, which record only where surfaces are, with information about what those surfaces are: floor, wall, door, table, person or shelf.

A semantic map lets a robot reason at the level that humans use. Instead of planning toward the coordinates (4.2, 7.9), the robot can execute “go to the kitchen and pick up the mug”. This shift from geometry to meaning is central to service robots, autonomous vehicles and mobile manipulators, which must interpret instructions, respect context-dependent rules and interact with specific objects.

How semantic mapping combines geometry and meaning

Semantic mapping works by fusing a geometric map with the output of recognition models, so that each map element receives a label and a confidence. The geometric backbone usually comes from SLAM, LiDAR odometry or visual odometry, which estimate the robot pose and reconstruct the surrounding structure as a point cloud, voxel grid or mesh.

In parallel, a perception pipeline runs semantic segmentation, instance segmentation or object detection on camera images and sometimes directly on point clouds. Using the camera calibration and the estimated pose, the system projects pixel labels into 3D space and associates them with the corresponding map elements.

Because a single observation is noisy, the map accumulates evidence over time. A common approach keeps a probability distribution over classes for each voxel or surfel and updates it with Bayesian fusion, so that one misclassified frame does not overwrite many consistent ones. Object-level systems instead maintain a list of object instances with pose, extent and class, and merge repeated detections through data association.

The following table summarizes the main stages of a typical semantic mapping pipeline and what each stage contributes.

StageInputOutputTypical methods
Localization and geometryCamera, LiDAR, IMU, wheel dataRobot pose, 3D structureSLAM, visual or LiDAR odometry
Semantic perceptionImages or point cloudsPer-pixel or per-point labelsSegmentation and detection networks
ProjectionLabels, depth, calibration, poseLabeled 3D pointsBack-projection with intrinsics and extrinsics
FusionLabeled points over timeClass probabilities per map elementBayesian update, CRF smoothing
AbstractionLabeled mapObjects, rooms, relationsClustering, scene graphs
Stages of a semantic mapping pipeline

Representations used for semantic maps

The choice of map representation determines memory cost, query speed and what reasoning is possible. Dense representations label every small volume of space, while sparse representations store only discrete objects and their relations.

Recent systems increasingly use hierarchical 3D scene graphs, in which nodes represent buildings, floors, rooms, places and objects, and edges express containment or adjacency. This structure supports task planning over meaningful entities and scales to large buildings far better than a dense labeled grid.

RepresentationGranularityStrengthsWeaknesses
Labeled occupancy grid2D cells, often 5-10 cmSimple, fast for navigationNo height, limited object detail
Semantic voxel map3D voxels, often 2-10 cmDense, supports free-space queriesHigh memory use in large areas
Semantic mesh or surfelsSurface elementsAccurate surfaces, good for visualizationCostly updates in dynamic scenes
Object-level mapIndividual instancesCompact, supports manipulationIgnores unlabeled structure
3D scene graphHierarchy of rooms and objectsScales well, supports reasoningComplex to build and maintain
Common semantic map representations

Many deployed robots combine representations: a dense occupancy layer for safe motion and a sparse object or room layer for task-level decisions.

Semantic mapping depends on several neighboring perception functions but is not identical to any of them. Object recognition identifies what appears in a single view; semantic mapping places that knowledge into a persistent, consistent spatial model that remains useful after the object leaves the field of view.

ConceptMain question answeredPersistenceRelation to semantic mapping
SLAMWhere am I and what is the geometryPersistent mapProvides the geometric backbone
Object recognitionWhat is in this imageSingle frameSupplies labels to be fused
Sensor fusionHow to combine sensor dataVariesMerges camera, LiDAR and radar evidence
Traversability estimationWhere can the robot driveLocal, often short-livedCan use semantic labels such as grass or curb
Semantic mappingWhat is where, and what it meansPersistent mapIntegrates all of the above
How semantic mapping relates to neighboring concepts

Semantics can also improve SLAM itself. Labels help reject points on moving objects such as people and vehicles, and recognized landmarks support more robust loop closure than raw appearance alone.

Applying semantic maps in robotic systems

Semantic maps are used wherever a robot must connect perception to goals expressed in human terms. Typical uses include object search, room-level navigation, rule-aware path planning and manipulation of named items.

  • Object search and fetch tasks: the robot looks up the last known position of a requested object instead of exploring blindly.
  • Context-aware navigation: costmaps assign higher cost to areas labeled as crosswalks, carpets or restricted zones.
  • Natural-language interaction: instructions such as “wait next to the reception desk” are grounded in labeled map entities.
  • Inspection and inventory: warehouse and retail robots record which products occupy which shelf locations.
  • Autonomous driving: high-definition maps store lanes, signs and traffic lights as semantic layers.

Configuring a system requires decisions about label set, resolution, update rate and memory budget. The decision table below links typical situations to recommended choices.

SituationRecommended choiceReason
Indoor service robot on embedded hardwareObject-level map over 2D gridLow memory, supports fetch tasks
Large multi-floor buildingHierarchical 3D scene graphScales and supports room-level queries
Manipulation in clutterDense voxel or mesh map near the armNeeds precise surfaces for grasping
Open-ended user instructionsOpen-vocabulary labels from vision-language modelsAvoids a fixed, closed class list
Highly dynamic scenesShort memory for movable classesPrevents stale positions of people and carts
Choosing a semantic mapping configuration

Limitations and common failure modes

The quality of a semantic map is bounded by both pose accuracy and recognition accuracy. A small calibration error between camera and LiDAR, or a drifting pose estimate, smears labels across neighboring surfaces, so a chair label may bleed onto the floor around it.

Closed-set classifiers recognize only the categories they were trained on and assign confident but wrong labels to unfamiliar objects. Open-vocabulary models reduce this problem but are slower and harder to validate. Domain shift, for example new lighting or a different building style, can reduce segmentation accuracy sharply without any visible warning.

SymptomLikely causeRemedy
Labels offset from objectsPoor extrinsic calibration or timing offsetRecalibrate, synchronize sensor timestamps
Duplicate instances of one objectWeak data associationUse geometric and appearance cues for matching
Ghost objects after they movedNo decay for dynamic classesAdd time-based forgetting and re-observation checks
Flickering class labelsFusing single frames without priorsUse probabilistic fusion over many views
Memory growth in large sitesDense map at uniform resolutionUse hierarchical or sparse representations
Typical problems, causes and remedies

A frequent misconception is that a better detector alone yields a better map. In practice, consistent fusion, uncertainty handling and map maintenance often matter as much as raw recognition accuracy.

A plain-language summary

A normal robot map is like a floor plan with only lines on it. It shows where walls and obstacles are, but it does not say what anything is. Semantic mapping adds the names and meanings, so the robot knows that this shape is a door, that one is a sofa and that area is the kitchen.

This matters because people give instructions in words, not coordinates. With a semantic map, a robot can understand requests like “bring the cup from the kitchen table” and find its way there. The robot builds this labeled map step by step, combining what its cameras recognize with where it knows it is.

An example from practice

A hospital deploys a mobile delivery robot to carry medication and supplies between wards on four floors. The robot uses LiDAR-based SLAM for its geometric map and a camera running instance segmentation to recognize doors, elevators, nurse stations, beds and supply carts. The system stores these as a hierarchical scene graph with floors, rooms and objects.

When a nurse requests a delivery to “the supply room next to ward 3B”, the robot queries the graph, plans a route through the correct elevator and stops at the labeled doorway. Supply carts are marked as movable objects with a 30-minute memory, so the planner does not trust outdated cart positions. After deployment, the operators reported fewer failed deliveries caused by blocked corridors, because the robot could reason about which obstacles were likely to move.