Gaussian splatting, also called 3D Gaussian splatting, is a scene representation and rendering technique that describes a captured environment as millions of anisotropic 3D Gaussians, each with a position, shape, opacity and view-dependent color, and draws them by rasterizing and alpha-blending them onto the screen. In XR development, it has become a leading way to turn ordinary photos or video of a real place into a photorealistic, navigable 3D scene that runs at headset frame rates.
The method was introduced in 2023 as an alternative to neural radiance fields (NeRF). It keeps the visual quality of learned radiance fields but replaces slow neural network queries with an explicit, GPU-friendly primitive, which is why it fits the tight latency budgets of extended reality.
How Gaussian splatting builds and renders a scene
A Gaussian splatting scene is produced by optimization from posed images, not by manual modeling. The pipeline starts like photogrammetry: a structure-from-motion step, typically with COLMAP or a similar solver, estimates camera poses and a sparse point cloud from 50 to several hundred overlapping photos.
Each sparse point then seeds one Gaussian. During training, a differentiable rasterizer renders the Gaussians from every known camera, compares the result with the real photo and adjusts all parameters by gradient descent. Adaptive density control clones Gaussians in under-reconstructed areas, splits oversized ones and prunes nearly transparent ones, so a final scene usually holds one to six million Gaussians.
Rendering projects each 3D Gaussian into a 2D elliptical splat, sorts the splats by depth in screen tiles and blends them front to back. Because there is no network evaluation per pixel, a desktop GPU reaches well above 100 frames per second at 1080p for typical scenes.
The table below lists the parameters stored for every Gaussian and what each one controls.
| Parameter | Typical storage | What it controls |
|---|---|---|
| Position (mean) | 3 floats | Center of the ellipsoid in world space |
| Scale | 3 floats | Extent along the three local axes |
| Rotation | 4 floats (quaternion) | Orientation of the ellipsoid |
| Opacity | 1 float | How strongly the splat occludes what lies behind it |
| Spherical harmonics | Up to 48 floats (degree 3) | Color that changes with viewing direction |
Why the representation suits real-time XR
Headsets must render two views at 72 to 120 Hz while keeping motion-to-photon latency low, so any representation used in extended reality needs predictable, fast rendering. Gaussian splatting meets this requirement because it is explicit geometry processed by rasterization, the same workload GPUs are designed for.
View-dependent color stored as spherical harmonics reproduces reflections, sheen and subtle lighting shifts as the user moves the head, which gives captured spaces a sense of presence that textured meshes often lack. Fine structures such as foliage, hair and fabric edges also look convincing, because many small soft splats approximate them better than a polygon surface.
The representation is also editable at the primitive level: splats can be cropped, deleted, transformed and combined with conventional meshes in engines such as Unity or Unreal Engine through plugins, and web viewers deliver them through WebXR in a browser.
Gaussian splatting compared with related capture methods
Gaussian splatting sits between photogrammetry, which produces explicit meshes, and neural radiance fields, which store the scene implicitly in a network. The comparison below shows where each approach fits in XR development.
| Method | Scene format | Real-time rendering | Best suited for |
|---|---|---|---|
| Gaussian splatting | Explicit ellipsoids | Yes, on current GPUs | Photorealistic walkthroughs and captured environments |
| Photogrammetry mesh | Textured triangles | Yes, very efficient | Physics, collision, measurements and game assets |
| Neural radiance field | Neural network weights | Rarely without baking | Offline novel-view synthesis and research |
| Point cloud | Colored points | Yes | Survey data and quick previews |
The practical consequence is that splats excel at appearance while meshes excel at function. Many production XR apps therefore pair a splat for visuals with a simplified invisible mesh for collision, occlusion and interaction.
Capturing and optimizing splats for headsets
Capture quality decides final quality, because training cannot invent detail that no camera saw. Sharp, evenly exposed images with 60 to 80 percent overlap, locked exposure and white balance, and orbits at several heights give the most stable results.
Raw training output is large, often 200 MB to over 1 GB per scene, which is too heavy for standalone headsets. Developers therefore apply compression and level-of-detail techniques before deployment, as summarized below.
| Step | Typical effect | Trade-off |
|---|---|---|
| Pruning low-impact splats | Removes 30 to 70 percent of Gaussians | Slight loss of fine detail |
| Reducing spherical harmonic degree | Cuts color data by up to about 80 percent | Weaker view-dependent highlights |
| Quantization and compressed formats | Files often 10 to 20 times smaller | Minor color banding |
| Level-of-detail chunks | Keeps frame time stable in large scenes | More complex streaming logic |
| Cropping floaters and background | Fewer artifacts and splats | Manual cleanup effort |
A useful rule for mobile-class headsets is to target roughly 500,000 to 1.5 million visible splats, depending on resolution and the rest of the scene, and to profile sorting cost, which grows with splat count and can dominate the frame on weaker GPUs.
Limitations and common artifacts
Gaussian splatting is not a universal replacement for meshes. Its main weakness is that lighting is baked into the capture: a splat scene cannot be relit realistically without additional research methods, so inserted virtual objects may look mismatched.
The following table links frequent problems to their causes and the usual remedies.
| Symptom | Likely cause | Typical fix |
|---|---|---|
| Floaters in empty space | Poor coverage or moving objects | Add views, mask movers, crop in an editor |
| Blurry or needle-like splats | Views far from the training cameras | Constrain navigation or capture more angles |
| Popping during head motion | Approximate depth sorting | Use per-pixel or stable sorting variants |
| Shimmering at a distance | Aliasing of very small splats | Apply anti-aliased or mip-filtered variants |
| Glossy surfaces look wrong | Limited view-dependent model | Accept limits or use reflection-aware methods |
- No native surface means collision, physics and accurate occlusion need a separate proxy mesh.
- Dynamic content such as moving people requires 4D Gaussian variants that are heavier and less mature.
- Standards are still forming, and file formats such as PLY, SPZ and compressed variants are not yet universally interchangeable, although work on a glTF extension is under way.
Stripped to the essentials
Gaussian splatting turns a set of photos of a real place into a 3D scene you can walk through. Instead of building the scene from flat triangles, it uses millions of tiny, soft, colored blobs that overlap like dabs of paint.
Each blob knows where it sits, how stretched it is, how see-through it is and how its color changes when you look from a different side. A computer adjusts all blobs until the scene matches the photos. The result looks almost like the real place and can be shown fast enough for a headset, but it is better at looking real than at behaving like solid objects.
Where it is applied
A small studio builds a virtual tour of a historic library for a standalone headset. The team photographs the reading room with about 400 images at three heights, trains a splat scene on a desktop GPU in under an hour and obtains roughly 3.5 million Gaussians.
For the headset build, the developers crop the exterior noise, prune weak splats, lower the color detail to degree 1 and compress the file from about 850 MB to 60 MB, ending with about 1.2 million splats. They add an invisible low-poly mesh of the floor and shelves for teleport targets and collision.
The final app holds a steady 72 frames per second, and visitors can lean toward the carved shelves and see the gold lettering catch the light. The only visible compromise is some softness in the far ceiling corners, which the camera rarely reached.