Gaussian splatting

Also known as: 3D Gaussian splatting

On this page 7

Gaussian splatting, also called 3D Gaussian splatting, is a scene representation and rendering technique that describes a captured environment as millions of anisotropic 3D Gaussians, each with a position, shape, opacity and view-dependent color, and draws them by rasterizing and alpha-blending them onto the screen. In XR development, it has become a leading way to turn ordinary photos or video of a real place into a photorealistic, navigable 3D scene that runs at headset frame rates.

The method was introduced in 2023 as an alternative to neural radiance fields (NeRF). It keeps the visual quality of learned radiance fields but replaces slow neural network queries with an explicit, GPU-friendly primitive, which is why it fits the tight latency budgets of extended reality.

How Gaussian splatting builds and renders a scene

A Gaussian splatting scene is produced by optimization from posed images, not by manual modeling. The pipeline starts like photogrammetry: a structure-from-motion step, typically with COLMAP or a similar solver, estimates camera poses and a sparse point cloud from 50 to several hundred overlapping photos.

Each sparse point then seeds one Gaussian. During training, a differentiable rasterizer renders the Gaussians from every known camera, compares the result with the real photo and adjusts all parameters by gradient descent. Adaptive density control clones Gaussians in under-reconstructed areas, splits oversized ones and prunes nearly transparent ones, so a final scene usually holds one to six million Gaussians.

Rendering projects each 3D Gaussian into a 2D elliptical splat, sorts the splats by depth in screen tiles and blends them front to back. Because there is no network evaluation per pixel, a desktop GPU reaches well above 100 frames per second at 1080p for typical scenes.

The table below lists the parameters stored for every Gaussian and what each one controls.

ParameterTypical storageWhat it controls
Position (mean)3 floatsCenter of the ellipsoid in world space
Scale3 floatsExtent along the three local axes
Rotation4 floats (quaternion)Orientation of the ellipsoid
Opacity1 floatHow strongly the splat occludes what lies behind it
Spherical harmonicsUp to 48 floats (degree 3)Color that changes with viewing direction
Per-Gaussian parameters

Why the representation suits real-time XR

Headsets must render two views at 72 to 120 Hz while keeping motion-to-photon latency low, so any representation used in extended reality needs predictable, fast rendering. Gaussian splatting meets this requirement because it is explicit geometry processed by rasterization, the same workload GPUs are designed for.

View-dependent color stored as spherical harmonics reproduces reflections, sheen and subtle lighting shifts as the user moves the head, which gives captured spaces a sense of presence that textured meshes often lack. Fine structures such as foliage, hair and fabric edges also look convincing, because many small soft splats approximate them better than a polygon surface.

The representation is also editable at the primitive level: splats can be cropped, deleted, transformed and combined with conventional meshes in engines such as Unity or Unreal Engine through plugins, and web viewers deliver them through WebXR in a browser.

Gaussian splatting sits between photogrammetry, which produces explicit meshes, and neural radiance fields, which store the scene implicitly in a network. The comparison below shows where each approach fits in XR development.

MethodScene formatReal-time renderingBest suited for
Gaussian splattingExplicit ellipsoidsYes, on current GPUsPhotorealistic walkthroughs and captured environments
Photogrammetry meshTextured trianglesYes, very efficientPhysics, collision, measurements and game assets
Neural radiance fieldNeural network weightsRarely without bakingOffline novel-view synthesis and research
Point cloudColored pointsYesSurvey data and quick previews
Scene capture representations compared

The practical consequence is that splats excel at appearance while meshes excel at function. Many production XR apps therefore pair a splat for visuals with a simplified invisible mesh for collision, occlusion and interaction.

Capturing and optimizing splats for headsets

Capture quality decides final quality, because training cannot invent detail that no camera saw. Sharp, evenly exposed images with 60 to 80 percent overlap, locked exposure and white balance, and orbits at several heights give the most stable results.

Raw training output is large, often 200 MB to over 1 GB per scene, which is too heavy for standalone headsets. Developers therefore apply compression and level-of-detail techniques before deployment, as summarized below.

StepTypical effectTrade-off
Pruning low-impact splatsRemoves 30 to 70 percent of GaussiansSlight loss of fine detail
Reducing spherical harmonic degreeCuts color data by up to about 80 percentWeaker view-dependent highlights
Quantization and compressed formatsFiles often 10 to 20 times smallerMinor color banding
Level-of-detail chunksKeeps frame time stable in large scenesMore complex streaming logic
Cropping floaters and backgroundFewer artifacts and splatsManual cleanup effort
Common optimization steps for standalone XR

A useful rule for mobile-class headsets is to target roughly 500,000 to 1.5 million visible splats, depending on resolution and the rest of the scene, and to profile sorting cost, which grows with splat count and can dominate the frame on weaker GPUs.

Limitations and common artifacts

Gaussian splatting is not a universal replacement for meshes. Its main weakness is that lighting is baked into the capture: a splat scene cannot be relit realistically without additional research methods, so inserted virtual objects may look mismatched.

The following table links frequent problems to their causes and the usual remedies.

SymptomLikely causeTypical fix
Floaters in empty spacePoor coverage or moving objectsAdd views, mask movers, crop in an editor
Blurry or needle-like splatsViews far from the training camerasConstrain navigation or capture more angles
Popping during head motionApproximate depth sortingUse per-pixel or stable sorting variants
Shimmering at a distanceAliasing of very small splatsApply anti-aliased or mip-filtered variants
Glossy surfaces look wrongLimited view-dependent modelAccept limits or use reflection-aware methods
Symptoms, causes and fixes
  • No native surface means collision, physics and accurate occlusion need a separate proxy mesh.
  • Dynamic content such as moving people requires 4D Gaussian variants that are heavier and less mature.
  • Standards are still forming, and file formats such as PLY, SPZ and compressed variants are not yet universally interchangeable, although work on a glTF extension is under way.

Stripped to the essentials

Gaussian splatting turns a set of photos of a real place into a 3D scene you can walk through. Instead of building the scene from flat triangles, it uses millions of tiny, soft, colored blobs that overlap like dabs of paint.

Each blob knows where it sits, how stretched it is, how see-through it is and how its color changes when you look from a different side. A computer adjusts all blobs until the scene matches the photos. The result looks almost like the real place and can be shown fast enough for a headset, but it is better at looking real than at behaving like solid objects.

Where it is applied

A small studio builds a virtual tour of a historic library for a standalone headset. The team photographs the reading room with about 400 images at three heights, trains a splat scene on a desktop GPU in under an hour and obtains roughly 3.5 million Gaussians.

For the headset build, the developers crop the exterior noise, prune weak splats, lower the color detail to degree 1 and compress the file from about 850 MB to 60 MB, ending with about 1.2 million splats. They add an invisible low-poly mesh of the floor and shelves for teleport targets and collision.

The final app holds a steady 72 frames per second, and visitors can lean toward the carved shelves and see the gold lettering catch the light. The only visible compromise is some softness in the far ceiling corners, which the camera rarely reached.