Neural radiance field

Also known as: NeRF

On this page 7

A neural radiance field (NeRF) is a neural network that represents a three-dimensional scene as a continuous function from position and viewing direction to color and volume density, learned from a set of posed photographs. In computer vision, NeRF is a leading method for novel view synthesis: once trained, it renders photorealistic images of the scene from camera positions that were never photographed.

Introduced in 2020, the approach replaced explicit meshes and point clouds with an implicit neural representation optimized through differentiable volume rendering, and it triggered a large family of faster and more robust variants.

How a neural radiance field represents a scene

The core of a NeRF is a multilayer perceptron that takes a 3D point (x, y, z) and a viewing direction and returns an RGB color and a density value sigma. Density depends only on position, which keeps geometry consistent across views, while color also depends on direction, which lets the model reproduce view-dependent effects such as specular highlights and reflections.

Raw coordinates are passed through positional encoding, a set of sine and cosine functions at increasing frequencies, before entering the network. Without this encoding, the network is biased toward smooth functions and renders blurry images; with it, a network of roughly eight layers and 256 units can capture fine texture. The table summarizes the main inputs, outputs and components.

ComponentRoleTypical setting
Input position3D sample point along a camera rayEncoded with about 10 frequency bands
Input directionViewing direction of the rayEncoded with about 4 frequency bands
Density outputHow much a point blocks lightNon-negative scalar, view independent
Color outputEmitted RGB radianceView dependent, values in 0 to 1
Sampling schemePoints evaluated per rayCoarse and fine passes, about 64 plus 128
Components of the original NeRF model

Volume rendering and training

A NeRF produces an image by casting one ray per pixel through the scene and sampling points along it. Each sample’s color is weighted by its density and by the transmittance, the probability that light reaches that point without being absorbed earlier. Summing these weighted colors gives the pixel color, a numerical approximation of the classical volume rendering integral.

Because every step is differentiable, the network is trained by minimizing the photometric error between rendered pixels and the real photographs. Training requires accurate camera poses, which are usually estimated beforehand with structure from motion tools; a typical capture uses 50 to 200 images. Hierarchical sampling concentrates the fine samples where the coarse pass found high density, which saves computation on empty space.

The following table lists the inputs a NeRF pipeline needs and how each one affects quality.

InputRequirementEffect when poor
PhotographsSharp, overlapping, consistent exposureBlur and floating artifacts
Camera posesAccurate intrinsics and extrinsicsGhosting and doubled geometry
View coverageMany angles around the subjectDistortion in unseen regions
Static sceneNo moving objects or lighting changesTransient blobs and flicker
Scene boundsKnown near and far ray limitsWasted samples and lower detail
Inputs to NeRF training and their effect

Variants and faster successors

The original NeRF needed many hours to a day of GPU training per scene and seconds per rendered frame. Later work attacked speed, scale and robustness, often by replacing the large network with explicit data structures such as voxel grids or hash tables that store learned features.

VariantMain ideaBenefit
Instant neural graphics primitivesMultiresolution hash encoding with a tiny networkTraining in seconds to minutes
Mip-NeRFRenders cones instead of raysReduces aliasing across scales
NeRF in the WildPer-image appearance and transient embeddingsHandles tourist photos with varied lighting
PlenoxelsSparse voxel grid without a neural networkFast optimization, simple structure
Dynamic NeRFsAdds time or deformation fieldsModels moving scenes
Notable NeRF variants

Since 2023, 3D Gaussian splatting has become a strong alternative. It represents a scene with millions of explicit anisotropic Gaussians and rasterizes them, reaching real-time rendering with comparable quality. Many practitioners now treat NeRF and Gaussian splatting as related members of the broader family of radiance field methods.

How NeRF compares with other 3D reconstruction methods

NeRF differs from classical 3D reconstruction because it optimizes for image appearance rather than explicit surfaces. Multi-view stereo and photogrammetry produce point clouds and meshes that are easy to edit and measure, whereas a NeRF excels at reproducing thin structures, transparency and reflections that meshes handle poorly.

TechniqueScene representationStrengthWeakness
NeRFImplicit neural fieldPhotorealistic novel viewsSlow rendering, hard to edit
PhotogrammetryTextured meshMeasurable, editable geometryStruggles with shiny or thin objects
Gaussian splattingExplicit 3D GaussiansReal-time renderingLarge files, less clean geometry
Depth estimationPer-image depth mapSingle image inputNo consistent full scene model
NeRF compared with related techniques

Limitations and practical considerations

A standard NeRF is one network per scene: it does not generalize to new scenes without retraining, although generalizable variants condition on input images. Geometry extracted from the density field is often noisy, so applications needing precise surfaces usually add signed distance function methods or convert results with marching cubes and accept some loss of detail.

  • Floaters: semi-transparent blobs appear in poorly observed space; regularization and better coverage reduce them.
  • Baked lighting: the model stores appearance under capture lighting, so relighting requires separate inverse rendering approaches.
  • Pose sensitivity: small calibration errors cause blur; joint pose refinement can partly compensate.
  • Sparse views: with fewer than about 10 images, quality drops sharply unless priors from pretrained models are added.

In simple terms

A neural radiance field is a way to turn a set of photos into a 3D scene you can fly through. You walk around an object taking pictures, and a computer learns what color and how solid every point in space is.

Once it has learned this, it can draw the scene from any new angle, including angles nobody photographed. It is like a painter who has studied a room from many corners and can now paint it convincingly from a spot where they never stood.

A practical example

A real estate visualization studio needs an interactive walkthrough of a furnished apartment with glass tables and polished floors that its photogrammetry pipeline renders badly. A technician captures about 180 photographs with a mirrorless camera at fixed exposure, moving through each room at two heights.

Camera poses are computed with structure from motion, and the scene is trained with a hash-encoded NeRF in about 20 minutes on a single workstation GPU. The result reproduces reflections on the floor and the transparency of the glass tables, and the studio exports smooth video flythroughs at 4K resolution. Rooms with a moving curtain show some ghosting, so the team recaptures them with windows closed, which removes the artifact.