Neural scaling laws

Also known as: scaling laws

On this page 7

Neural scaling laws are empirical relationships showing that the test loss of a neural network falls as a smooth power law when model size, dataset size or training compute increases. In the training of artificial intelligence systems, and of large language models in particular, they are the main tool for predicting the outcome of an expensive training run from a series of much smaller experiments.

The concept, often shortened to scaling laws, turned model development from guesswork into budget planning. A laboratory can fit a curve on runs that cost a fraction of a percent of the final budget, then extrapolate across several orders of magnitude to choose the model size, the token count and the expected loss before committing thousands of accelerators.

How scaling laws describe loss

A scaling law expresses loss as a function of a resource, typically in the form L(x) = E + A / x^α, where x is the resource, α is the scaling exponent and E is the irreducible loss. On a log-log plot the reducible part of the loss appears as a straight line, which is why the relationship is easy to fit and to extrapolate.

The three resources that matter are the number of non-embedding parameters (N), the number of training tokens (D) and the training compute (C). For dense transformers, compute is approximated as C ≈ 6ND floating-point operations, because each token requires roughly two operations per parameter in the forward pass and four in the backward pass.

The table below summarizes the variables that appear in most scaling-law fits and what each one controls.

SymbolMeaningTypical role
NNon-embedding parametersCapacity of the model
DTraining tokens seenAmount of information absorbed
CTraining compute in FLOPsBudget that links N and D
α, βScaling exponentsSpeed of improvement per resource
EIrreducible lossEntropy of the data itself
Core variables in a scaling-law fit

The exponents are small. Early fits reported values of roughly 0.05 to 0.1 for loss against parameters, data or compute in isolation, which means that a tenfold increase in compute buys only a modest drop in loss. That diminishing return is the central economic fact behind the size of modern training budgets.

From the Kaplan laws to compute-optimal training

Two families of results define the field. The Kaplan et al. study of 2020 found clean power laws across seven orders of magnitude and concluded that, for a fixed budget, parameters should grow faster than data. Its practical consequence was a generation of very large but undertrained models.

The Chinchilla study of 2022 revisited the question with learning-rate schedules matched to each run length and reached a different answer: parameters and tokens should scale in roughly equal proportion with compute. Its widely quoted rule of thumb is about 20 training tokens per parameter for a compute-optimal dense model.

The comparison below shows how the two results differ and why the later one changed industry practice.

AspectKaplan et al. (2020)Chinchilla (2022)
Optimal N versus CGrows faster than dataGrows about as fast as data
Tokens per parameterWell below 20About 20
Learning-rate scheduleFixed length for all runsMatched to each run length
Resulting practiceLarge, undertrained modelsSmaller models on more data
Comparison of two influential scaling studies

The discrepancy is a useful lesson: a scaling law is only as reliable as the experimental protocol behind it. Differences in schedules, tokenizers, embedding accounting and the range of model sizes can shift fitted exponents enough to change the optimal allocation.

Variants beyond pretraining loss

Scaling laws now cover more than pretraining cross-entropy. Researchers fit similar curves to data repetition, downstream accuracy, fine-tuning transfer and the amount of computation spent at inference time. Each variant answers a different planning question.

VariantResource variedPlanning question
Compute-optimal lawsN and D under fixed CHow large a model for this budget
Data-constrained lawsRepeated epochs of dataHow useful repeated tokens are
Transfer lawsPretraining scaleHow much fine-tuning data is saved
Downstream task lawsCompute or lossWhich benchmark score to expect
Inference-time lawsTokens spent reasoningHow much test-time compute pays off
Common scaling-law variants and the questions they answer

Data-constrained studies found that up to about four epochs of repeated data perform almost as well as fresh data, after which returns decline quickly. This matters because high-quality text is finite, and the data mixture increasingly limits scale more than hardware does.

Using scaling laws to plan a training run

In practice, a team trains a ladder of small models, often spanning two to three orders of magnitude in compute, fits the loss curves and selects the configuration for the final training run. The same ladder is used to tune hyperparameters such as learning rate and batch size, which themselves follow predictable trends with scale.

The compute-optimal point is not always the economically optimal one. A model that will serve billions of requests is cheaper overall when it is deliberately overtrained, meaning it receives far more than 20 tokens per parameter, because a smaller model lowers the cost of every inference call. The decision table below captures these trade-offs.

SituationRecommended allocationReason
Research model, little servingNear 20 tokens per parameterLowest loss for the compute spent
High-volume production modelSmaller model, many more tokensInference cost dominates lifetime cost
Limited unique dataLarger model, at most four epochsRepetition loses value quickly
Uncertain architectureFit a new ladder firstExponents differ between designs
Decision guide for allocating a training budget

Extrapolation should stay modest. Practitioners usually trust a fit for one to two orders of magnitude beyond the largest experiment and verify the prediction with an intermediate checkpoint before the full budget is spent.

Limitations and common misconceptions

Scaling laws predict average next-token loss, not specific capabilities. Several frequent misreadings recur in practice:

  • Lower loss does not guarantee a given skill. Benchmark scores can stay flat and then rise sharply, a pattern described as emergence, although part of that effect comes from discontinuous metrics such as exact match.
  • Exponents are not universal constants. They depend on architecture, data quality, tokenizer and optimizer settings, so a fit from one setup should not be reused for another.
  • Curves can break. Instabilities such as a loss spike, data exhaustion or a poor mixture can push a large run off its predicted line.
  • Bigger is not automatically better. Better data or a stronger recipe can shift the entire curve downward, which is often cheaper than moving along it.

Finally, the laws say nothing about safety, factuality or alignment, which depend on post-training methods. Treating predicted loss as a proxy for usefulness is a category error that experienced teams avoid.

Explained without jargon

Neural scaling laws are rules of thumb, measured by experiment, that tell engineers how much better an AI model gets when it is made bigger, given more reading material or trained with more computing power. The surprising finding is that the improvement follows a smooth and predictable curve rather than jumping around.

Think of it like fuel efficiency for learning. By testing several small cars, an engineer can estimate how far a much larger truck will go on a full tank. In the same way, small practice models reveal what a giant model will achieve before anyone pays for it. The catch is that each extra step of improvement costs far more than the previous one.

A typical use case

A research team has a fixed budget of about 10^22 floating-point operations for a new language model. Instead of guessing, it first trains eight small models ranging from 20 million to 1 billion parameters, each with a learning-rate schedule matched to its length, and records the final validation loss of every run.

Fitting the scaling curve suggests that a compute-optimal model would have roughly 9 billion parameters trained on about 180 billion tokens. Because the model will power a busy customer-support assistant, the team instead chooses a 4-billion-parameter model trained on 400 billion tokens, accepting a slightly higher loss in exchange for roughly half the serving cost. An intermediate checkpoint lands within 1 percent of the predicted loss, and the team proceeds with confidence.