Neural scaling laws are empirical relationships showing that the test loss of a neural network falls as a smooth power law when model size, dataset size or training compute increases. In the training of artificial intelligence systems, and of large language models in particular, they are the main tool for predicting the outcome of an expensive training run from a series of much smaller experiments.
The concept, often shortened to scaling laws, turned model development from guesswork into budget planning. A laboratory can fit a curve on runs that cost a fraction of a percent of the final budget, then extrapolate across several orders of magnitude to choose the model size, the token count and the expected loss before committing thousands of accelerators.
How scaling laws describe loss
A scaling law expresses loss as a function of a resource, typically in the form L(x) = E + A / x^α, where x is the resource, α is the scaling exponent and E is the irreducible loss. On a log-log plot the reducible part of the loss appears as a straight line, which is why the relationship is easy to fit and to extrapolate.
The three resources that matter are the number of non-embedding parameters (N), the number of training tokens (D) and the training compute (C). For dense transformers, compute is approximated as C ≈ 6ND floating-point operations, because each token requires roughly two operations per parameter in the forward pass and four in the backward pass.
The table below summarizes the variables that appear in most scaling-law fits and what each one controls.
| Symbol | Meaning | Typical role |
|---|---|---|
| N | Non-embedding parameters | Capacity of the model |
| D | Training tokens seen | Amount of information absorbed |
| C | Training compute in FLOPs | Budget that links N and D |
| α, β | Scaling exponents | Speed of improvement per resource |
| E | Irreducible loss | Entropy of the data itself |
The exponents are small. Early fits reported values of roughly 0.05 to 0.1 for loss against parameters, data or compute in isolation, which means that a tenfold increase in compute buys only a modest drop in loss. That diminishing return is the central economic fact behind the size of modern training budgets.
From the Kaplan laws to compute-optimal training
Two families of results define the field. The Kaplan et al. study of 2020 found clean power laws across seven orders of magnitude and concluded that, for a fixed budget, parameters should grow faster than data. Its practical consequence was a generation of very large but undertrained models.
The Chinchilla study of 2022 revisited the question with learning-rate schedules matched to each run length and reached a different answer: parameters and tokens should scale in roughly equal proportion with compute. Its widely quoted rule of thumb is about 20 training tokens per parameter for a compute-optimal dense model.
The comparison below shows how the two results differ and why the later one changed industry practice.
| Aspect | Kaplan et al. (2020) | Chinchilla (2022) |
|---|---|---|
| Optimal N versus C | Grows faster than data | Grows about as fast as data |
| Tokens per parameter | Well below 20 | About 20 |
| Learning-rate schedule | Fixed length for all runs | Matched to each run length |
| Resulting practice | Large, undertrained models | Smaller models on more data |
The discrepancy is a useful lesson: a scaling law is only as reliable as the experimental protocol behind it. Differences in schedules, tokenizers, embedding accounting and the range of model sizes can shift fitted exponents enough to change the optimal allocation.
Variants beyond pretraining loss
Scaling laws now cover more than pretraining cross-entropy. Researchers fit similar curves to data repetition, downstream accuracy, fine-tuning transfer and the amount of computation spent at inference time. Each variant answers a different planning question.
| Variant | Resource varied | Planning question |
|---|---|---|
| Compute-optimal laws | N and D under fixed C | How large a model for this budget |
| Data-constrained laws | Repeated epochs of data | How useful repeated tokens are |
| Transfer laws | Pretraining scale | How much fine-tuning data is saved |
| Downstream task laws | Compute or loss | Which benchmark score to expect |
| Inference-time laws | Tokens spent reasoning | How much test-time compute pays off |
Data-constrained studies found that up to about four epochs of repeated data perform almost as well as fresh data, after which returns decline quickly. This matters because high-quality text is finite, and the data mixture increasingly limits scale more than hardware does.
Using scaling laws to plan a training run
In practice, a team trains a ladder of small models, often spanning two to three orders of magnitude in compute, fits the loss curves and selects the configuration for the final training run. The same ladder is used to tune hyperparameters such as learning rate and batch size, which themselves follow predictable trends with scale.
The compute-optimal point is not always the economically optimal one. A model that will serve billions of requests is cheaper overall when it is deliberately overtrained, meaning it receives far more than 20 tokens per parameter, because a smaller model lowers the cost of every inference call. The decision table below captures these trade-offs.
| Situation | Recommended allocation | Reason |
|---|---|---|
| Research model, little serving | Near 20 tokens per parameter | Lowest loss for the compute spent |
| High-volume production model | Smaller model, many more tokens | Inference cost dominates lifetime cost |
| Limited unique data | Larger model, at most four epochs | Repetition loses value quickly |
| Uncertain architecture | Fit a new ladder first | Exponents differ between designs |
Extrapolation should stay modest. Practitioners usually trust a fit for one to two orders of magnitude beyond the largest experiment and verify the prediction with an intermediate checkpoint before the full budget is spent.
Limitations and common misconceptions
Scaling laws predict average next-token loss, not specific capabilities. Several frequent misreadings recur in practice:
- Lower loss does not guarantee a given skill. Benchmark scores can stay flat and then rise sharply, a pattern described as emergence, although part of that effect comes from discontinuous metrics such as exact match.
- Exponents are not universal constants. They depend on architecture, data quality, tokenizer and optimizer settings, so a fit from one setup should not be reused for another.
- Curves can break. Instabilities such as a loss spike, data exhaustion or a poor mixture can push a large run off its predicted line.
- Bigger is not automatically better. Better data or a stronger recipe can shift the entire curve downward, which is often cheaper than moving along it.
Finally, the laws say nothing about safety, factuality or alignment, which depend on post-training methods. Treating predicted loss as a proxy for usefulness is a category error that experienced teams avoid.
Explained without jargon
Neural scaling laws are rules of thumb, measured by experiment, that tell engineers how much better an AI model gets when it is made bigger, given more reading material or trained with more computing power. The surprising finding is that the improvement follows a smooth and predictable curve rather than jumping around.
Think of it like fuel efficiency for learning. By testing several small cars, an engineer can estimate how far a much larger truck will go on a full tank. In the same way, small practice models reveal what a giant model will achieve before anyone pays for it. The catch is that each extra step of improvement costs far more than the previous one.
A typical use case
A research team has a fixed budget of about 10^22 floating-point operations for a new language model. Instead of guessing, it first trains eight small models ranging from 20 million to 1 billion parameters, each with a learning-rate schedule matched to its length, and records the final validation loss of every run.
Fitting the scaling curve suggests that a compute-optimal model would have roughly 9 billion parameters trained on about 180 billion tokens. Because the model will power a busy customer-support assistant, the team instead chooses a 4-billion-parameter model trained on 400 billion tokens, accepting a slightly higher loss in exchange for roughly half the serving cost. An intermediate checkpoint lands within 1 percent of the predicted loss, and the team proceeds with confidence.