Training compute

Also known as: training FLOPs

On this page 7

Training compute is the total number of floating-point operations (FLOPs) performed to train a machine learning model, from random initialization to the final checkpoint. Often written as training FLOPs, it is the standard unit in AI infrastructure for comparing model scale, estimating cost and planning GPU cluster capacity, and it is distinct from the speed of the hardware, which is measured in FLOP per second.

Training compute links three quantities that every large-scale project must balance: model size, amount of training data and hardware time. Because it is hardware-neutral, it also appears in scaling research and in regulation, where compute thresholds serve as a proxy for model capability.

How training compute is calculated

For dense transformer models, training compute is commonly approximated as C ≈ 6 × N × D, where N is the number of parameters and D is the number of training tokens. The factor of six comes from roughly two operations per parameter per token in the forward pass and about four in the backward pass, which computes gradients for both activations and weights.

The approximation ignores attention cost, which grows with sequence length, and it excludes recomputation from activation checkpointing. For long-context training or mixture-of-experts models, engineers adjust the formula: in a sparse model, N counts only the active parameters per token, not the total parameter count.

The table below shows how the formula translates into orders of magnitude for illustrative model sizes.

ParametersTraining tokensApproximate compute
1 billion20 billion1.2 × 10^20 FLOPs
7 billion2 trillion8.4 × 10^22 FLOPs
70 billion15 trillion6.3 × 10^24 FLOPs
400 billion15 trillion3.6 × 10^25 FLOPs
Illustrative training compute for dense models

From FLOPs to GPU hours and cost

Converting training compute into wall-clock time requires the hardware’s peak throughput and the achieved model FLOPs utilization (MFU). Training time equals total FLOPs divided by the number of accelerators, their peak FLOP per second and the MFU, which for well-tuned large runs typically falls between 30 and 50 percent.

Utilization is the factor most often overestimated in budgets. Communication overhead from all-reduce operations, pipeline bubbles, data loading stalls, hardware failures and restarts all reduce effective throughput below the datasheet figure.

FactorTypical rangeEffect on the budget
Peak precision throughputHundreds to thousands of teraFLOP/s per GPUSets the theoretical ceiling
Model FLOPs utilization30 to 50 percent at scaleDirectly scales total GPU hours
Numeric precisionBF16, FP8 or mixedLower precision raises throughput
Cluster sizeHundreds to tens of thousands of GPUsShortens time, adds communication cost
Goodput lossFailures, restarts, checkpoint timeAdds idle hours to the bill
Factors that convert compute into time and cost

A useful rule is that the final training run is only part of the total. Ablations, hyperparameter searches, failed runs and evaluation often consume a comparable or larger amount of compute than the headline run, so budgets based only on 6ND underestimate project cost.

Compute-optimal scaling

Scaling laws describe how model loss decreases predictably as training compute increases, provided parameters and data grow in balance. Research on compute-optimal training found that, for a fixed budget, parameters and tokens should grow roughly in proportion, at about 20 training tokens per parameter.

In practice many teams deliberately overtrain smaller models on far more tokens than the compute-optimal ratio. The extra training compute raises one-time cost but produces a smaller model that is cheaper to serve, because inference cost scales with parameter count and is paid on every request.

StrategyTokens per parameterBest suited for
Compute-optimalAbout 20Lowest loss for the training budget
Inference-optimized overtraining100 to several thousandHigh-volume serving, on-device AI
Undertrained large modelBelow 10Rarely justified, wastes parameters
Allocation strategies for a fixed compute budget

Training compute is one of several measures used in AI infrastructure, and confusing them leads to planning errors. The distinctions below are the most frequent sources of misunderstanding.

MeasureUnitWhat it describes
Training computeFLOPs (total operations)Work done to produce the model
Hardware throughputFLOP/sSpeed of a chip or cluster
Inference computeFLOPs per tokenWork per generated token, about 2 × N
GPU hoursDevice-hoursRented or consumed hardware time
Model FLOPs utilizationPercentShare of peak throughput actually used
Training compute and neighboring measures

Because training compute is a one-time cost while inference compute recurs with every user request, a widely deployed model can consume more total compute in serving than it did in training.

Limitations and common misconceptions

Training compute is a useful but imperfect proxy for capability. Data quality, architecture, tokenizer design and post-training methods can make two models trained with the same compute behave very differently.

  • More compute does not guarantee a better model: poor data or unstable training can waste large budgets.
  • Published figures are usually estimates: developers rarely disclose exact token counts or utilization.
  • Regulatory thresholds such as 10^25 or 10^26 FLOPs are legal triggers, not measures of risk in themselves.
  • Fine-tuning compute is usually orders of magnitude smaller than pretraining compute and is reported separately.
  • Algorithmic progress steadily lowers the compute needed to reach a given level of performance.

In plain words

Training an AI model means making a computer do an enormous number of small calculations, over and over, until the model learns patterns in its data. Training compute is simply the total count of those calculations.

Think of it like the total distance driven on a long road trip rather than the top speed of the car. A faster car finishes sooner, but the distance stays the same. In the same way, faster chips finish training sooner, while the training compute stays fixed. The key takeaway is that bigger models trained on more data need more compute, which means more chips, more electricity and more money.

A real-world example

A research lab plans to pretrain a 7-billion-parameter language model on 2 trillion tokens. Using the 6ND approximation, the team estimates about 8.4 × 10^22 FLOPs of training compute before choosing hardware.

The lab rents 512 GPUs, each rated at roughly 1,000 teraFLOP/s in BF16, and assumes 40 percent model FLOPs utilization. That gives an effective throughput of about 2 × 10^17 FLOP/s, so the run needs around 4.1 × 10^5 seconds, or roughly five days of wall-clock time and about 59,000 GPU hours.

The finance team initially budgets only for that figure. The infrastructure lead adds a 50 percent reserve for ablations, restarts and evaluation, based on earlier projects. The run eventually consumes about 85,000 GPU hours in total, and the reserve prevents a mid-project funding gap, showing why planning beyond the headline FLOP count is standard practice.