Training compute is the total number of floating-point operations (FLOPs) performed to train a machine learning model, from random initialization to the final checkpoint. Often written as training FLOPs, it is the standard unit in AI infrastructure for comparing model scale, estimating cost and planning GPU cluster capacity, and it is distinct from the speed of the hardware, which is measured in FLOP per second.
Training compute links three quantities that every large-scale project must balance: model size, amount of training data and hardware time. Because it is hardware-neutral, it also appears in scaling research and in regulation, where compute thresholds serve as a proxy for model capability.
How training compute is calculated
For dense transformer models, training compute is commonly approximated as C ≈ 6 × N × D, where N is the number of parameters and D is the number of training tokens. The factor of six comes from roughly two operations per parameter per token in the forward pass and about four in the backward pass, which computes gradients for both activations and weights.
The approximation ignores attention cost, which grows with sequence length, and it excludes recomputation from activation checkpointing. For long-context training or mixture-of-experts models, engineers adjust the formula: in a sparse model, N counts only the active parameters per token, not the total parameter count.
The table below shows how the formula translates into orders of magnitude for illustrative model sizes.
| Parameters | Training tokens | Approximate compute |
|---|---|---|
| 1 billion | 20 billion | 1.2 × 10^20 FLOPs |
| 7 billion | 2 trillion | 8.4 × 10^22 FLOPs |
| 70 billion | 15 trillion | 6.3 × 10^24 FLOPs |
| 400 billion | 15 trillion | 3.6 × 10^25 FLOPs |
From FLOPs to GPU hours and cost
Converting training compute into wall-clock time requires the hardware’s peak throughput and the achieved model FLOPs utilization (MFU). Training time equals total FLOPs divided by the number of accelerators, their peak FLOP per second and the MFU, which for well-tuned large runs typically falls between 30 and 50 percent.
Utilization is the factor most often overestimated in budgets. Communication overhead from all-reduce operations, pipeline bubbles, data loading stalls, hardware failures and restarts all reduce effective throughput below the datasheet figure.
| Factor | Typical range | Effect on the budget |
|---|---|---|
| Peak precision throughput | Hundreds to thousands of teraFLOP/s per GPU | Sets the theoretical ceiling |
| Model FLOPs utilization | 30 to 50 percent at scale | Directly scales total GPU hours |
| Numeric precision | BF16, FP8 or mixed | Lower precision raises throughput |
| Cluster size | Hundreds to tens of thousands of GPUs | Shortens time, adds communication cost |
| Goodput loss | Failures, restarts, checkpoint time | Adds idle hours to the bill |
A useful rule is that the final training run is only part of the total. Ablations, hyperparameter searches, failed runs and evaluation often consume a comparable or larger amount of compute than the headline run, so budgets based only on 6ND underestimate project cost.
Compute-optimal scaling
Scaling laws describe how model loss decreases predictably as training compute increases, provided parameters and data grow in balance. Research on compute-optimal training found that, for a fixed budget, parameters and tokens should grow roughly in proportion, at about 20 training tokens per parameter.
In practice many teams deliberately overtrain smaller models on far more tokens than the compute-optimal ratio. The extra training compute raises one-time cost but produces a smaller model that is cheaper to serve, because inference cost scales with parameter count and is paid on every request.
| Strategy | Tokens per parameter | Best suited for |
|---|---|---|
| Compute-optimal | About 20 | Lowest loss for the training budget |
| Inference-optimized overtraining | 100 to several thousand | High-volume serving, on-device AI |
| Undertrained large model | Below 10 | Rarely justified, wastes parameters |
Training compute compared with related measures
Training compute is one of several measures used in AI infrastructure, and confusing them leads to planning errors. The distinctions below are the most frequent sources of misunderstanding.
| Measure | Unit | What it describes |
|---|---|---|
| Training compute | FLOPs (total operations) | Work done to produce the model |
| Hardware throughput | FLOP/s | Speed of a chip or cluster |
| Inference compute | FLOPs per token | Work per generated token, about 2 × N |
| GPU hours | Device-hours | Rented or consumed hardware time |
| Model FLOPs utilization | Percent | Share of peak throughput actually used |
Because training compute is a one-time cost while inference compute recurs with every user request, a widely deployed model can consume more total compute in serving than it did in training.
Limitations and common misconceptions
Training compute is a useful but imperfect proxy for capability. Data quality, architecture, tokenizer design and post-training methods can make two models trained with the same compute behave very differently.
- More compute does not guarantee a better model: poor data or unstable training can waste large budgets.
- Published figures are usually estimates: developers rarely disclose exact token counts or utilization.
- Regulatory thresholds such as 10^25 or 10^26 FLOPs are legal triggers, not measures of risk in themselves.
- Fine-tuning compute is usually orders of magnitude smaller than pretraining compute and is reported separately.
- Algorithmic progress steadily lowers the compute needed to reach a given level of performance.
In plain words
Training an AI model means making a computer do an enormous number of small calculations, over and over, until the model learns patterns in its data. Training compute is simply the total count of those calculations.
Think of it like the total distance driven on a long road trip rather than the top speed of the car. A faster car finishes sooner, but the distance stays the same. In the same way, faster chips finish training sooner, while the training compute stays fixed. The key takeaway is that bigger models trained on more data need more compute, which means more chips, more electricity and more money.
A real-world example
A research lab plans to pretrain a 7-billion-parameter language model on 2 trillion tokens. Using the 6ND approximation, the team estimates about 8.4 × 10^22 FLOPs of training compute before choosing hardware.
The lab rents 512 GPUs, each rated at roughly 1,000 teraFLOP/s in BF16, and assumes 40 percent model FLOPs utilization. That gives an effective throughput of about 2 × 10^17 FLOP/s, so the run needs around 4.1 × 10^5 seconds, or roughly five days of wall-clock time and about 59,000 GPU hours.
The finance team initially budgets only for that figure. The infrastructure lead adds a 50 percent reserve for ablations, restarts and evaluation, based on earlier projects. The run eventually consumes about 85,000 GPU hours in total, and the reserve prevents a mid-project funding gap, showing why planning beyond the headline FLOP count is standard practice.