Model FLOPs utilization (MFU) is the ratio between the floating-point operations per second that a model must perform to make progress and the theoretical peak throughput of the hardware running it. In AI infrastructure it is the standard efficiency metric for large-scale training, because it shows how close a cluster comes to the arithmetic ceiling of its GPUs or other accelerators.
MFU counts only the operations that the model mathematically requires, derived from its architecture and the observed tokens per second, and ignores any extra work the software chooses to do. This makes it a hardware-agnostic and implementation-agnostic yardstick: two training stacks can be compared fairly even when one recomputes activations and the other does not.
How model FLOPs utilization is calculated
MFU is calculated by multiplying the observed throughput in tokens per second by the number of floating-point operations needed per token, then dividing by the aggregate peak FLOPs per second of all devices. The result is a fraction, usually reported as a percentage.
For a dense transformer, the widely used approximation is 6N FLOPs per token for training, where N is the parameter count: about 2N for the forward pass and 4N for the backward pass. Inference needs about 2N FLOPs per token. Attention adds a term that grows with sequence length, roughly proportional to layers times context length times model width, which becomes significant for long contexts.
| Input | Typical source | Common error |
|---|---|---|
| Model FLOPs per token | 6N plus attention term for training | Omitting attention at long context |
| Observed throughput | Tokens processed per second, steady state | Including warm-up or evaluation steps |
| Peak device FLOPs | Vendor dense figure for the data type | Using sparsity-doubled peak numbers |
| Device count | All accelerators in the job | Excluding idle pipeline or spare devices |
The choice of peak figure is the most frequent source of inflated or deflated numbers. Vendors often quote a structured sparsity peak that is twice the dense peak, and an MFU calculated against it looks artificially low; a credible report states the data type and dense peak used.
MFU compared with hardware FLOPs utilization
MFU differs from hardware FLOPs utilization (HFU) in what work it counts. HFU includes every operation the hardware executes, including activation recomputation, also called gradient checkpointing, in which parts of the forward pass are repeated during the backward pass to save memory.
| Metric | What it counts | Main use |
|---|---|---|
| Model FLOPs utilization | Only operations the model requires | Comparing training stacks fairly |
| Hardware FLOPs utilization | All executed operations, recomputation included | Profiling kernel efficiency |
| GPU utilization percentage | Time a kernel is running | Detecting idle devices only |
| Tokens per second | Raw throughput | Capacity planning for one setup |
Because recomputation adds work that does not advance training, HFU is always equal to or higher than MFU. A team that turns on full recomputation can raise HFU while MFU falls, which is why MFU is the better measure of useful progress. The common GPU utilization counter in monitoring tools is a much weaker signal: it reports that a kernel is active, not how efficiently that kernel uses the arithmetic units.
Typical MFU values in training and inference
Realistic MFU values are well below 100 percent, because memory transfers, communication and non-matrix operations consume time without engaging the tensor cores at full rate. Values depend on model size, sequence length, parallelism strategy and interconnect bandwidth.
| Workload | Indicative MFU | Main limiting factor |
|---|---|---|
| Large dense training, tuned stack | About 40-60% | Communication and pipeline bubbles |
| Training, default configuration | About 20-40% | Small micro-batches, unfused kernels |
| Mixture-of-experts training | Often 20-40% | All-to-all traffic and load imbalance |
| Inference prefill phase | Can exceed 50% | Compute-bound large matrix products |
| Inference decode phase | Often below 10% | Memory bandwidth for weights and KV cache |
The gap between prefill and decode is fundamental. Decoding one token per sequence reads every weight from memory for little arithmetic, so it is memory-bandwidth-bound, and techniques such as continuous batching and speculative decoding raise MFU by increasing arithmetic per byte loaded. For this reason, serving teams usually track tokens per second and cost per token alongside MFU rather than MFU alone.
What lowers MFU in distributed training
MFU in distributed training falls whenever accelerators wait instead of multiplying matrices. Each parallelism strategy introduces its own overhead that competes with computation, and the art of large-scale training lies in overlapping that overhead with useful work.
- Communication time: all-reduce in data parallelism and FSDP, and frequent collectives in tensor parallelism, stall computation when they cannot be overlapped.
- Pipeline bubbles: in pipeline parallelism, stages sit idle at the start and end of each batch unless enough micro-batches fill the pipeline.
- Small matrix shapes: narrow layers or tiny micro-batches cannot saturate the arithmetic units, so larger effective batch sizes usually help.
- Memory-bound operations: layer normalization, activation functions and unfused attention move data without heavy arithmetic, which fused kernels such as FlashAttention reduce.
- Input and checkpoint stalls: slow data loading, synchronous checkpoint writes and failure restarts subtract time from the effective throughput.
Because MFU is measured over steady-state training steps, it does not capture downtime from hardware failures. Operators of large clusters therefore also report goodput, the share of wall-clock time that produces retained training progress.
Using MFU to plan compute and cost
MFU links model size, dataset size and hardware into a direct estimate of training time. Total training compute is approximately 6N multiplied by the number of training tokens, and dividing it by peak cluster FLOPs times MFU gives the expected duration.
| Decision | How MFU is used | Practical effect |
|---|---|---|
| Cluster sizing | Convert training compute into GPU-hours | Avoids under-provisioned budgets |
| Parallelism layout | Compare MFU across configurations | Selects the fastest sharding plan |
| Hardware comparison | Normalize throughput by peak | Separates software from hardware gains |
| Optimization priorities | Track MFU after each change | Targets the largest bottleneck first |
A rise from 30 to 45 percent MFU shortens a fixed training run by one third, so even a few points of improvement translate into large savings at scale. The metric does have a ceiling of relevance: a higher MFU achieved with a less efficient model architecture or a smaller batch that harms convergence can still produce a worse model per dollar.
A non-technical view
Every AI chip has a maximum number of calculations it could do per second if it never had to wait. Model FLOPs utilization tells you what share of that maximum actually goes into useful work for the model.
Think of a delivery truck that could carry ten tons but usually leaves the depot with four. The truck still drives the full route, burns fuel and costs money, so a higher load factor means cheaper deliveries. In the same way, a higher MFU means a model finishes training sooner and costs less, because fewer expensive chip-hours are spent waiting for data or messages from other chips.
A concrete example
An engineering team trains a 7-billion-parameter dense language model on 64 GPUs, each with a dense BF16 peak of about 989 teraFLOPs per second. The cluster peak is therefore about 63 petaFLOPs per second, and each training token requires about 6 x 7 billion, or 42 billion FLOPs.
The first run processes about 560,000 tokens per second, which corresponds to roughly 23.5 petaFLOPs per second of model work and an MFU of about 37 percent. Profiling shows that tensor-parallel communication is crossing node boundaries and that full activation recomputation is enabled even though memory is not tight.
The team keeps tensor parallelism inside each eight-GPU node, switches to selective recomputation and enables fused attention kernels. Throughput rises to about 700,000 tokens per second, an MFU of about 46 percent. For a planned run of 1 trillion tokens, training time drops from roughly 21 days to about 17 days on the same hardware.