Model FLOPs utilization

Also known as: MFU

On this page 7

Model FLOPs utilization (MFU) is the ratio between the floating-point operations per second that a model must perform to make progress and the theoretical peak throughput of the hardware running it. In AI infrastructure it is the standard efficiency metric for large-scale training, because it shows how close a cluster comes to the arithmetic ceiling of its GPUs or other accelerators.

MFU counts only the operations that the model mathematically requires, derived from its architecture and the observed tokens per second, and ignores any extra work the software chooses to do. This makes it a hardware-agnostic and implementation-agnostic yardstick: two training stacks can be compared fairly even when one recomputes activations and the other does not.

How model FLOPs utilization is calculated

MFU is calculated by multiplying the observed throughput in tokens per second by the number of floating-point operations needed per token, then dividing by the aggregate peak FLOPs per second of all devices. The result is a fraction, usually reported as a percentage.

For a dense transformer, the widely used approximation is 6N FLOPs per token for training, where N is the parameter count: about 2N for the forward pass and 4N for the backward pass. Inference needs about 2N FLOPs per token. Attention adds a term that grows with sequence length, roughly proportional to layers times context length times model width, which becomes significant for long contexts.

InputTypical sourceCommon error
Model FLOPs per token6N plus attention term for trainingOmitting attention at long context
Observed throughputTokens processed per second, steady stateIncluding warm-up or evaluation steps
Peak device FLOPsVendor dense figure for the data typeUsing sparsity-doubled peak numbers
Device countAll accelerators in the jobExcluding idle pipeline or spare devices
Inputs to the MFU formula

The choice of peak figure is the most frequent source of inflated or deflated numbers. Vendors often quote a structured sparsity peak that is twice the dense peak, and an MFU calculated against it looks artificially low; a credible report states the data type and dense peak used.

MFU compared with hardware FLOPs utilization

MFU differs from hardware FLOPs utilization (HFU) in what work it counts. HFU includes every operation the hardware executes, including activation recomputation, also called gradient checkpointing, in which parts of the forward pass are repeated during the backward pass to save memory.

MetricWhat it countsMain use
Model FLOPs utilizationOnly operations the model requiresComparing training stacks fairly
Hardware FLOPs utilizationAll executed operations, recomputation includedProfiling kernel efficiency
GPU utilization percentageTime a kernel is runningDetecting idle devices only
Tokens per secondRaw throughputCapacity planning for one setup
Efficiency metrics in AI infrastructure

Because recomputation adds work that does not advance training, HFU is always equal to or higher than MFU. A team that turns on full recomputation can raise HFU while MFU falls, which is why MFU is the better measure of useful progress. The common GPU utilization counter in monitoring tools is a much weaker signal: it reports that a kernel is active, not how efficiently that kernel uses the arithmetic units.

Typical MFU values in training and inference

Realistic MFU values are well below 100 percent, because memory transfers, communication and non-matrix operations consume time without engaging the tensor cores at full rate. Values depend on model size, sequence length, parallelism strategy and interconnect bandwidth.

WorkloadIndicative MFUMain limiting factor
Large dense training, tuned stackAbout 40-60%Communication and pipeline bubbles
Training, default configurationAbout 20-40%Small micro-batches, unfused kernels
Mixture-of-experts trainingOften 20-40%All-to-all traffic and load imbalance
Inference prefill phaseCan exceed 50%Compute-bound large matrix products
Inference decode phaseOften below 10%Memory bandwidth for weights and KV cache
Indicative MFU ranges

The gap between prefill and decode is fundamental. Decoding one token per sequence reads every weight from memory for little arithmetic, so it is memory-bandwidth-bound, and techniques such as continuous batching and speculative decoding raise MFU by increasing arithmetic per byte loaded. For this reason, serving teams usually track tokens per second and cost per token alongside MFU rather than MFU alone.

What lowers MFU in distributed training

MFU in distributed training falls whenever accelerators wait instead of multiplying matrices. Each parallelism strategy introduces its own overhead that competes with computation, and the art of large-scale training lies in overlapping that overhead with useful work.

  • Communication time: all-reduce in data parallelism and FSDP, and frequent collectives in tensor parallelism, stall computation when they cannot be overlapped.
  • Pipeline bubbles: in pipeline parallelism, stages sit idle at the start and end of each batch unless enough micro-batches fill the pipeline.
  • Small matrix shapes: narrow layers or tiny micro-batches cannot saturate the arithmetic units, so larger effective batch sizes usually help.
  • Memory-bound operations: layer normalization, activation functions and unfused attention move data without heavy arithmetic, which fused kernels such as FlashAttention reduce.
  • Input and checkpoint stalls: slow data loading, synchronous checkpoint writes and failure restarts subtract time from the effective throughput.

Because MFU is measured over steady-state training steps, it does not capture downtime from hardware failures. Operators of large clusters therefore also report goodput, the share of wall-clock time that produces retained training progress.

Using MFU to plan compute and cost

MFU links model size, dataset size and hardware into a direct estimate of training time. Total training compute is approximately 6N multiplied by the number of training tokens, and dividing it by peak cluster FLOPs times MFU gives the expected duration.

DecisionHow MFU is usedPractical effect
Cluster sizingConvert training compute into GPU-hoursAvoids under-provisioned budgets
Parallelism layoutCompare MFU across configurationsSelects the fastest sharding plan
Hardware comparisonNormalize throughput by peakSeparates software from hardware gains
Optimization prioritiesTrack MFU after each changeTargets the largest bottleneck first
Decisions informed by MFU

A rise from 30 to 45 percent MFU shortens a fixed training run by one third, so even a few points of improvement translate into large savings at scale. The metric does have a ceiling of relevance: a higher MFU achieved with a less efficient model architecture or a smaller batch that harms convergence can still produce a worse model per dollar.

A non-technical view

Every AI chip has a maximum number of calculations it could do per second if it never had to wait. Model FLOPs utilization tells you what share of that maximum actually goes into useful work for the model.

Think of a delivery truck that could carry ten tons but usually leaves the depot with four. The truck still drives the full route, burns fuel and costs money, so a higher load factor means cheaper deliveries. In the same way, a higher MFU means a model finishes training sooner and costs less, because fewer expensive chip-hours are spent waiting for data or messages from other chips.

A concrete example

An engineering team trains a 7-billion-parameter dense language model on 64 GPUs, each with a dense BF16 peak of about 989 teraFLOPs per second. The cluster peak is therefore about 63 petaFLOPs per second, and each training token requires about 6 x 7 billion, or 42 billion FLOPs.

The first run processes about 560,000 tokens per second, which corresponds to roughly 23.5 petaFLOPs per second of model work and an MFU of about 37 percent. Profiling shows that tensor-parallel communication is crossing node boundaries and that full activation recomputation is enabled even though memory is not tight.

The team keeps tensor parallelism inside each eight-GPU node, switches to selective recomputation and enables fused attention kernels. Throughput rises to about 700,000 tokens per second, an MFU of about 46 percent. For a planned run of 1 trillion tokens, training time drops from roughly 21 days to about 17 days on the same hardware.