FlashAttention

On this page 7

FlashAttention is an IO-aware algorithm for computing exact attention in transformer models that minimizes reads and writes to slow GPU memory by processing attention in small tiles held in fast on-chip memory. In AI infrastructure, it is a foundational kernel that makes long context windows and efficient training practical on current accelerators.

Introduced in 2022 and followed by FlashAttention-2 and FlashAttention-3, it produces the same mathematical result as standard attention, within floating-point rounding, while reducing attention memory from quadratic to linear in sequence length. Most major training frameworks and inference servers now use it or a close derivative by default.

Why standard attention is limited by memory traffic

Standard attention is slow mainly because it moves too much data, not because it performs too much arithmetic. A naive implementation computes the score matrix QK^T, writes it to high-bandwidth memory (HBM), reads it back for the softmax, writes the probabilities and reads them again to multiply by V.

For a sequence of N tokens, that intermediate matrix has N squared entries per head. At 32,000 tokens it holds about one billion values per head, so attention becomes memory-bandwidth bound: the GPU cores wait for data while their compute capacity sits idle.

Memory levelTypical capacityTypical bandwidthRole in FlashAttention
HBM40 to 192 GB per GPUAbout 1.5 to 8 TB/sStores Q, K, V and the output
On-chip SRAMRoughly 100 to 230 KB per SMAn order of magnitude fasterHolds the active tiles
RegistersA few hundred KB per SMFastestAccumulates partial results
GPU memory levels relevant to attention

How FlashAttention tiles and fuses the computation

FlashAttention restructures attention into a single fused kernel that loads blocks of Q, K and V into on-chip SRAM, computes partial results and writes only the final output back to HBM. The full N by N score matrix is never materialized in slow memory.

The key enabler is the online softmax technique. Softmax normally needs the maximum and the sum over an entire row, but FlashAttention keeps a running maximum and running normalizer for each row and rescales earlier partial outputs whenever a new block changes them. The final result is therefore exact rather than approximate.

In the backward pass, the algorithm stores only the output and small softmax statistics, and it recomputes attention blocks on the fly instead of saving the score matrix. This adds arithmetic but removes far more memory traffic, so the trade is net faster and far smaller.

TechniqueWhat it doesBenefit
TilingSplits Q, K and V into SRAM-sized blocksKeeps data on chip
Kernel fusionCombines matmul, mask, softmax and dropoutAvoids intermediate HBM writes
Online softmaxRescales partial sums incrementallyExact result without full rows
RecomputationRebuilds scores during backward passLinear activation memory
Core techniques inside FlashAttention

FlashAttention versions and their improvements

Each FlashAttention version targets better hardware utilization on newer GPU generations while keeping exact attention semantics. The gains come from work partitioning, parallelism and hardware-specific features rather than from changing the mathematics.

VersionMain changeTarget hardwareTypical effect
FlashAttentionTiling, fusion and recomputationAmpere-class GPUs such as A100About 2 to 4 times faster attention
FlashAttention-2Fewer non-matmul operations, parallelism over sequence lengthA100 and H100Roughly 2 times faster than version 1
FlashAttention-3Asynchronous data movement, warp specialization, FP8Hopper GPUs such as H100Higher utilization of tensor cores
Evolution of FlashAttention

FlashAttention-2 reaches a large share of theoretical peak throughput on A100 for forward passes, while FlashAttention-3 exploits Hopper features such as the Tensor Memory Accelerator and low-precision FP8 arithmetic. Exact speedups depend on head dimension, sequence length, causal masking and batch size, so benchmarks should be repeated on the target workload.

FlashAttention optimizes how a single attention operation executes, which makes it complementary to most other AI infrastructure techniques rather than a replacement for them. It is frequently combined with KV cache management and parallelism strategies in the same serving stack.

TechniqueProblem addressedRelationship to FlashAttention
PagedAttentionFragmented KV cache memory in servingComplementary; paged kernels often reuse its ideas
KV cacheRepeated computation during decodingStored keys and values are read by attention kernels
Sparse or linear attentionQuadratic compute costApproximate; FlashAttention stays exact
Context parallelismSequences too long for one GPUDistributes blocks across GPUs, each using fused kernels
QuantizationModel size and bandwidthFP8 support in version 3 combines both
How FlashAttention relates to neighboring techniques

During inference, the prefill phase benefits most, because it processes the whole prompt at once. The decoding phase reads a growing KV cache for one new token, where variants such as Flash-Decoding split the work along the key dimension to keep the GPU busy.

Limitations and practical considerations

FlashAttention removes the memory bottleneck of the attention matrix but does not change the quadratic number of operations, so very long contexts still cost substantial compute. Teams adopting it should also check several practical constraints.

  • Hardware support varies; newest versions target specific GPU architectures, and older or non-NVIDIA accelerators may need alternative kernels.
  • Head dimension limits apply; unusual head sizes may fall back to slower paths.
  • Custom attention masks or biases that are not supported by the kernel can silently force a non-fused implementation.
  • Numerical differences at the level of rounding are normal and should not be mistaken for bugs.
  • Model FLOPs utilization must be measured end to end, since attention is only part of each layer.

In everyday language

Imagine a chef who keeps walking to a distant storeroom for every ingredient. The cooking is quick, but the walking wastes most of the time. FlashAttention is like preparing ingredients in small batches right at the counter, so the chef rarely makes the trip.

A GPU has a small, very fast workspace and a large, slower memory. FlashAttention keeps the attention work in the fast workspace, piece by piece. The answer stays exactly the same. The model simply runs faster and needs much less memory, which lets it read longer documents.

Seen in the real world

A research team fine-tunes a 7-billion-parameter language model to answer questions over long legal contracts. With standard attention on 80 GB GPUs, training at a context length of 16,000 tokens runs out of memory unless the batch size drops to one, and each step is slow.

After the team switches the attention implementation to FlashAttention-2 in its training framework, activation memory for attention falls sharply and the same GPUs fit four sequences per device. Throughput per GPU rises by roughly a factor of two, and the team extends training to 32,000-token sequences without buying additional hardware. Evaluation scores match the earlier run, confirming that the change affected speed and memory, not model accuracy.