FlashAttention is an IO-aware algorithm for computing exact attention in transformer models that minimizes reads and writes to slow GPU memory by processing attention in small tiles held in fast on-chip memory. In AI infrastructure, it is a foundational kernel that makes long context windows and efficient training practical on current accelerators.
Introduced in 2022 and followed by FlashAttention-2 and FlashAttention-3, it produces the same mathematical result as standard attention, within floating-point rounding, while reducing attention memory from quadratic to linear in sequence length. Most major training frameworks and inference servers now use it or a close derivative by default.
Why standard attention is limited by memory traffic
Standard attention is slow mainly because it moves too much data, not because it performs too much arithmetic. A naive implementation computes the score matrix QK^T, writes it to high-bandwidth memory (HBM), reads it back for the softmax, writes the probabilities and reads them again to multiply by V.
For a sequence of N tokens, that intermediate matrix has N squared entries per head. At 32,000 tokens it holds about one billion values per head, so attention becomes memory-bandwidth bound: the GPU cores wait for data while their compute capacity sits idle.
| Memory level | Typical capacity | Typical bandwidth | Role in FlashAttention |
|---|---|---|---|
| HBM | 40 to 192 GB per GPU | About 1.5 to 8 TB/s | Stores Q, K, V and the output |
| On-chip SRAM | Roughly 100 to 230 KB per SM | An order of magnitude faster | Holds the active tiles |
| Registers | A few hundred KB per SM | Fastest | Accumulates partial results |
How FlashAttention tiles and fuses the computation
FlashAttention restructures attention into a single fused kernel that loads blocks of Q, K and V into on-chip SRAM, computes partial results and writes only the final output back to HBM. The full N by N score matrix is never materialized in slow memory.
The key enabler is the online softmax technique. Softmax normally needs the maximum and the sum over an entire row, but FlashAttention keeps a running maximum and running normalizer for each row and rescales earlier partial outputs whenever a new block changes them. The final result is therefore exact rather than approximate.
In the backward pass, the algorithm stores only the output and small softmax statistics, and it recomputes attention blocks on the fly instead of saving the score matrix. This adds arithmetic but removes far more memory traffic, so the trade is net faster and far smaller.
| Technique | What it does | Benefit |
|---|---|---|
| Tiling | Splits Q, K and V into SRAM-sized blocks | Keeps data on chip |
| Kernel fusion | Combines matmul, mask, softmax and dropout | Avoids intermediate HBM writes |
| Online softmax | Rescales partial sums incrementally | Exact result without full rows |
| Recomputation | Rebuilds scores during backward pass | Linear activation memory |
FlashAttention versions and their improvements
Each FlashAttention version targets better hardware utilization on newer GPU generations while keeping exact attention semantics. The gains come from work partitioning, parallelism and hardware-specific features rather than from changing the mathematics.
| Version | Main change | Target hardware | Typical effect |
|---|---|---|---|
| FlashAttention | Tiling, fusion and recomputation | Ampere-class GPUs such as A100 | About 2 to 4 times faster attention |
| FlashAttention-2 | Fewer non-matmul operations, parallelism over sequence length | A100 and H100 | Roughly 2 times faster than version 1 |
| FlashAttention-3 | Asynchronous data movement, warp specialization, FP8 | Hopper GPUs such as H100 | Higher utilization of tensor cores |
FlashAttention-2 reaches a large share of theoretical peak throughput on A100 for forward passes, while FlashAttention-3 exploits Hopper features such as the Tensor Memory Accelerator and low-precision FP8 arithmetic. Exact speedups depend on head dimension, sequence length, causal masking and batch size, so benchmarks should be repeated on the target workload.
FlashAttention compared with related inference techniques
FlashAttention optimizes how a single attention operation executes, which makes it complementary to most other AI infrastructure techniques rather than a replacement for them. It is frequently combined with KV cache management and parallelism strategies in the same serving stack.
| Technique | Problem addressed | Relationship to FlashAttention |
|---|---|---|
| PagedAttention | Fragmented KV cache memory in serving | Complementary; paged kernels often reuse its ideas |
| KV cache | Repeated computation during decoding | Stored keys and values are read by attention kernels |
| Sparse or linear attention | Quadratic compute cost | Approximate; FlashAttention stays exact |
| Context parallelism | Sequences too long for one GPU | Distributes blocks across GPUs, each using fused kernels |
| Quantization | Model size and bandwidth | FP8 support in version 3 combines both |
During inference, the prefill phase benefits most, because it processes the whole prompt at once. The decoding phase reads a growing KV cache for one new token, where variants such as Flash-Decoding split the work along the key dimension to keep the GPU busy.
Limitations and practical considerations
FlashAttention removes the memory bottleneck of the attention matrix but does not change the quadratic number of operations, so very long contexts still cost substantial compute. Teams adopting it should also check several practical constraints.
- Hardware support varies; newest versions target specific GPU architectures, and older or non-NVIDIA accelerators may need alternative kernels.
- Head dimension limits apply; unusual head sizes may fall back to slower paths.
- Custom attention masks or biases that are not supported by the kernel can silently force a non-fused implementation.
- Numerical differences at the level of rounding are normal and should not be mistaken for bugs.
- Model FLOPs utilization must be measured end to end, since attention is only part of each layer.
In everyday language
Imagine a chef who keeps walking to a distant storeroom for every ingredient. The cooking is quick, but the walking wastes most of the time. FlashAttention is like preparing ingredients in small batches right at the counter, so the chef rarely makes the trip.
A GPU has a small, very fast workspace and a large, slower memory. FlashAttention keeps the attention work in the fast workspace, piece by piece. The answer stays exactly the same. The model simply runs faster and needs much less memory, which lets it read longer documents.
Seen in the real world
A research team fine-tunes a 7-billion-parameter language model to answer questions over long legal contracts. With standard attention on 80 GB GPUs, training at a context length of 16,000 tokens runs out of memory unless the batch size drops to one, and each step is slow.
After the team switches the attention implementation to FlashAttention-2 in its training framework, activation memory for attention falls sharply and the same GPUs fit four sequences per device. Throughput per GPU rises by roughly a factor of two, and the team extends training to 32,000-token sequences without buying additional hardware. Evaluation scores match the earlier run, confirming that the change affected speed and memory, not model accuracy.