Sliding window attention is a form of local attention in which each token attends only to a fixed number of nearby tokens, typically the previous w positions, instead of the entire sequence. In neural networks built on the transformer architecture, it reduces the cost of self-attention from quadratic to linear in sequence length while preserving strong modeling of local context.
The technique became prominent through long-document encoders such as Longformer and decoder language models such as Mistral 7B and several Gemma variants. It is a specific, structured case of sparse attention, and it is often combined with global attention layers so that information can still travel across the whole input.
How sliding window attention restricts the attention pattern
Sliding window attention applies a band-shaped mask to the attention score matrix. For a query at position i, keys outside the range from i - w to i (for causal models) or from i - w/2 to i + w/2 (for bidirectional encoders) receive a score of negative infinity before the softmax, so they contribute nothing to the output.
Because every query interacts with at most w keys, compute and memory scale as O(n · w) rather than O(n²). With a sequence of 32,768 tokens and a window of 4,096, each layer evaluates roughly one eighth of the full score matrix, and the saving grows as the sequence lengthens.
The table below lists the main parameters that define a sliding window attention layer and what each one controls.
| Parameter | Typical values | What it affects |
|---|---|---|
| Window size | 128 to 8,192 tokens | Local context per layer and cost |
| Direction | Causal or bidirectional | Use in decoders or encoders |
| Dilation rate | 1 to 8 | Reach per layer at fixed cost |
| Layer pattern | All layers or interleaved | Balance of local and global context |
| Global tokens | None to a few hundred | Shortcuts for long-range information |
Effective receptive field across stacked layers
A single layer sees only w tokens, but stacking layers expands the effective receptive field. After L layers, information can in principle propagate across about L × w positions, much as stacked convolutions widen the receptive field of a convolutional network. A 32-layer model with a 4,096-token window therefore has a theoretical reach of roughly 131,000 tokens.
This theoretical reach is an upper bound, not a guarantee. Information must pass through many intermediate hidden states, and each hop dilutes it, so long-range retrieval is weaker than in full attention. Benchmarks that ask a model to recall a single fact from far back in the context expose this limitation clearly.
Variants and how they compare with full attention
Designers rarely use pure sliding window attention in every layer. Most production models mix it with other patterns to recover global connectivity. The following comparison shows the most common choices.
| Pattern | Cost per layer | Long-range access | Typical use |
|---|---|---|---|
| Full self-attention | O(n²) | Direct | Short and medium contexts |
| Sliding window attention | O(n · w) | Indirect, through layers | Long sequences with local structure |
| Dilated sliding window | O(n · w) | Wider reach, gaps in coverage | Upper layers of long encoders |
| Window plus global tokens | O(n · (w + g)) | Through designated tokens | Document classification and QA |
| Interleaved local and global layers | Mixed | Direct in global layers | Modern long-context decoders |
| Linear attention | O(n) | Compressed approximation | Very long streams |
Interleaving has become the dominant compromise. A model may use a local window in most layers and full attention in, for example, one layer out of every four to six, keeping most of the efficiency gain while preserving direct access to distant tokens.
Memory, the KV cache and inference behavior
During autoregressive generation, sliding window attention allows a rolling buffer KV cache. Keys and values older than the window are never read again, so the cache can be a fixed-size ring buffer of w entries per layer. This keeps inference memory constant regardless of how long generation continues, which is valuable on memory-limited accelerators.
Efficient implementations rely on block-sparse kernels. FlashAttention and similar kernels skip tiles of the score matrix that fall entirely outside the band, so the theoretical saving becomes a real speedup. Naive implementations that compute the full matrix and then mask it gain nothing in compute and only a little in clarity.
The next table summarizes practical effects on training and inference.
| Aspect | Effect | Condition |
|---|---|---|
| Training compute | Grows linearly with length | Block-sparse kernel available |
| KV cache size | Fixed at window length | Rolling buffer implemented |
| Throughput | Higher for long prompts | Sequences much longer than the window |
| Short sequences | No benefit | Length below window size |
Limitations and common design mistakes
The main weakness of sliding window attention is lossy long-range dependency. Tasks such as multi-document question answering, code referencing distant definitions, or retrieval of a detail from the start of a long contract can fail when no layer has direct global access.
| Mistake | Symptom | Fix |
|---|---|---|
| Window too small | Loss of paragraph-level coherence | Increase window or add global layers |
| Only local layers | Poor recall of distant facts | Interleave full attention layers |
| Dense masked kernel | No speed gain | Use block-sparse attention kernels |
| Positional mismatch | Degradation beyond training length | Align rotary embedding scaling with the window |
A further subtlety concerns attention sinks: models tend to place large attention weight on the first tokens, and evicting them from a rolling cache can destabilize generation. Keeping a few initial tokens permanently cached is a widely used remedy.
Understanding it simply
Imagine reading a very long book while only being allowed to look at the last few pages at any moment. You can still follow the story, because each page builds on the previous ones, but you cannot instantly flip back to chapter one.
Sliding window attention works the same way. Each word looks only at its recent neighbors, which makes processing long texts much faster and cheaper. The cost is that distant details can fade, so designers often add occasional layers that see everything.
Applied in practice
An engineering team deploying a 7-billion-parameter language model for summarizing long support transcripts on a single 24 GB GPU faces a memory problem: full attention over 32,000-token conversations makes the KV cache too large to fit alongside the model weights.
They choose a model that uses sliding window attention with a 4,096-token window and a rolling cache. Memory use stays flat during generation, and throughput on long transcripts rises noticeably. Testing shows one weakness: questions about details from the first minutes of a call are answered less reliably. The team adds a short extractive step that copies key facts to the end of the prompt, and the final result is stable long-context summarization on modest hardware.