Sliding window attention

On this page 7

Sliding window attention is a form of local attention in which each token attends only to a fixed number of nearby tokens, typically the previous w positions, instead of the entire sequence. In neural networks built on the transformer architecture, it reduces the cost of self-attention from quadratic to linear in sequence length while preserving strong modeling of local context.

The technique became prominent through long-document encoders such as Longformer and decoder language models such as Mistral 7B and several Gemma variants. It is a specific, structured case of sparse attention, and it is often combined with global attention layers so that information can still travel across the whole input.

How sliding window attention restricts the attention pattern

Sliding window attention applies a band-shaped mask to the attention score matrix. For a query at position i, keys outside the range from i - w to i (for causal models) or from i - w/2 to i + w/2 (for bidirectional encoders) receive a score of negative infinity before the softmax, so they contribute nothing to the output.

Because every query interacts with at most w keys, compute and memory scale as O(n · w) rather than O(n²). With a sequence of 32,768 tokens and a window of 4,096, each layer evaluates roughly one eighth of the full score matrix, and the saving grows as the sequence lengthens.

The table below lists the main parameters that define a sliding window attention layer and what each one controls.

ParameterTypical valuesWhat it affects
Window size128 to 8,192 tokensLocal context per layer and cost
DirectionCausal or bidirectionalUse in decoders or encoders
Dilation rate1 to 8Reach per layer at fixed cost
Layer patternAll layers or interleavedBalance of local and global context
Global tokensNone to a few hundredShortcuts for long-range information
Key parameters of sliding window attention

Effective receptive field across stacked layers

A single layer sees only w tokens, but stacking layers expands the effective receptive field. After L layers, information can in principle propagate across about L × w positions, much as stacked convolutions widen the receptive field of a convolutional network. A 32-layer model with a 4,096-token window therefore has a theoretical reach of roughly 131,000 tokens.

This theoretical reach is an upper bound, not a guarantee. Information must pass through many intermediate hidden states, and each hop dilutes it, so long-range retrieval is weaker than in full attention. Benchmarks that ask a model to recall a single fact from far back in the context expose this limitation clearly.

Variants and how they compare with full attention

Designers rarely use pure sliding window attention in every layer. Most production models mix it with other patterns to recover global connectivity. The following comparison shows the most common choices.

PatternCost per layerLong-range accessTypical use
Full self-attentionO(n²)DirectShort and medium contexts
Sliding window attentionO(n · w)Indirect, through layersLong sequences with local structure
Dilated sliding windowO(n · w)Wider reach, gaps in coverageUpper layers of long encoders
Window plus global tokensO(n · (w + g))Through designated tokensDocument classification and QA
Interleaved local and global layersMixedDirect in global layersModern long-context decoders
Linear attentionO(n)Compressed approximationVery long streams
Sliding window attention compared with related attention patterns

Interleaving has become the dominant compromise. A model may use a local window in most layers and full attention in, for example, one layer out of every four to six, keeping most of the efficiency gain while preserving direct access to distant tokens.

Memory, the KV cache and inference behavior

During autoregressive generation, sliding window attention allows a rolling buffer KV cache. Keys and values older than the window are never read again, so the cache can be a fixed-size ring buffer of w entries per layer. This keeps inference memory constant regardless of how long generation continues, which is valuable on memory-limited accelerators.

Efficient implementations rely on block-sparse kernels. FlashAttention and similar kernels skip tiles of the score matrix that fall entirely outside the band, so the theoretical saving becomes a real speedup. Naive implementations that compute the full matrix and then mask it gain nothing in compute and only a little in clarity.

The next table summarizes practical effects on training and inference.

AspectEffectCondition
Training computeGrows linearly with lengthBlock-sparse kernel available
KV cache sizeFixed at window lengthRolling buffer implemented
ThroughputHigher for long promptsSequences much longer than the window
Short sequencesNo benefitLength below window size
Practical effects of sliding window attention

Limitations and common design mistakes

The main weakness of sliding window attention is lossy long-range dependency. Tasks such as multi-document question answering, code referencing distant definitions, or retrieval of a detail from the start of a long contract can fail when no layer has direct global access.

MistakeSymptomFix
Window too smallLoss of paragraph-level coherenceIncrease window or add global layers
Only local layersPoor recall of distant factsInterleave full attention layers
Dense masked kernelNo speed gainUse block-sparse attention kernels
Positional mismatchDegradation beyond training lengthAlign rotary embedding scaling with the window
Common mistakes when applying sliding window attention

A further subtlety concerns attention sinks: models tend to place large attention weight on the first tokens, and evicting them from a rolling cache can destabilize generation. Keeping a few initial tokens permanently cached is a widely used remedy.

Understanding it simply

Imagine reading a very long book while only being allowed to look at the last few pages at any moment. You can still follow the story, because each page builds on the previous ones, but you cannot instantly flip back to chapter one.

Sliding window attention works the same way. Each word looks only at its recent neighbors, which makes processing long texts much faster and cheaper. The cost is that distant details can fade, so designers often add occasional layers that see everything.

Applied in practice

An engineering team deploying a 7-billion-parameter language model for summarizing long support transcripts on a single 24 GB GPU faces a memory problem: full attention over 32,000-token conversations makes the KV cache too large to fit alongside the model weights.

They choose a model that uses sliding window attention with a 4,096-token window and a rolling cache. Memory use stays flat during generation, and throughput on long transcripts rises noticeably. Testing shows one weakness: questions about details from the first minutes of a call are answered less reliably. The team adds a short extractive step that copies key facts to the end of the prompt, and the final result is stable long-context summarization on modest hardware.