A state space model (SSM) is a sequence modeling layer that maps an input sequence to an output sequence through a fixed-size hidden state updated by linear recurrence. In neural networks and modern artificial intelligence, SSM layers serve as an alternative or complement to attention, because they process sequences in linear time with constant memory per step during generation.
The idea comes from control theory, where a continuous system is described by a state equation and an output equation. Deep learning research adapted it into trainable layers such as S4, H3 and Mamba, which made SSMs competitive with Transformers on long-sequence tasks in language, audio and genomics.
How a state space model processes a sequence
The continuous form defines a hidden state h(t) that evolves as h'(t) = A h(t) + B x(t), with output y(t) = C h(t) + D x(t). The matrix A governs memory dynamics, B controls how input enters the state, and C reads the state out. The term D acts as a skip connection from input to output.
Neural SSMs work on discrete tokens, so the continuous system is discretized with a step size usually called delta. Methods such as the zero-order hold or the bilinear transform turn A and B into discrete matrices, giving the recurrence h_t = Ā h_(t-1) + B̄ x_t. A larger step lets the state absorb the current input strongly; a smaller step preserves older context longer.
The table below lists the core components and what each one controls.
| Component | Typical shape | Role |
|---|---|---|
| State matrix A | N × N, often diagonal | Decay and memory of past inputs |
| Input matrix B | N × 1 per channel | Writes the input into the state |
| Output matrix C | 1 × N per channel | Reads the state into the output |
| Step size delta | Scalar or per token | Balances new input against memory |
| State size N | Commonly 16 to 256 | Capacity of the compressed history |
Recurrent and convolutional views of the same layer
A linear time-invariant SSM has a useful dual computation mode. Unrolling the recurrence shows that the output equals the input convolved with a kernel built from powers of Ā, so the whole sequence can be processed as one long convolution during training, often computed with the fast Fourier transform.
At inference the same layer runs as a recurrence, updating the state one token at a time with constant cost. This combination gives parallel training like a convolutional network and cheap autoregressive generation like a recurrent neural network, without the quadratic cost of full self-attention.
| Mode | Best used for | Cost per sequence | Limitation |
|---|---|---|---|
| Convolutional | Training on full sequences | About L log L with FFT | Requires time-invariant parameters |
| Recurrent | Token-by-token generation | Linear in L, constant state | Sequential, slow to train |
| Parallel scan | Training selective SSMs | Linear in L, parallel | Needs custom GPU kernels |
From S4 to selective models such as Mamba
Early neural SSMs struggled because naive recurrences forget quickly or explode. The S4 model solved this with a structured initialization of A derived from HiPPO theory, which keeps a compressed approximation of the input history, plus an efficient parameterization that made long kernels tractable.
A key weakness remained: time-invariant SSMs apply the same dynamics to every token, so they cannot decide what to keep based on content. Selective SSMs such as Mamba make B, C and delta functions of the current input. This input-dependent gating lets the model ignore filler and retain key tokens, at the price of losing the convolutional mode.
Mamba-2 later showed that selective SSMs are closely related to linear attention, a link described as structured state space duality. The table summarizes the main milestones.
| Model | Key idea | Main effect |
|---|---|---|
| S4 | HiPPO-based structured state matrix | Stable memory over very long inputs |
| S4D and DSS | Diagonal state matrices | Simpler, nearly equal accuracy |
| H3 | SSMs combined with multiplicative gates | Better recall in language tasks |
| Mamba | Input-dependent selection | Content-aware memory, linear scaling |
| Mamba-2 | State space duality with attention | Faster training with matrix units |
State space models compared with attention and RNNs
The SSM occupies a middle position between the classic recurrent neural network and the Transformer. Like an RNN it compresses history into a fixed state, but its linear recurrence trains in parallel and avoids much of the vanishing gradient problem.
| Property | Self-attention | LSTM or GRU | Selective SSM |
|---|---|---|---|
| Training cost | Quadratic in length | Linear, sequential | Linear, parallel |
| Generation memory | KV cache grows with length | Constant state | Constant state |
| Exact recall of distant tokens | Strong | Weak | Moderate |
| Very long contexts | Expensive | Unstable | Efficient |
The main trade-off is recall versus efficiency. Attention stores every past token and can copy any of them exactly, while an SSM must compress history into N numbers per channel. For this reason many production models are hybrids that interleave a few attention layers among many SSM layers.
Where SSM layers fit and where they fall short
SSMs are most attractive when sequences are long, continuous or streaming. Raw audio, genomic sequences, time series and long documents benefit from linear scaling, and edge devices benefit from the constant memory footprint during generation.
Practitioners should keep several known limitations in mind before replacing attention:
- Weaker associative recall: pure SSMs underperform attention on copying, retrieval and in-context lookup tasks.
- Fixed state capacity: information beyond what N dimensions can hold is lost, regardless of context length.
- Kernel dependence: selective scans need specialized GPU kernels, and tooling is less mature than for attention.
- Sensitive initialization: poor choices of A and delta ranges can make training unstable or memory short.
A sound default is to benchmark a hybrid against both pure designs on the actual task, measuring long-context retrieval separately from perplexity, because averages can hide recall failures.
The idea in brief
Think of reading a long novel while keeping a small notebook. Instead of rereading every earlier page, the reader updates a few notes after each paragraph and relies on them later. A state space model works the same way: it keeps a small running summary and updates it with each new word or sound sample.
This makes it fast and memory efficient on very long inputs. The cost is that a small notebook cannot hold every detail, so the model sometimes forgets exact facts that a method rereading everything would find.
A practical scenario
A research team builds a model that classifies heart rhythms from wearable ECG recordings sampled at 250 hertz. A ten-minute recording contains 150,000 time steps, far beyond what a standard Transformer can attend to on a single GPU without heavy downsampling.
The team uses a stack of selective SSM layers with a state size of 64. Training fits in memory on one 24-gigabyte card, and inference runs continuously on a phone because each new sample only updates the fixed state. Compared with a downsampled Transformer baseline, the model detects short arrhythmic episodes more reliably, since no samples had to be discarded. The team adds two attention layers later to improve detection of rare patterns that depend on precise timing across distant beats.