Mamba

Also known as: Mamba architecture

On this page 7

Mamba (also called the Mamba architecture) is a sequence model built on selective state space models, which process tokens through a compact recurrent state whose dynamics depend on the input. Introduced by Albert Gu and Tri Dao in late 2023, it is one of the most prominent alternatives to the Transformer among modern neural networks in artificial intelligence.

Its central promise is linear scaling with sequence length during training and constant memory per generated token during inference, while matching Transformers of similar size on many language modeling benchmarks. That trade-off makes Mamba attractive for long documents, genomics, audio and time series.

How Mamba processes a sequence

Mamba starts from the state space model formulation: a hidden state h evolves according to h’ = A h + B x, and the output is y = C h. The continuous system is discretized with a step size Δ, usually through a zero-order hold, which turns it into a recurrence that updates the state once per token.

Earlier structured state space models such as S4 used fixed, input-independent parameters, which allowed fast convolutional training but prevented the model from deciding what to remember. Mamba makes Δ, B and C functions of the current input. This selection mechanism lets the model reset its state, ignore filler tokens or retain a key fact, behavior that resembles gating in an LSTM but runs at much larger scale.

Input-dependent parameters break the convolution trick, so Mamba uses a hardware-aware parallel scan. The kernel keeps the expanded state in fast on-chip SRAM, fuses discretization, scan and output into one operation, and recomputes intermediate states in the backward pass instead of storing them.

The main parameters of the selective scan are summarized below.

ParameterRoleTypical setting
A (state matrix)Controls how fast state decaysDiagonal, learned, input-independent
Δ (step size)Decides how much of the current token enters the stateInput-dependent, per channel
B (input matrix)Writes the input into the stateInput-dependent
C (output matrix)Reads the state into the outputInput-dependent
N (state size)Memory capacity per channel16 in Mamba, 64 to 256 in Mamba-2
Core parameters of the selective state space layer

Inside the Mamba block

A Mamba block replaces both the attention layer and the feedforward layer of a Transformer block with one homogeneous unit. The input is normalized, usually with RMSNorm, and projected to about twice the model width. One branch passes through a short causal convolution, a SiLU activation and the selective scan; the other branch acts as a multiplicative gate. The gated result is projected back to model width and added through a residual connection.

The following table lists the stages of the block in order.

StageOperationPurpose
NormalizationRMSNorm or layer normalizationStabilizes training
Input projectionLinear layer with expansion factor 2Creates main and gate branches
Causal convolutionDepthwise 1D convolution, kernel about 4Mixes nearby tokens
Selective scanInput-dependent state space recurrenceModels long-range dependencies
GatingElementwise product with SiLU-activated branchControls information flow
Output projectionLinear layer back to model widthRestores residual stream size
Stages of a standard Mamba block

Mamba compared with Transformers and recurrent networks

Mamba sits between attention-based Transformers and classic recurrent networks. It trains in parallel like a Transformer but generates like an RNN, carrying a fixed-size state instead of a growing key-value cache.

ArchitectureTraining cost in sequence lengthInference memory per stepExact recall of distant tokens
TransformerQuadraticGrows with context (KV cache)Strong
Recurrent neural networkLinear, but sequentialConstantWeak
Linear attentionLinearConstantLimited
MambaLinear, parallel scanConstantModerate, improves with state size
Scaling behavior of sequence architectures

The fixed state is both the advantage and the limit. Because all history must be compressed into N values per channel, Mamba struggles more than attention on tasks that require verbatim copying or associative recall across long contexts, such as retrieving a phone number seen thousands of tokens earlier.

Mamba-2 and hybrid models

Mamba-2, published in 2024, introduced state space duality, a framework showing that a restricted selective SSM is equivalent to a form of masked linear attention. Restricting A to a scalar per head allows the computation to use matrix multiplications, which run efficiently on tensor cores, and permits much larger state sizes at similar speed.

Many production models now use hybrid stacks that interleave Mamba layers with a small share of attention layers. The attention layers restore precise recall, while the Mamba layers keep most of the stack cheap for long contexts. Published hybrids include Jamba, Zamba, Samba and Nemotron-H, typically with one attention layer for every several Mamba layers.

SituationRecommended designReason
Very long sequences such as DNA or audioPure MambaLinear cost dominates the budget
General chat and retrieval-heavy tasksHybrid Mamba and attentionAttention handles exact recall
Short contexts under a few thousand tokensTransformerMature tooling, little speed gain
Edge inference with tight memoryPure or mostly MambaNo growing KV cache
Choosing between pure and hybrid designs

Limitations and common misconceptions

  • Mamba is not an attention variant: it uses no query-key comparison, although Mamba-2 shows a formal link to linear attention.
  • Linear complexity does not guarantee faster wall-clock time at short lengths, where optimized attention kernels remain competitive.
  • Recall-intensive tasks such as in-context retrieval and copying remain a known weakness of pure SSMs.
  • Tooling is less mature: custom CUDA kernels limit portability to some accelerators and inference servers.
  • Benchmark parity is size-dependent: results reported at one scale do not automatically hold at larger or smaller scales.

A non-technical view

Imagine reading a long book in two ways. A Transformer keeps every page open on a huge table and glances back at all of them for each new word, which gets slow and crowded as the book grows. Mamba instead keeps a compact notebook of what matters, updating it page by page and deciding what to write down or erase.

The notebook stays the same size no matter how long the book is, so Mamba stays fast and light on very long texts. The cost is that some exact details may not make it into the notebook, which is why many systems mix Mamba with a few attention layers.

A concrete example

A genomics research group wants to classify regulatory regions in DNA sequences of up to one million base pairs. A Transformer at that length would need prohibitive memory for attention, so the team trains a Mamba-based model with about 100 million parameters that reads the sequence one nucleotide at a time.

Training fits on a small cluster of GPUs because cost grows linearly with sequence length, and inference runs with a constant memory footprint. Compared with a Transformer limited to 16,000-token windows, the Mamba model captures long-range interactions between distant regions, and the group reports better accuracy on enhancer prediction while keeping the experiment within its compute budget.