HBM (High Bandwidth Memory) is a type of 3D-stacked DRAM in which several memory dies are piled vertically, connected by through-silicon vias (TSVs) and placed next to a processor on a shared silicon interposer. In computer memory and hardware design, HBM is the standard choice when a GPU, AI accelerator or high-end processor needs terabytes per second of bandwidth within a tight power budget.
HBM is standardized by JEDEC in successive generations, from the original HBM through HBM2, HBM2E, HBM3, HBM3E and HBM4. Its defining feature is an extremely wide interface at moderate clock speeds, the opposite of the narrow and very fast approach used by GDDR graphics memory.
How HBM stacks and connects memory dies
An HBM device, called a stack, consists of several DRAM dies on top of an optional base logic die. Vertical TSVs pass through the silicon of each die and carry data, address and power signals, while microbumps join neighboring dies. Typical stacks contain 4, 8, 12 or 16 DRAM layers, a configuration written as 4-Hi, 8-Hi and so on.
Each stack exposes a 1,024-bit interface in generations up to HBM3E, and HBM4 doubles this to 2,048 bits. Such a wide bus cannot be routed through an ordinary circuit board, so the stack and the processor sit side by side on a silicon interposer or an embedded bridge that provides thousands of fine-pitch connections. This arrangement is called 2.5D packaging.
The interface is divided into independent channels and pseudo-channels, which lets the memory controller serve many requests in parallel and keeps the wide bus busy.
HBM generations and their key parameters
Each HBM generation raised the per-pin data rate, stack height and capacity, which multiplies directly into bandwidth per stack. The table lists representative values from the JEDEC specifications and common products; exact figures vary between vendors and speed bins.
| Generation | Interface width | Data rate per pin | Bandwidth per stack | Typical stack capacity |
|---|---|---|---|---|
| HBM | 1,024 bits | About 1 Gb/s | About 128 GB/s | 1 GB |
| HBM2 | 1,024 bits | Up to 2.4 Gb/s | Up to about 307 GB/s | 4 to 8 GB |
| HBM2E | 1,024 bits | Up to about 3.6 Gb/s | About 460 GB/s | 8 to 16 GB |
| HBM3 | 1,024 bits | Up to 6.4 Gb/s | About 819 GB/s | 16 to 24 GB |
| HBM3E | 1,024 bits | About 8 to 9.8 Gb/s | About 1 to 1.2 TB/s | 24 to 36 GB |
| HBM4 | 2,048 bits | About 6.4 to 8 Gb/s | About 1.6 to 2 TB/s | Up to 48 GB or more |
Processors combine several stacks to reach their total figure. An AI accelerator with eight HBM3E stacks can exceed 8 TB/s of aggregate bandwidth and nearly 200 GB or more of capacity, which is why HBM capacity has become a limiting factor for large language models.
HBM compared with GDDR and DDR memory
HBM trades cost and capacity flexibility for bandwidth and efficiency. GDDR memory, used on most consumer graphics cards, runs a narrow 32-bit interface per chip at very high data rates, while DDR5 modules serve CPUs with large, socketed and upgradeable capacity.
| Property | HBM | GDDR7 | DDR5 |
|---|---|---|---|
| Interface per device | 1,024 or 2,048 bits per stack | 32 bits per chip | 64 bits per DIMM |
| Data rate per pin | Moderate | Very high | Moderate |
| Placement | Same package as processor | Circuit board near GPU | Socketed modules |
| Energy per bit | Lowest | Higher | Moderate |
| Cost and complexity | Highest | Medium | Lowest |
| Upgradeability | None | None | User replaceable |
The short on-package connections are the source of HBM efficiency: driving signals a few millimeters across an interposer costs far less energy than driving them across a circuit board. HBM also saves board area, because the memory no longer surrounds the processor as a ring of separate chips.
Where HBM is used and why
HBM is used wherever memory bandwidth limits performance more than raw compute. Its main markets today are AI training and inference accelerators, data center GPUs, high-performance computing and high-end network processors.
- AI accelerators: large models must stream billions of weights per generated token, so bandwidth sets inference speed.
- Scientific computing: simulations with low arithmetic intensity depend on fast memory access.
- Some server CPUs and FPGAs: integrated HBM serves as a fast memory tier or cache.
- Earlier consumer graphics cards: a few gaming GPUs used HBM, but cost moved the consumer market back to GDDR.
Demand from AI has made HBM one of the most supply-constrained components in the semiconductor industry. Only a few manufacturers produce it, and advanced packaging capacity for interposers is a further bottleneck.
Limitations and engineering challenges of HBM
HBM brings manufacturing and thermal challenges that conventional memory avoids. Stacking many thin dies lowers yield, and a defect found after assembly can scrap both the memory and the expensive processor sharing its package.
The table lists the main limitations, their causes and how designers mitigate them.
| Limitation | Cause | Typical mitigation |
|---|---|---|
| High cost | TSVs, stacking and interposer packaging | Reserve for bandwidth-critical products |
| Fixed capacity | Memory sealed in the package | Add more stacks or a DDR tier |
| Heat in the stack | Upper dies insulate lower dies | Thinner dies, improved bonding, strong cooling |
| Yield risk | Many dies and bonds per stack | Known-good-die testing before assembly |
| Latency not reduced | Same DRAM cell physics | Large on-chip caches |
A common misconception is that HBM is faster in latency than other DRAM. Its advantage is bandwidth and energy per bit; access latency remains in the same range as other DRAM types because the memory cells work the same way.
Explained without jargon
HBM is a way of building very fast memory for powerful chips. Instead of spreading memory chips around a circuit board, manufacturers stack them like floors of a building and place the stack right next to the processor.
Picture a highway with a thousand lanes instead of a few very fast lanes. Each lane moves at an ordinary speed, but so many run side by side that a huge amount of data flows at once. The short distance also means less energy is wasted. The downside is that HBM is expensive, hard to make and cannot be upgraded later.
A typical use case
A cloud provider deploys AI accelerators to serve a large language model with about 70 billion parameters. Stored at 16-bit precision, the weights alone take roughly 140 GB, and generating each token requires reading most of them from memory. Each accelerator carries eight HBM3E stacks with a combined capacity above 140 GB and aggregate bandwidth of several terabytes per second.
Because the whole model fits on one device, the provider avoids splitting it across several cards, and the high bandwidth lets the accelerator produce tokens quickly for many users at once. Engineers monitoring the system find that throughput rises almost in step with memory bandwidth, confirming that the workload is memory-bound rather than compute-bound. The outcome is lower response time and lower energy per request than an equivalent setup built from GDDR-based cards.