HBM

Also known as: High Bandwidth Memory

On this page 7

HBM (High Bandwidth Memory) is a type of 3D-stacked DRAM in which several memory dies are piled vertically, connected by through-silicon vias (TSVs) and placed next to a processor on a shared silicon interposer. In computer memory and hardware design, HBM is the standard choice when a GPU, AI accelerator or high-end processor needs terabytes per second of bandwidth within a tight power budget.

HBM is standardized by JEDEC in successive generations, from the original HBM through HBM2, HBM2E, HBM3, HBM3E and HBM4. Its defining feature is an extremely wide interface at moderate clock speeds, the opposite of the narrow and very fast approach used by GDDR graphics memory.

How HBM stacks and connects memory dies

An HBM device, called a stack, consists of several DRAM dies on top of an optional base logic die. Vertical TSVs pass through the silicon of each die and carry data, address and power signals, while microbumps join neighboring dies. Typical stacks contain 4, 8, 12 or 16 DRAM layers, a configuration written as 4-Hi, 8-Hi and so on.

Each stack exposes a 1,024-bit interface in generations up to HBM3E, and HBM4 doubles this to 2,048 bits. Such a wide bus cannot be routed through an ordinary circuit board, so the stack and the processor sit side by side on a silicon interposer or an embedded bridge that provides thousands of fine-pitch connections. This arrangement is called 2.5D packaging.

The interface is divided into independent channels and pseudo-channels, which lets the memory controller serve many requests in parallel and keeps the wide bus busy.

HBM generations and their key parameters

Each HBM generation raised the per-pin data rate, stack height and capacity, which multiplies directly into bandwidth per stack. The table lists representative values from the JEDEC specifications and common products; exact figures vary between vendors and speed bins.

GenerationInterface widthData rate per pinBandwidth per stackTypical stack capacity
HBM1,024 bitsAbout 1 Gb/sAbout 128 GB/s1 GB
HBM21,024 bitsUp to 2.4 Gb/sUp to about 307 GB/s4 to 8 GB
HBM2E1,024 bitsUp to about 3.6 Gb/sAbout 460 GB/s8 to 16 GB
HBM31,024 bitsUp to 6.4 Gb/sAbout 819 GB/s16 to 24 GB
HBM3E1,024 bitsAbout 8 to 9.8 Gb/sAbout 1 to 1.2 TB/s24 to 36 GB
HBM42,048 bitsAbout 6.4 to 8 Gb/sAbout 1.6 to 2 TB/sUp to 48 GB or more
Representative parameters of HBM generations

Processors combine several stacks to reach their total figure. An AI accelerator with eight HBM3E stacks can exceed 8 TB/s of aggregate bandwidth and nearly 200 GB or more of capacity, which is why HBM capacity has become a limiting factor for large language models.

HBM compared with GDDR and DDR memory

HBM trades cost and capacity flexibility for bandwidth and efficiency. GDDR memory, used on most consumer graphics cards, runs a narrow 32-bit interface per chip at very high data rates, while DDR5 modules serve CPUs with large, socketed and upgradeable capacity.

PropertyHBMGDDR7DDR5
Interface per device1,024 or 2,048 bits per stack32 bits per chip64 bits per DIMM
Data rate per pinModerateVery highModerate
PlacementSame package as processorCircuit board near GPUSocketed modules
Energy per bitLowestHigherModerate
Cost and complexityHighestMediumLowest
UpgradeabilityNoneNoneUser replaceable
HBM, GDDR and DDR memory compared

The short on-package connections are the source of HBM efficiency: driving signals a few millimeters across an interposer costs far less energy than driving them across a circuit board. HBM also saves board area, because the memory no longer surrounds the processor as a ring of separate chips.

Where HBM is used and why

HBM is used wherever memory bandwidth limits performance more than raw compute. Its main markets today are AI training and inference accelerators, data center GPUs, high-performance computing and high-end network processors.

  • AI accelerators: large models must stream billions of weights per generated token, so bandwidth sets inference speed.
  • Scientific computing: simulations with low arithmetic intensity depend on fast memory access.
  • Some server CPUs and FPGAs: integrated HBM serves as a fast memory tier or cache.
  • Earlier consumer graphics cards: a few gaming GPUs used HBM, but cost moved the consumer market back to GDDR.

Demand from AI has made HBM one of the most supply-constrained components in the semiconductor industry. Only a few manufacturers produce it, and advanced packaging capacity for interposers is a further bottleneck.

Limitations and engineering challenges of HBM

HBM brings manufacturing and thermal challenges that conventional memory avoids. Stacking many thin dies lowers yield, and a defect found after assembly can scrap both the memory and the expensive processor sharing its package.

The table lists the main limitations, their causes and how designers mitigate them.

LimitationCauseTypical mitigation
High costTSVs, stacking and interposer packagingReserve for bandwidth-critical products
Fixed capacityMemory sealed in the packageAdd more stacks or a DDR tier
Heat in the stackUpper dies insulate lower diesThinner dies, improved bonding, strong cooling
Yield riskMany dies and bonds per stackKnown-good-die testing before assembly
Latency not reducedSame DRAM cell physicsLarge on-chip caches
Common HBM limitations and mitigations

A common misconception is that HBM is faster in latency than other DRAM. Its advantage is bandwidth and energy per bit; access latency remains in the same range as other DRAM types because the memory cells work the same way.

Explained without jargon

HBM is a way of building very fast memory for powerful chips. Instead of spreading memory chips around a circuit board, manufacturers stack them like floors of a building and place the stack right next to the processor.

Picture a highway with a thousand lanes instead of a few very fast lanes. Each lane moves at an ordinary speed, but so many run side by side that a huge amount of data flows at once. The short distance also means less energy is wasted. The downside is that HBM is expensive, hard to make and cannot be upgraded later.

A typical use case

A cloud provider deploys AI accelerators to serve a large language model with about 70 billion parameters. Stored at 16-bit precision, the weights alone take roughly 140 GB, and generating each token requires reading most of them from memory. Each accelerator carries eight HBM3E stacks with a combined capacity above 140 GB and aggregate bandwidth of several terabytes per second.

Because the whole model fits on one device, the provider avoids splitting it across several cards, and the high bandwidth lets the accelerator produce tokens quickly for many users at once. Engineers monitoring the system find that throughput rises almost in step with memory bandwidth, confirming that the workload is memory-bound rather than compute-bound. The outcome is lower response time and lower energy per request than an equivalent setup built from GDDR-based cards.