A cache, in processors, is a small, very fast memory built into the CPU that holds copies of recently or frequently used data and instructions, so the cores can reach them in a few nanoseconds instead of waiting for main memory. It is organized as a hierarchy of levels, usually L1, L2 and L3, each larger and slower than the one before.
Caches exist because processor speed has grown much faster than memory speed. A modern core can execute several instructions per clock cycle, but a trip to DRAM costs hundreds of cycles. Without caches, even a fast CPU would spend most of its time idle, which makes cache design one of the decisive factors in hardware performance.
How a CPU cache works
Caches exploit locality of reference. Temporal locality means that data used once is likely to be used again soon; spatial locality means that data near a recently used address is likely to be needed next. Programs naturally show both, because they run loops and walk through arrays.
Data moves between memory and cache in fixed blocks called cache lines, typically 64 bytes on x86-64 and most ARM64 designs, with some processors using 128 bytes. When the core requests an address, the cache checks its tags to see whether the line is present. A match is a cache hit; otherwise a cache miss forces the line to be fetched from the next level, and an existing line may be evicted to make room.
Each cache is set-associative: an address can be stored only in a small set of locations, commonly 8 to 16 ways. Higher associativity reduces conflicts but makes lookups slower and more power-hungry, so designers balance the two at every level.
Cache levels and typical parameters
The cache hierarchy trades size against speed. The values below are typical for current desktop and server processors; exact figures vary by architecture and generation.
| Level | Typical size | Typical latency | Shared by |
|---|---|---|---|
| L1 instruction and data | 32-64 KB each per core | About 3-5 cycles | One core |
| L2 | 256 KB-3 MB per core | About 10-20 cycles | One core or a small cluster |
| L3 (last-level cache) | 4-128 MB and more | About 30-70 cycles | All cores on a die or chiplet |
| Main memory (DRAM) | Gigabytes | About 60-120 ns, hundreds of cycles | Whole system |
Because latency grows so sharply, the L1 hit rate, often above 90 percent in typical workloads, has a large effect on performance. Some processors stack extra L3 directly on the die, which helps workloads such as games whose working set barely fits in a standard cache.
Design policies that shape cache behavior
Beyond size, a cache is defined by policies for writing, replacing and sharing data. These choices influence speed, power use and how cores see each other’s changes.
| Policy | Options | Effect |
|---|---|---|
| Write policy | Write-back or write-through | Write-back reduces memory traffic and is standard for data caches |
| Replacement policy | LRU approximations, random, adaptive | Decides which line to evict on a miss |
| Inclusion policy | Inclusive, exclusive or non-inclusive | Determines whether lower levels duplicate upper ones |
| Coherence protocol | MESI, MOESI and variants | Keeps copies consistent between cores |
| Prefetching | Stride, stream and pattern prefetchers | Loads lines before they are requested |
Cache coherence is essential in multi-core CPUs. When one core writes to a line, copies held by other cores must be invalidated or updated, and the hardware tracks each line’s state so that all cores see a consistent view of memory.
Cache compared with other processor and memory concepts
The word cache appears throughout computing, so it helps to separate the CPU cache from neighboring ideas. The table shows how it relates to cores, threads and other buffers.
| Concept | What it is | Relation to the cache |
|---|---|---|
| CPU core | An independent execution unit | Each core has private L1 and usually L2 |
| CPU thread | A hardware context sharing a core | Threads on one core compete for the same L1 and L2 |
| Clock speed | Cycles per second of the core | Higher clocks make memory misses relatively more costly |
| Registers | Tiny storage inside the core | Faster than L1, managed by the compiler |
| TLB | Cache of address translations | Speeds virtual-to-physical lookups for cache access |
| Software cache | Stored results in RAM or on disk | Same principle, managed by programs not hardware |
Why cache matters for performance and buyers
Software that respects the cache can run several times faster than code doing the same work with poor memory access patterns. Iterating through an array in order uses each 64-byte line fully, while jumping randomly through memory wastes most of each line and triggers constant misses.
When comparing processors, a larger L3 cache benefits games, databases, compilers and simulations with large working sets, while simple streaming workloads gain less. Cache size should be read together with core count, because a large shared L3 divided among many cores may offer little per-core capacity.
Common pitfalls and misconceptions
Cache behavior causes several performance problems that are invisible in source code and confusing even to experienced developers.
- False sharing: two threads write different variables that share one cache line, forcing constant coherence traffic.
- Cache thrashing: data mapped to the same set repeatedly evicts itself, often with power-of-two strides.
- Assuming more cache always helps: workloads that already fit or stream data see little gain.
- Ignoring the instruction cache: very large code paths can miss in L1i and slow down otherwise efficient programs.
- Side-channel exposure: timing differences between hits and misses enabled attacks such as Spectre.
In simple terms
Imagine a chef in a busy kitchen. The pantry holds everything, but walking there takes time. So the chef keeps the most used ingredients on the counter and a few more on a nearby shelf. A processor cache is that counter and shelf.
The processor keeps the data it needs most right next to where it works. When the data is already there, work continues instantly. When it is not, the processor must wait. The key takeaway is that a good cache keeps the processor busy instead of waiting.
A practical example
A developer at an analytics company profiles a reporting service that processes 50 million records. The original code stores each record as a separate object scattered across memory, and hardware counters show a last-level cache miss rate above 30 percent.
The developer restructures the data so that the fields used in the calculation sit in contiguous arrays. Each 64-byte cache line now holds sixteen useful 4-byte values instead of fragments of unrelated objects. Prefetchers recognize the sequential pattern, the miss rate falls sharply, and the report that took 40 seconds now finishes in about 12, with no change to the hardware.