A TPU (Tensor Processing Unit) is an application-specific integrated circuit designed by Google to accelerate the matrix and tensor operations of neural networks, both for training and for inference. Instead of executing general-purpose instructions like a CPU, a TPU dedicates most of its silicon to a large systolic array of multiply-accumulate units fed by high-bandwidth memory.
Among processors in computer hardware, the TPU belongs to the family of AI accelerators, alongside GPUs and NPUs. It is mainly offered as a cloud service in large interconnected clusters called pods, while small Edge TPU variants run inference in embedded devices.
How a TPU computes
The heart of a TPU is the matrix multiply unit (MXU), a two-dimensional systolic array. Weights are loaded into a grid of multiply-accumulate cells, and input activations flow through the grid in a rhythmic wave; each cell multiplies, adds its partial sum and passes the result to its neighbor. Data is reused many times without returning to memory, which is the main reason for the efficiency.
The first-generation TPU used a 256 by 256 array, or 65,536 multiply-accumulate units working on 8-bit integers. Later generations use several smaller arrays per core, typically 128 by 128, together with vector units for activation functions and normalization and a scalar unit for control flow.
TPUs popularized the bfloat16 number format, which keeps the 8-bit exponent of 32-bit floats but shortens the mantissa. This preserves the numeric range needed for training while halving memory and bandwidth. Newer chips also support INT8 and FP8 for faster inference.
Programs reach the hardware through the XLA compiler, which takes graphs from frameworks such as JAX, TensorFlow or PyTorch and fuses operations into large, statically shaped kernels. This compiler-centric design removes much of the scheduling logic a GPU performs in hardware, but it also means that dynamic shapes and unusual operations can run slowly.
TPU generations
Each TPU generation has increased compute, memory capacity and interconnect scale. Figures below are approximate peak values per chip as publicly described and depend on numeric precision.
| Generation | Introduced | Main use | Notable characteristics |
|---|---|---|---|
| TPU v1 | 2015 (internal) | Inference only | INT8, about 92 TOPS, DDR3 memory |
| TPU v2 | 2017 | Training and inference | bfloat16, HBM, first pods |
| TPU v3 | 2018 | Training | Liquid cooling, about 123 TFLOPS |
| TPU v4 | 2021 | Large-scale training | About 275 TFLOPS, optical circuit switches, 4,096-chip pods |
| TPU v5e and v5p | 2023 | Cost-efficient serving, large training | v5p pods with nearly 9,000 chips |
| Trillium (v6e) | 2024 | Training and serving | Larger MXUs, higher energy efficiency |
| Ironwood | 2025 | Inference at scale | FP8 support, large HBM capacity per chip |
Pods connect chips through a dedicated inter-chip interconnect (ICI) in torus topologies, so thousands of TPUs can work on one model as if they shared a fabric. This scale-out design matters as much as the speed of an individual chip.
Memory has grown in importance with every generation. Large language models are often limited by memory bandwidth rather than arithmetic, so recent TPUs pair their compute units with several stacks of high-bandwidth memory (HBM) delivering terabytes per second. Capacity per chip determines how many chips are needed just to hold a model’s weights and activations.
TPU compared with CPU, GPU and NPU
All four processor types can run neural networks, but they make different trade-offs between flexibility and efficiency.
| Processor | Design focus | Strengths | Limitations |
|---|---|---|---|
| CPU | General-purpose, low latency | Any workload, rich software | Low throughput for large matrix math |
| GPU | Massively parallel programmable cores | Flexible, broad ecosystem, available everywhere | Higher power per operation for dense AI math |
| TPU | Dense tensor math at datacenter scale | High efficiency, huge pods, fast interconnect | Mainly cloud-only, compiler constraints |
| NPU | Low-power on-device inference | Very efficient for small models | Limited memory and training ability |
Modern GPUs with tensor cores narrow the gap, because they also contain dedicated matrix units. The practical distinction is therefore less about raw speed and more about availability, software ecosystem and cost per training run.
When a TPU is the right choice
TPUs fit workloads with large, regular matrix operations and static shapes, such as transformer training, recommendation models and high-volume inference. They are less suitable for small experiments that rely on custom CUDA kernels or highly dynamic code.
| Situation | Recommended choice | Reason |
|---|---|---|
| Training a large transformer in the cloud | TPU pod or GPU cluster | Both scale; compare price per step |
| Code depends on custom CUDA kernels | GPU | TPUs do not run CUDA |
| JAX-based research at scale | TPU | JAX and XLA are first-class on TPUs |
| On-device inference in a camera | Edge TPU or NPU | Milliwatt-to-watt power budget |
| On-premises deployment required | GPU | Datacenter TPUs are not sold as cards |
Cost comparisons should use price per completed training run or per million served tokens, not price per chip-hour. A cheaper accelerator that reaches low utilization because of padding, recompilation or input bottlenecks can end up more expensive in total than a faster one that stays busy.
Limitations and misconceptions
The TPU is a specialized instrument, and most problems arise when it is treated as a drop-in replacement for other processors.
- A TPU is not a faster CPU: it cannot run an operating system or general applications.
- Peak TFLOPS rarely equal real speed; utilization depends on batch size, padding and memory bandwidth.
- Vendor dependence: datacenter TPUs are available mainly through one cloud provider.
- Edge TPU is a different product, delivering about 4 TOPS at roughly 2 W, far below datacenter chips.
- Recompilation costs: changing tensor shapes can trigger slow recompilation in XLA.
In simple terms
A TPU is a chip built for one job: the math behind artificial intelligence. A normal processor is like a skilled worker who can do any task but only a few at a time. A TPU is like a huge factory assembly line where thousands of small stations each do one multiplication and pass the result along.
Because it does only this kind of work, it finishes it much faster and with less electricity. Companies rent thousands of these chips at once to teach large AI models.
A practical example
A research team wants to train a language model with about one billion parameters. On a small GPU workstation the estimate is several weeks. The team rewrites its training loop in JAX, keeps all tensor shapes fixed and rents a TPU v5e slice of 64 chips for the job.
After an initial XLA compilation of a few minutes, the model trains at a steady pace and finishes in roughly two days. The team discovers that a variable sequence length caused repeated recompilation, so it pads sequences into a few fixed buckets, which raises chip utilization noticeably. For deployment, the same model is later served on smaller inference instances to keep running costs low.