A GPU (graphics processing unit) is a massively parallel processor designed to perform the same operation on very large numbers of data elements at once. In hardware, and in graphics specifically, the GPU turns geometry, textures and lighting into the pixels shown on a display, and the same architecture now also accelerates machine learning, scientific simulation and video processing.
Where a CPU devotes most of its transistors to making a few threads run fast, a GPU devotes them to thousands of simple arithmetic lanes and a very wide memory system. The result is far higher throughput on uniform work, at the cost of poor performance on branch-heavy, sequential code.
How a GPU processes work in parallel
A GPU is organized into many identical clusters, called streaming multiprocessors by NVIDIA and compute units by AMD. Each cluster contains dozens of arithmetic lanes, a register file, a small programmable cache and schedulers that keep the lanes busy.
GPUs execute threads in fixed groups, a model known as SIMT (single instruction, multiple threads). NVIDIA groups 32 threads into a warp, and AMD uses wavefronts of 32 or 64 threads. All threads in a group execute the same instruction on different data; when threads take different branches, the paths run one after another, a penalty called branch divergence.
GPUs hide memory latency through massive multithreading rather than large caches. While one warp waits hundreds of cycles for data, the scheduler switches instantly to another ready warp, so the arithmetic units rarely sit idle when enough threads are available.
| Stage | What happens | Executed by |
|---|---|---|
| Vertex shading | Transforms 3D vertices into screen positions | Programmable shader cores |
| Rasterization | Converts triangles into pixel fragments | Fixed-function hardware |
| Pixel shading | Computes color from textures and lighting | Programmable shader cores |
| Ray tracing | Tests rays against scene geometry for reflections and shadows | Dedicated ray tracing units |
| Output merging | Applies depth testing and blending, writes the frame | Render output units |
Key GPU specifications and what they mean
Peak arithmetic throughput equals lanes multiplied by two operations per fused multiply-add multiplied by clock speed. A GPU with 16,384 FP32 lanes at a 2.52 GHz boost clock reaches 16,384 x 2 x 2.52 GHz = about 82.6 TFLOPS of single-precision compute. Real applications rarely sustain this peak because they are often limited by memory bandwidth.
Memory bandwidth follows from bus width and data rate. A 384-bit bus with 21 Gbps GDDR6X memory delivers 384 x 21 / 8 = 1,008 GB/s, roughly ten times the bandwidth of a dual-channel DDR5 desktop memory system.
| Parameter | Typical range | What it affects |
|---|---|---|
| Shader lanes | A few hundred to over 16,000 | Raw parallel throughput |
| Boost clock | About 1.5 to 3 GHz | Speed of each lane |
| VRAM capacity | 8 to 32 GB consumer, 80 to 288 GB data center | Largest model, texture set or dataset that fits |
| Memory bandwidth | About 250 GB/s to several TB/s | Speed of data-hungry workloads |
| Board power | 75 to 600 W consumer, 700 W or more data center | Cooling and power supply needs |
| Matrix units | Present on most current GPUs | AI training and inference speed |
Most GPUs now include specialized matrix units, such as Tensor Cores, that multiply small matrices at reduced precision (FP16, BF16, FP8 or INT8). For neural networks these units deliver many times the throughput of the general shader lanes.
Types of GPUs and how they compare
GPUs differ mainly in where they sit and which memory they use. An integrated GPU shares the CPU package and system memory, while a discrete GPU has its own board, power delivery and dedicated VRAM.
| Type | Memory | Typical use | Main constraint |
|---|---|---|---|
| Integrated GPU | Shared system RAM | Office work, video playback, light gaming | Shared bandwidth and power budget |
| Discrete consumer GPU | GDDR6, GDDR6X or GDDR7 | Gaming, content creation, local AI | VRAM capacity |
| Workstation GPU | GDDR with ECC | CAD, visualization, certified software | Price per performance |
| Data center accelerator | HBM stacks | AI training, inference, HPC | Power, cooling and supply |
Compared with a CPU, a GPU offers far higher throughput but higher latency per task. The CPU still runs the operating system, prepares work and launches GPU kernels, so the two act as host and accelerator rather than as replacements for each other.
Programming and using a GPU
Graphics applications reach the GPU through APIs such as Direct3D 12, Vulkan and Metal, which submit command lists that the driver translates for the hardware. General-purpose computing uses platforms such as CUDA, ROCm (HIP), OpenCL or SYCL, and most AI developers use them indirectly through frameworks that ship optimized GPU libraries.
A workload suits a GPU when it has abundant data parallelism, regular memory access and high arithmetic intensity. Operations on images, matrices, particles and video frames fit well; parsing, transactional databases and code with unpredictable branches usually do not.
Limitations and common mistakes
The most frequent GPU bottleneck in practice is memory, not compute. A model or dataset that does not fit in VRAM either fails or spills over the PCIe link, which at about 32 GB/s per direction for PCIe 4.0 x16 is far slower than on-board memory.
| Symptom | Likely cause | Remedy |
|---|---|---|
| Out-of-memory errors | Model or batch exceeds VRAM | Smaller batches, lower precision or a larger-memory GPU |
| Low GPU utilization | CPU or data loading cannot keep up | Faster input pipeline, more CPU threads |
| Poor speedup over CPU | Small problem size or heavy branching | Batch work or keep the task on the CPU |
| Stuttering in games | Shader compilation or VRAM exhaustion | Lower texture settings, updated drivers |
| Falling clocks under load | Thermal or power limit reached | Better airflow or a lower power target |
A plain-language summary
A GPU is a chip built to do a huge number of simple calculations at the same time. A single video frame contains millions of pixels. Each pixel can be colored on its own. Handling them all at once is much faster than going one by one.
Picture a CPU as a few expert craftspeople. A GPU is more like a stadium full of workers who all follow the same short instruction. For repetitive jobs such as images, video and AI, the stadium wins easily.
An example from practice
A research group fine-tunes a language model with 7 billion parameters for medical text classification. On a 32-core server CPU, one training epoch over its dataset takes about two days, which makes experimentation impractical.
The group moves training to a data center GPU with 80 GB of HBM memory and uses BF16 mixed precision with a parameter-efficient fine-tuning method, so the model weights fit in VRAM. The matrix units handle the bulk of the work, and one epoch now finishes in roughly two hours.
The team learns that the speedup depends on keeping the GPU fed: early runs reached only 40% utilization until data loading was moved to multiple CPU worker processes.