CUDA is NVIDIA’s parallel computing platform and programming model that lets developers run general-purpose programs on NVIDIA graphics processors. Within graphics hardware, it turns the GPU from a fixed rendering engine into a massively parallel processor for tasks such as machine learning, scientific simulation, video processing and data analytics.
The name originally stood for Compute Unified Device Architecture, but today NVIDIA CUDA refers to a whole stack: a hardware execution model, extensions to C and C++, a compiler (nvcc), a runtime and driver API, and optimized libraries such as cuBLAS and cuDNN. CUDA runs only on NVIDIA GPUs, a fact that shapes both its strengths and its main drawback.
How the CUDA programming model works
CUDA follows a host and device model. The CPU (host) prepares data, copies it to GPU memory (device), and launches a kernel, a function that thousands of threads execute in parallel. Each thread works on a different element, such as one pixel or one matrix entry, identified by built-in indices like threadIdx and blockIdx.
Threads are organized in a hierarchy that matches the hardware. Groups of 32 threads form a warp, which executes one instruction at a time in lockstep on a streaming multiprocessor (SM). Warps are grouped into thread blocks of up to 1,024 threads, and blocks form a grid. Threads in one block can share fast on-chip memory and synchronize, while different blocks run independently, so the same program scales across GPUs with different numbers of SMs.
The table below maps the CUDA thread hierarchy to the hardware that executes it.
| Level | Typical size | Hardware mapping | Cooperation available |
|---|---|---|---|
| Thread | 1 | One lane of a CUDA core | Private registers |
| Warp | 32 threads | Scheduled together on an SM | Warp shuffle and vote instructions |
| Thread block | Up to 1,024 threads | Resident on one SM | Shared memory, barriers |
| Grid | Thousands to millions of threads | Spread across all SMs | Global memory, atomics |
Memory spaces and performance factors
CUDA performance depends more on memory access patterns than on raw arithmetic. GPUs offer very high peak FLOPS, but a kernel that reads memory inefficiently will wait on data most of the time. Understanding the memory spaces is therefore the core skill of CUDA optimization.
| Memory space | Scope | Relative speed | Typical use |
|---|---|---|---|
| Registers | One thread | Fastest | Local variables |
| Shared memory | One thread block | Very fast, on-chip | Tiles of data reused by a block |
| L1 and L2 cache | SM and whole GPU | Fast | Automatic caching of global reads |
| Global memory (VRAM) | All threads and host copies | High bandwidth, high latency | Input and output arrays |
| Host memory | CPU side | Slowest via PCIe | Data before transfer |
Three factors dominate tuning. Coalesced access means that threads of a warp read consecutive addresses so the hardware combines them into few transactions. Occupancy describes how many warps are active per SM, which helps hide memory latency. Warp divergence occurs when threads in one warp take different branches, forcing the hardware to execute both paths one after another. Reducing host-to-device transfers over PCIe is equally important, because copying data can easily cost more time than the computation itself.
The CUDA software stack and compute capability
Most CUDA work does not involve writing kernels by hand. The CUDA Toolkit ships with libraries that frameworks such as PyTorch and TensorFlow call internally, so many users benefit from CUDA without seeing it. Each GPU generation is identified by a compute capability number (for example 8.6 or 9.0), which tells the compiler which instructions and features the hardware supports.
| Component | Purpose | Typical user |
|---|---|---|
| nvcc compiler | Compiles CUDA C++ to PTX and machine code | Kernel developers |
| CUDA runtime and driver API | Memory management, kernel launch, streams | Application developers |
| cuBLAS, cuFFT, cuSPARSE | Linear algebra and signal processing | Scientific and engineering software |
| cuDNN and TensorRT | Neural network training and inference | Machine learning frameworks |
| Nsight tools | Profiling and debugging | Performance engineers |
Compatibility has two parts. The installed NVIDIA driver must support the CUDA version an application was built with, and the compiled code must include either machine code for the GPU’s compute capability or PTX intermediate code that the driver can compile just in time. Mismatches between driver, toolkit and framework versions are the most common source of installation problems.
CUDA compared with other GPU computing options
CUDA competes with several vendor-neutral or rival GPU programming interfaces. Its advantage is ecosystem maturity: libraries, tools, documentation and framework support that have grown since 2007. Its main limitation is vendor lock-in to NVIDIA hardware.
| Platform | Hardware support | Main strength | Main limitation |
|---|---|---|---|
| CUDA | NVIDIA GPUs only | Most mature libraries and tooling | Vendor lock-in |
| ROCm and HIP | AMD GPUs, HIP can target NVIDIA | CUDA-like syntax, open source | Smaller ecosystem |
| OpenCL | Many vendors and device types | Broad portability | Uneven vendor support and tooling |
| SYCL and oneAPI | Intel, NVIDIA and AMD backends | Single-source modern C++ | Less widespread adoption |
| Vulkan compute and Metal | Graphics API compute shaders | Integration with rendering | Lower-level, fewer math libraries |
CUDA is also distinct from CUDA cores, the marketing name for the scalar arithmetic units inside an SM. A GPU with more CUDA cores is not automatically faster for CUDA programs, because Tensor cores, memory bandwidth and clock speed often decide real performance.
Limitations and common mistakes
- Porting serial algorithms directly: code with heavy branching or data dependencies may run slower on a GPU than on a CPU.
- Ignoring transfer costs: small workloads often spend more time on PCIe copies than on computation.
- Running out of VRAM: large models or datasets that exceed GPU memory fail or require batching and memory-saving techniques.
- Version mismatches: incompatible driver, toolkit and framework builds cause errors such as missing kernel images.
- Assuming portability: CUDA code does not run on AMD, Intel or Apple GPUs without translation layers or a rewrite.
Put simply
A normal processor is like a few very skilled chefs who can cook anything but only a few dishes at a time. A graphics card is like a huge kitchen with thousands of helpers who each do one simple step very quickly. CUDA is the set of instructions and tools that lets programmers organize those thousands of helpers for jobs other than drawing images.
That is why the same graphics card that runs games can also train AI models or simulate weather. The catch is that CUDA only works with NVIDIA graphics cards.
In practice
A medical imaging lab reconstructs 3D scans from thousands of 2D X-ray projections. On a multi-core CPU, the reconstruction of one high-resolution volume takes several minutes, which slows down research that requires hundreds of scans per week.
An engineer rewrites the back-projection step as a CUDA kernel in which each thread computes one voxel and nearby threads read neighboring projection data so that memory access is coalesced. Projection data is copied to the GPU once, and intermediate results stay in VRAM. Profiling with Nsight shows that the kernel is limited by memory bandwidth, so the engineer adds shared-memory tiling. The final version reconstructs a volume in a few seconds, and the team can process an entire day of scans in under an hour.