PyTorch

On this page 7

PyTorch is an open-source deep learning framework for Python that provides GPU-accelerated tensors, automatic differentiation and a library of neural network building blocks for training and running models. Among AI frameworks & tools, it is the dominant choice in machine learning research and a standard foundation for large language models, computer vision and generative systems in Artificial intelligence.

Originally developed at Meta AI and released in 2016, PyTorch is now governed by the PyTorch Foundation under the Linux Foundation. Its defining trait is eager execution: operations run immediately, line by line, so a model is ordinary Python code that can be inspected, debugged and modified like any other program.

How PyTorch executes and differentiates code

PyTorch works by recording tensor operations as they execute and replaying them backward to compute gradients. The core data structure is the tensor, an n-dimensional array that lives on a CPU, an NVIDIA or AMD GPU, or Apple silicon, and carries a data type such as float32, bfloat16 or int8.

When a tensor has requires_grad=True, the autograd engine builds a dynamic computational graph during the forward pass. Calling loss.backward() walks that graph in reverse and applies the chain rule, storing gradients in each parameter’s .grad field. Because the graph is rebuilt on every iteration, control flow can change per input: loops, conditionals and variable sequence lengths need no special syntax.

The main components that a typical training program touches are summarized below.

ComponentRoleTypical use
torch.TensorMultidimensional array with device and dtypeInputs, weights, activations
torch.autogradReverse-mode automatic differentiationComputing gradients of a loss
torch.nnLayers, losses and the Module base classDefining model architecture
torch.optimOptimizers such as SGD, Adam and AdamWUpdating parameters each step
torch.utils.dataDataset and DataLoader abstractionsBatching, shuffling, parallel loading
torch.compileGraph capture and kernel generationSpeeding up training and inference
Core PyTorch components and their roles

From eager mode to compiled graphs

PyTorch offers several execution modes that trade flexibility for speed and portability. Eager mode is the default and is easiest to debug, but each operation launches separately, which leaves Python overhead and memory traffic on the table.

Since PyTorch 2.0, torch.compile captures Python bytecode with TorchDynamo, lowers it through AOTAutograd and generates fused kernels with TorchInductor, often in the Triton language for GPUs. Reported speedups commonly fall between about 1.3x and 2x for training, but the gain depends heavily on model shape, batch size and how often graph breaks force a fallback to eager code.

The following table compares the execution and export paths a team can choose from.

ModeStrengthLimitation
Eager executionImmediate results, simple debuggingHighest per-operation overhead
torch.compileFused kernels with minimal code changesCompile time, graph breaks, recompilation
TorchScriptPython-free serialized modelsLegacy path, restricted Python subset
torch.exportClean full graph for deploymentRequires traceable, static-friendly code
ONNX exportPortability to other runtimesOperator coverage gaps for new layers
PyTorch execution and export options

PyTorch compared with TensorFlow and JAX

PyTorch differs from its main rivals mostly in programming model and ecosystem rather than in raw capability. TensorFlow began with static graphs and remains strong in established production pipelines and mobile tooling, while JAX uses functional transformations such as jit, grad and vmap over pure functions and is popular for large-scale research on TPUs.

In practice, the research share of PyTorch is very high: most new model code, reference implementations and checkpoints on public model hubs target it first. Higher-level libraries such as PyTorch Lightning, fastai and Hugging Face Transformers build directly on PyTorch, and Keras 3 can use it as a backend.

AspectPyTorchTensorFlowJAX
Programming styleObject-oriented, imperativeKeras layers, graph functionsFunctional, pure functions
Default executionEagerEager with tf.function graphsTraced and compiled via XLA
Strongest hardwareNVIDIA GPUsGPUs and TPUsTPUs and GPUs
Research adoptionDominantDecliningStrong in large labs
Edge deploymentExecuTorchLiteRTThrough export to other formats
Framework comparison at a glance

Scaling PyTorch training across devices

PyTorch scales beyond one GPU through the torch.distributed package, which uses collective communication backends such as NCCL for NVIDIA hardware and Gloo for CPUs. The right strategy depends on whether the model fits in one device’s memory.

StrategyWhat is splitWhen to use it
DistributedDataParallelData batches; full model per GPUModel fits on one GPU
Fully Sharded Data ParallelParameters, gradients, optimizer stateModel too large for one GPU
Tensor parallelismIndividual weight matricesVery wide layers in large transformers
Pipeline parallelismGroups of layers across devicesVery deep models across many nodes
Parallelism strategies in PyTorch

Memory savings also come from mixed precision with torch.autocast in bfloat16 or float16, which roughly halves activation memory, and from activation checkpointing, which recomputes intermediate values instead of storing them. Libraries such as DeepSpeed and Ray add further orchestration on top of these primitives.

Limitations and common mistakes

PyTorch makes experimentation easy, but several frequent errors silently waste compute or corrupt results. Most of them stem from forgetting that the framework does exactly what the code says, with no implicit safeguards.

SymptomCauseFix
Loss explodes or behaves oddlyGradients never clearedCall optimizer.zero_grad() each step
Different results at evaluationDropout and batch norm in training modeCall model.eval() before inference
GPU memory grows every stepStoring tensors with attached graphsUse loss.item() or detach()
Slow data pipeline, idle GPUSingle-process loadingRaise num_workers, pin memory
Unsafe checkpoint loadingPickle can execute arbitrary codeUse weights_only=True or safetensors
Typical PyTorch mistakes and fixes

Broader limits apply as well. Eager execution is slower than a fully compiled graph unless torch.compile is used, deployment to mobile or microcontrollers needs a separate path such as ExecuTorch, and non-NVIDIA hardware support through ROCm, MPS or XLA is real but less mature. Bit-exact reproducibility across GPUs and versions is not guaranteed, even with fixed seeds, because some kernels are nondeterministic.

The short version

PyTorch is a toolkit that lets programmers build and train artificial neural networks in Python. It handles the heavy math on graphics cards, which are far faster than ordinary processors for this kind of work.

Its most useful trick is that it works out the corrections automatically. A model makes a guess, PyTorch measures how wrong the guess was, and then it calculates how every internal setting should change to do better next time. Repeating this millions of times is what training means. Because the code runs step by step like a normal program, researchers can test ideas quickly, which is why most new AI models start life in PyTorch.

Where it makes a difference

A hospital imaging team wants a model that flags suspicious nodules on chest CT scans. Two engineers write a 3D convolutional network as a PyTorch nn.Module, load about 40,000 labeled scans through a DataLoader with eight worker processes, and start from a publicly released pretrained checkpoint instead of random weights.

Training on four GPUs with DistributedDataParallel and bfloat16 autocast takes about two days instead of more than a week on one card without mixed precision. When a validation curve looks wrong, an engineer sets a breakpoint inside the forward pass and finds a mislabeled orientation in one data source, a bug that is easy to see in eager mode.

For deployment, the team captures the final model with torch.export, converts it to ONNX and serves it with an optimized runtime on the radiology workstation. The result is a reviewed screening aid that runs in under a second per scan, while the PyTorch training code remains the single source for future retraining.