PyTorch is an open-source deep learning framework for Python that provides GPU-accelerated tensors, automatic differentiation and a library of neural network building blocks for training and running models. Among AI frameworks & tools, it is the dominant choice in machine learning research and a standard foundation for large language models, computer vision and generative systems in Artificial intelligence.
Originally developed at Meta AI and released in 2016, PyTorch is now governed by the PyTorch Foundation under the Linux Foundation. Its defining trait is eager execution: operations run immediately, line by line, so a model is ordinary Python code that can be inspected, debugged and modified like any other program.
How PyTorch executes and differentiates code
PyTorch works by recording tensor operations as they execute and replaying them backward to compute gradients. The core data structure is the tensor, an n-dimensional array that lives on a CPU, an NVIDIA or AMD GPU, or Apple silicon, and carries a data type such as float32, bfloat16 or int8.
When a tensor has requires_grad=True, the autograd engine builds a dynamic computational graph during the forward pass. Calling loss.backward() walks that graph in reverse and applies the chain rule, storing gradients in each parameter’s .grad field. Because the graph is rebuilt on every iteration, control flow can change per input: loops, conditionals and variable sequence lengths need no special syntax.
The main components that a typical training program touches are summarized below.
| Component | Role | Typical use |
|---|---|---|
| torch.Tensor | Multidimensional array with device and dtype | Inputs, weights, activations |
| torch.autograd | Reverse-mode automatic differentiation | Computing gradients of a loss |
| torch.nn | Layers, losses and the Module base class | Defining model architecture |
| torch.optim | Optimizers such as SGD, Adam and AdamW | Updating parameters each step |
| torch.utils.data | Dataset and DataLoader abstractions | Batching, shuffling, parallel loading |
| torch.compile | Graph capture and kernel generation | Speeding up training and inference |
From eager mode to compiled graphs
PyTorch offers several execution modes that trade flexibility for speed and portability. Eager mode is the default and is easiest to debug, but each operation launches separately, which leaves Python overhead and memory traffic on the table.
Since PyTorch 2.0, torch.compile captures Python bytecode with TorchDynamo, lowers it through AOTAutograd and generates fused kernels with TorchInductor, often in the Triton language for GPUs. Reported speedups commonly fall between about 1.3x and 2x for training, but the gain depends heavily on model shape, batch size and how often graph breaks force a fallback to eager code.
The following table compares the execution and export paths a team can choose from.
| Mode | Strength | Limitation |
|---|---|---|
| Eager execution | Immediate results, simple debugging | Highest per-operation overhead |
| torch.compile | Fused kernels with minimal code changes | Compile time, graph breaks, recompilation |
| TorchScript | Python-free serialized models | Legacy path, restricted Python subset |
| torch.export | Clean full graph for deployment | Requires traceable, static-friendly code |
| ONNX export | Portability to other runtimes | Operator coverage gaps for new layers |
PyTorch compared with TensorFlow and JAX
PyTorch differs from its main rivals mostly in programming model and ecosystem rather than in raw capability. TensorFlow began with static graphs and remains strong in established production pipelines and mobile tooling, while JAX uses functional transformations such as jit, grad and vmap over pure functions and is popular for large-scale research on TPUs.
In practice, the research share of PyTorch is very high: most new model code, reference implementations and checkpoints on public model hubs target it first. Higher-level libraries such as PyTorch Lightning, fastai and Hugging Face Transformers build directly on PyTorch, and Keras 3 can use it as a backend.
| Aspect | PyTorch | TensorFlow | JAX |
|---|---|---|---|
| Programming style | Object-oriented, imperative | Keras layers, graph functions | Functional, pure functions |
| Default execution | Eager | Eager with tf.function graphs | Traced and compiled via XLA |
| Strongest hardware | NVIDIA GPUs | GPUs and TPUs | TPUs and GPUs |
| Research adoption | Dominant | Declining | Strong in large labs |
| Edge deployment | ExecuTorch | LiteRT | Through export to other formats |
Scaling PyTorch training across devices
PyTorch scales beyond one GPU through the torch.distributed package, which uses collective communication backends such as NCCL for NVIDIA hardware and Gloo for CPUs. The right strategy depends on whether the model fits in one device’s memory.
| Strategy | What is split | When to use it |
|---|---|---|
| DistributedDataParallel | Data batches; full model per GPU | Model fits on one GPU |
| Fully Sharded Data Parallel | Parameters, gradients, optimizer state | Model too large for one GPU |
| Tensor parallelism | Individual weight matrices | Very wide layers in large transformers |
| Pipeline parallelism | Groups of layers across devices | Very deep models across many nodes |
Memory savings also come from mixed precision with torch.autocast in bfloat16 or float16, which roughly halves activation memory, and from activation checkpointing, which recomputes intermediate values instead of storing them. Libraries such as DeepSpeed and Ray add further orchestration on top of these primitives.
Limitations and common mistakes
PyTorch makes experimentation easy, but several frequent errors silently waste compute or corrupt results. Most of them stem from forgetting that the framework does exactly what the code says, with no implicit safeguards.
| Symptom | Cause | Fix |
|---|---|---|
| Loss explodes or behaves oddly | Gradients never cleared | Call optimizer.zero_grad() each step |
| Different results at evaluation | Dropout and batch norm in training mode | Call model.eval() before inference |
| GPU memory grows every step | Storing tensors with attached graphs | Use loss.item() or detach() |
| Slow data pipeline, idle GPU | Single-process loading | Raise num_workers, pin memory |
| Unsafe checkpoint loading | Pickle can execute arbitrary code | Use weights_only=True or safetensors |
Broader limits apply as well. Eager execution is slower than a fully compiled graph unless torch.compile is used, deployment to mobile or microcontrollers needs a separate path such as ExecuTorch, and non-NVIDIA hardware support through ROCm, MPS or XLA is real but less mature. Bit-exact reproducibility across GPUs and versions is not guaranteed, even with fixed seeds, because some kernels are nondeterministic.
The short version
PyTorch is a toolkit that lets programmers build and train artificial neural networks in Python. It handles the heavy math on graphics cards, which are far faster than ordinary processors for this kind of work.
Its most useful trick is that it works out the corrections automatically. A model makes a guess, PyTorch measures how wrong the guess was, and then it calculates how every internal setting should change to do better next time. Repeating this millions of times is what training means. Because the code runs step by step like a normal program, researchers can test ideas quickly, which is why most new AI models start life in PyTorch.
Where it makes a difference
A hospital imaging team wants a model that flags suspicious nodules on chest CT scans. Two engineers write a 3D convolutional network as a PyTorch nn.Module, load about 40,000 labeled scans through a DataLoader with eight worker processes, and start from a publicly released pretrained checkpoint instead of random weights.
Training on four GPUs with DistributedDataParallel and bfloat16 autocast takes about two days instead of more than a week on one card without mixed precision. When a validation curve looks wrong, an engineer sets a breakpoint inside the forward pass and finds a mislabeled orientation in one data source, a bug that is easy to see in eager mode.
For deployment, the team captures the final model with torch.export, converts it to ONNX and serves it with an optimized runtime on the radiology workstation. The result is a reviewed screening aid that runs in under a second per scan, while the PyTorch training code remains the single source for future retraining.