JAX is an open-source Python library for high-performance numerical computing that pairs a NumPy-compatible array API with composable function transformations: automatic differentiation, just-in-time compilation, vectorization and parallelization. Among AI frameworks & tools, it serves as the computational foundation for many research codebases and for large-scale model training on GPUs and TPUs.
Unlike frameworks that center on layers and modules, JAX centers on pure functions. A program written as ordinary Python and NumPy-style code can be differentiated with grad, compiled with jit, batched with vmap and distributed across devices, and these transformations can be nested in any order. Neural network libraries such as Flax are built on top of it.
How JAX transforms Python functions
JAX works by tracing a Python function with abstract placeholder values and recording the operations into an intermediate representation called a jaxpr. Transformations then rewrite that representation: differentiation produces a new function that computes gradients, and compilation hands the result to XLA (Accelerated Linear Algebra), the compiler that generates fused kernels for CPU, GPU or TPU.
Because tracing records operations rather than executing them eagerly, the traced function must be free of side effects. Printing, mutating global state or reading a random seed from a hidden source behaves differently under tracing, which is the most common surprise for new users. The table below lists the core transformations and what each one produces.
| Transformation | What it produces | Typical use |
|---|---|---|
| grad | A function returning the gradient of a scalar output | Loss gradients for training |
| jit | A compiled version of the function via XLA | Speeding up training and inference steps |
| vmap | A version that maps over a batch axis automatically | Per-example gradients, batched evaluation |
| pmap and shard_map | Execution spread across several devices | Data and model parallelism |
| jacfwd and jacrev | Full Jacobians in forward or reverse mode | Scientific computing, second-order methods |
Key design principles and parameters
Several design choices distinguish JAX from imperative frameworks, and each has practical consequences. Arrays are immutable: instead of in-place assignment, code uses the functional update syntax x.at[i].set(v), which XLA usually turns into an efficient in-place operation after compilation. Randomness is explicit: every random call takes a PRNG key that must be split and passed along, which makes experiments reproducible across devices.
Compilation is driven by array shapes and dtypes. A jitted function is compiled once per distinct input shape, so code that feeds variable-length sequences without padding triggers repeated recompilation. Control flow that depends on array values must use structured primitives such as lax.cond, lax.scan and lax.while_loop rather than Python if and for statements.
| Design choice | Consequence for the developer | Benefit |
|---|---|---|
| Pure functions | State must be passed in and returned explicitly | Transformations compose safely |
| Immutable arrays | Updates use the at-set syntax | Easier reasoning and compiler optimization |
| Explicit PRNG keys | Keys are split and threaded through code | Reproducible randomness in parallel runs |
| Shape-specialized compilation | Dynamic shapes cause recompilation | Highly optimized fused kernels |
| Pytrees | Nested dicts, lists and tuples act as containers | Model parameters handled as plain data |
The pytree abstraction deserves emphasis: any nested structure of arrays can be passed through grad or jit, so model parameters, optimizer states and batches are simply data structures. Optimizers from the Optax library and checkpointing tools such as Orbax operate on these pytrees directly.
How JAX compares with PyTorch and TensorFlow
JAX occupies a lower level of abstraction than PyTorch or TensorFlow with Keras: it is a numerical engine with transformations, while model-building APIs come from companion libraries. The comparison below summarizes the main practical differences that guide framework selection.
| Aspect | JAX | PyTorch | TensorFlow |
|---|---|---|---|
| Programming model | Functional, pure functions | Imperative, object-oriented modules | Graph functions with Keras layers |
| Default execution | Eager, with jit for compilation | Eager, with torch.compile optional | Eager, with tf.function for graphs |
| Compiler backend | XLA | TorchInductor and others | XLA or the TensorFlow runtime |
| TPU support | First-class | Available through PyTorch/XLA | Strong |
| Ecosystem breadth | Smaller, research-focused | Largest, broad industry use | Mature, strong for deployment |
In practice, PyTorch dominates general industry adoption, while JAX is favored where very large-scale training on TPUs, custom gradient research or scientific simulation matter most. Keras 3 can also use JAX as one of its interchangeable backends, which lets teams keep a familiar layer API while running on the JAX compiler stack.
Where JAX is used and how to adopt it
JAX is common in large language model pretraining, reinforcement learning research, differentiable physics and probabilistic programming. Its strength is scaling the same code from a laptop CPU to thousands of accelerator chips by declaring how arrays are partitioned across a device mesh, after which the compiler inserts the required communication.
- Flax provides neural network modules and training state management on top of JAX.
- Optax supplies composable gradient transformations and optimizers such as Adam and AdamW.
- Orbax handles checkpointing of large sharded parameter sets.
- Libraries for probabilistic modeling and physics simulation reuse the same automatic differentiation.
A reasonable adoption path starts with writing the model as pure functions, wrapping the full training step in jit, and only then adding sharding. Profiling the first compiled step is essential, because compilation time for a large model can range from seconds to several minutes and is paid again whenever input shapes change.
Limitations and common mistakes
The functional model that gives JAX its power is also its main barrier to entry. Errors raised during tracing can be hard to read, and debugging compiled code requires tools such as jax.debug.print instead of ordinary print statements. The following table pairs frequent symptoms with their causes and fixes.
| Symptom | Likely cause | Fix |
|---|---|---|
| Slow first step every batch | Changing input shapes trigger recompilation | Pad or bucket inputs to fixed shapes |
| ConcretizationTypeError | Python control flow on traced values | Use lax.cond, lax.scan or static arguments |
| Identical random numbers | Reusing the same PRNG key | Split the key before each use |
| Print shows abstract tracers | Printing inside a jitted function | Use jax.debug.print |
| Out-of-memory on accelerator | Default memory preallocation or large activations | Adjust preallocation, use rematerialization |
The ecosystem is also narrower than PyTorch: fewer pretrained checkpoints, fewer deployment tools and fewer tutorials target JAX first. Teams that need mobile export or broad third-party integrations often convert trained weights or choose another framework for serving.
Understanding it simply
JAX is a toolkit that lets programmers write math in Python and then ask the computer to do three extra things with it automatically. It can work out how every input affects the result, which is what a model needs in order to learn. It can translate the code into fast machine instructions for graphics cards and specialized chips. It can also run the same calculation on many examples or many chips at once.
A useful comparison is a recipe that a kitchen can read, scale up for a banquet and split among many cooks without anyone rewriting it. The catch is that the recipe must be written very cleanly, with no hidden steps, so the kitchen can safely reorganize it. That discipline is the main trade-off of JAX.
Applied in practice
A university research group trains a transformer language model with about one billion parameters on a cloud allocation of 64 TPU chips. The team writes the model with Flax modules, uses Optax for the AdamW optimizer and wraps the entire training step, including the forward pass, loss, gradient computation and parameter update, in a single jitted function.
To fit the model, the researchers define a two-dimensional device mesh and annotate parameters for data and model sharding. Initial runs stall because evaluation batches have varying sequence lengths, so every new length forces recompilation. After padding sequences into four fixed length buckets, compilation happens only four times, and steady-state throughput becomes predictable. The same code later runs unchanged on a single GPU workstation for debugging, which shortens the iteration loop for the whole group.