JAX

On this page 7

JAX is an open-source Python library for high-performance numerical computing that pairs a NumPy-compatible array API with composable function transformations: automatic differentiation, just-in-time compilation, vectorization and parallelization. Among AI frameworks & tools, it serves as the computational foundation for many research codebases and for large-scale model training on GPUs and TPUs.

Unlike frameworks that center on layers and modules, JAX centers on pure functions. A program written as ordinary Python and NumPy-style code can be differentiated with grad, compiled with jit, batched with vmap and distributed across devices, and these transformations can be nested in any order. Neural network libraries such as Flax are built on top of it.

How JAX transforms Python functions

JAX works by tracing a Python function with abstract placeholder values and recording the operations into an intermediate representation called a jaxpr. Transformations then rewrite that representation: differentiation produces a new function that computes gradients, and compilation hands the result to XLA (Accelerated Linear Algebra), the compiler that generates fused kernels for CPU, GPU or TPU.

Because tracing records operations rather than executing them eagerly, the traced function must be free of side effects. Printing, mutating global state or reading a random seed from a hidden source behaves differently under tracing, which is the most common surprise for new users. The table below lists the core transformations and what each one produces.

TransformationWhat it producesTypical use
gradA function returning the gradient of a scalar outputLoss gradients for training
jitA compiled version of the function via XLASpeeding up training and inference steps
vmapA version that maps over a batch axis automaticallyPer-example gradients, batched evaluation
pmap and shard_mapExecution spread across several devicesData and model parallelism
jacfwd and jacrevFull Jacobians in forward or reverse modeScientific computing, second-order methods
Core function transformations in JAX

Key design principles and parameters

Several design choices distinguish JAX from imperative frameworks, and each has practical consequences. Arrays are immutable: instead of in-place assignment, code uses the functional update syntax x.at[i].set(v), which XLA usually turns into an efficient in-place operation after compilation. Randomness is explicit: every random call takes a PRNG key that must be split and passed along, which makes experiments reproducible across devices.

Compilation is driven by array shapes and dtypes. A jitted function is compiled once per distinct input shape, so code that feeds variable-length sequences without padding triggers repeated recompilation. Control flow that depends on array values must use structured primitives such as lax.cond, lax.scan and lax.while_loop rather than Python if and for statements.

Design choiceConsequence for the developerBenefit
Pure functionsState must be passed in and returned explicitlyTransformations compose safely
Immutable arraysUpdates use the at-set syntaxEasier reasoning and compiler optimization
Explicit PRNG keysKeys are split and threaded through codeReproducible randomness in parallel runs
Shape-specialized compilationDynamic shapes cause recompilationHighly optimized fused kernels
PytreesNested dicts, lists and tuples act as containersModel parameters handled as plain data
Design choices and their practical consequences

The pytree abstraction deserves emphasis: any nested structure of arrays can be passed through grad or jit, so model parameters, optimizer states and batches are simply data structures. Optimizers from the Optax library and checkpointing tools such as Orbax operate on these pytrees directly.

How JAX compares with PyTorch and TensorFlow

JAX occupies a lower level of abstraction than PyTorch or TensorFlow with Keras: it is a numerical engine with transformations, while model-building APIs come from companion libraries. The comparison below summarizes the main practical differences that guide framework selection.

AspectJAXPyTorchTensorFlow
Programming modelFunctional, pure functionsImperative, object-oriented modulesGraph functions with Keras layers
Default executionEager, with jit for compilationEager, with torch.compile optionalEager, with tf.function for graphs
Compiler backendXLATorchInductor and othersXLA or the TensorFlow runtime
TPU supportFirst-classAvailable through PyTorch/XLAStrong
Ecosystem breadthSmaller, research-focusedLargest, broad industry useMature, strong for deployment
JAX compared with related frameworks

In practice, PyTorch dominates general industry adoption, while JAX is favored where very large-scale training on TPUs, custom gradient research or scientific simulation matter most. Keras 3 can also use JAX as one of its interchangeable backends, which lets teams keep a familiar layer API while running on the JAX compiler stack.

Where JAX is used and how to adopt it

JAX is common in large language model pretraining, reinforcement learning research, differentiable physics and probabilistic programming. Its strength is scaling the same code from a laptop CPU to thousands of accelerator chips by declaring how arrays are partitioned across a device mesh, after which the compiler inserts the required communication.

  • Flax provides neural network modules and training state management on top of JAX.
  • Optax supplies composable gradient transformations and optimizers such as Adam and AdamW.
  • Orbax handles checkpointing of large sharded parameter sets.
  • Libraries for probabilistic modeling and physics simulation reuse the same automatic differentiation.

A reasonable adoption path starts with writing the model as pure functions, wrapping the full training step in jit, and only then adding sharding. Profiling the first compiled step is essential, because compilation time for a large model can range from seconds to several minutes and is paid again whenever input shapes change.

Limitations and common mistakes

The functional model that gives JAX its power is also its main barrier to entry. Errors raised during tracing can be hard to read, and debugging compiled code requires tools such as jax.debug.print instead of ordinary print statements. The following table pairs frequent symptoms with their causes and fixes.

SymptomLikely causeFix
Slow first step every batchChanging input shapes trigger recompilationPad or bucket inputs to fixed shapes
ConcretizationTypeErrorPython control flow on traced valuesUse lax.cond, lax.scan or static arguments
Identical random numbersReusing the same PRNG keySplit the key before each use
Print shows abstract tracersPrinting inside a jitted functionUse jax.debug.print
Out-of-memory on acceleratorDefault memory preallocation or large activationsAdjust preallocation, use rematerialization
Common JAX problems and remedies

The ecosystem is also narrower than PyTorch: fewer pretrained checkpoints, fewer deployment tools and fewer tutorials target JAX first. Teams that need mobile export or broad third-party integrations often convert trained weights or choose another framework for serving.

Understanding it simply

JAX is a toolkit that lets programmers write math in Python and then ask the computer to do three extra things with it automatically. It can work out how every input affects the result, which is what a model needs in order to learn. It can translate the code into fast machine instructions for graphics cards and specialized chips. It can also run the same calculation on many examples or many chips at once.

A useful comparison is a recipe that a kitchen can read, scale up for a banquet and split among many cooks without anyone rewriting it. The catch is that the recipe must be written very cleanly, with no hidden steps, so the kitchen can safely reorganize it. That discipline is the main trade-off of JAX.

Applied in practice

A university research group trains a transformer language model with about one billion parameters on a cloud allocation of 64 TPU chips. The team writes the model with Flax modules, uses Optax for the AdamW optimizer and wraps the entire training step, including the forward pass, loss, gradient computation and parameter update, in a single jitted function.

To fit the model, the researchers define a two-dimensional device mesh and annotate parameters for data and model sharding. Initial runs stall because evaluation batches have varying sequence lengths, so every new length forces recompilation. After padding sequences into four fixed length buckets, compilation happens only four times, and steady-state throughput becomes predictable. The same code later runs unchanged on a single GPU workstation for debugging, which shortens the iteration loop for the whole group.