Flax

On this page 7

Flax is an open-source neural network library built on JAX, developed at Google, that provides layers, model containers and parameter management so engineers can define and train deep learning models while keeping access to the composable function transformations of JAX. Within AI frameworks & tools for artificial intelligence, Flax occupies the role that Keras or the torch.nn package play elsewhere: it turns low-level array programming into reusable model building blocks.

Flax does not run computations itself. Every forward pass, gradient and parameter update is ultimately a JAX function compiled by XLA for CPUs, GPUs or TPUs. The library therefore concentrates on structure, state handling and ergonomics, and it is commonly combined with Optax for optimizers and Orbax for checkpointing to form a complete training stack.

How Flax organizes models on top of JAX

Flax reconciles object-oriented model definitions with the pure-function requirement of JAX transformations. Functions passed to jax.jit, jax.grad or jax.vmap must not hide mutable state, yet neural networks carry weights, batch statistics and random number generator keys. Flax solves this by making state explicit and movable as pytrees, the nested containers of arrays that JAX can trace.

A typical workflow has four stages. The developer declares a module, initializes parameters with a random key and a sample input, runs the forward pass inside a compiled training step, and passes the resulting gradients to an Optax optimizer. Because parameters live in ordinary pytrees, they can be sharded across devices, saved, inspected or transformed with the same tools that handle any JAX data.

The table below summarizes the main components that a Flax project usually touches.

ComponentRoleTypical example
ModuleDefines layers and the forward computationDense, Conv, MultiHeadAttention
VariablesHold parameters and mutable stateparams, batch_stats, cache
RNG streamsSupply keys for initialization and dropoutparams key, dropout key
Optax optimizerTurns gradients into parameter updatesAdamW with a cosine schedule
Orbax checkpointerSaves and restores sharded stateAsynchronous multi-host checkpoints
Core building blocks of a Flax training stack

Flax Linen and Flax NNX

Flax currently exposes two module APIs, and choosing between them is the first design decision in any new project. Flax Linen is the older, functional API: modules are dataclasses, parameters are created lazily inside setup or @nn.compact methods, and the model is executed through init and apply, which receive and return variable collections explicitly.

Flax NNX is the newer API that the Flax team now recommends for new code. NNX modules are regular Python objects that own their parameters as attributes, so a layer can be created, called and inspected directly, much like a PyTorch module. Lifted transforms such as nnx.jit and nnx.grad, together with nnx.split and nnx.merge, convert between the stateful object and the pure state pytree that JAX requires.

AspectFlax LinenFlax NNX
State ownershipKept outside the module in collectionsStored as module attributes
Parameter creationLazy, shape inferred on first callEager, shapes given in the constructor
ExecutionExplicit init and apply callsDirect method calls on the object
DebuggingRequires reading variable dictionariesAttributes visible in a debugger
StatusMaintained, large existing codebaseRecommended for new projects
Differences between the two Flax module APIs

Both APIs produce models that compile to the same XLA programs, so runtime performance is broadly equivalent. The difference lies in readability, refactoring cost and how naturally the code expresses stateful layers such as batch normalization or key-value caches.

Flax is best understood as one layer in the JAX ecosystem, not as a standalone framework. JAX supplies arrays, automatic differentiation and compilation; Flax supplies the neural network abstractions; Optax and Orbax handle optimization and persistence. Other frameworks bundle these concerns differently.

ToolMain abstractionRelation to Flax
JAXTransformable array functionsThe foundation Flax builds on
PyTorchEager mutable modulesComparable role, different execution model
KerasHigh-level layers and fit loopKeras 3 can also run on a JAX backend
TensorFlowGraphs and Keras integrationShares XLA but not the module system
EquinoxModules as pytreesAlternative JAX library with a lighter design
Flax in relation to neighboring tools

The practical distinction from PyTorch is that compilation is the default path in Flax. Training steps are traced once and executed as optimized programs, which rewards static shapes and penalizes Python control flow that depends on array values.

When Flax is the right choice

Flax is strongest for large-scale training on TPUs and multi-accelerator clusters, where the sharding primitives of JAX and the explicit state model simplify data, tensor and pipeline parallelism. Research groups also value it for reproducible randomness, because every random operation consumes an explicit key.

SituationRecommendationReason
Training on Cloud TPUUse Flax with JAXFirst-class XLA and sharding support
New JAX projectStart with Flax NNXSimpler state handling and debugging
Existing Linen codebaseKeep Linen or migrate graduallyInteroperability bridges exist
Team fluent only in PyTorchWeigh retraining costFunctional habits take time to learn
Mobile or edge deploymentPlan an export path earlyExport tooling is narrower than PyTorch
Decision guide for adopting Flax

Several large language model codebases at Google and in the open research community are written in Flax, and model families such as Gemma have been released with JAX and Flax reference implementations, which makes Flax a practical entry point for studying those architectures.

Limitations and common mistakes

Most problems with Flax stem from the functional constraints of JAX rather than from Flax itself. Engineers coming from eager frameworks frequently trigger recompilation, leak state or misuse random keys.

  • Changing input shapes between steps forces XLA to recompile, so batches should be padded or bucketed.
  • Reusing the same RNG key for dropout in every step silently removes randomness; keys must be split or advanced.
  • Forgetting mutable collections in Linen, such as batch statistics, leaves normalization layers frozen at initial values.
  • Mixing Linen and NNX without the provided bridge utilities produces confusing state structures.
  • Python side effects inside compiled functions run only during tracing, which makes print-based debugging misleading.

The ecosystem is also smaller than that of PyTorch: fewer third-party layers, fewer tutorials and a narrower selection of deployment converters. Teams should verify that the components they need, such as specific attention kernels or quantization schemes, are available before committing.

Understanding it simply

Imagine JAX as a very fast calculator that can also work out how every result changes when its inputs change. That ability is exactly what neural networks need for learning, but the calculator only understands raw numbers and formulas.

Flax is the toolkit of ready-made parts that sits on top of that calculator. It lets an engineer say “a layer of this size, then an attention block, then an output layer” and keeps track of the millions of numbers the model learns. The key idea is a clean split: JAX does the heavy math quickly, and Flax keeps the model organized and easy to train.

Applied in practice

A university research group wants to train a 350-million-parameter language model on a donated allocation of cloud TPU time. The team writes the model in Flax NNX, defining attention, feed-forward and embedding layers as ordinary Python classes, and uses Optax for an AdamW optimizer with warmup and cosine decay.

Because parameters are plain pytrees, the engineers annotate them with sharding rules and spread the model across 32 TPU cores without rewriting the layers. Orbax writes checkpoints every 2,000 steps, so a preempted job resumes within minutes. After about two weeks the group reaches its target validation loss, and the same Flax code runs unchanged on a single GPU workstation for evaluation and ablation experiments.