A3C, short for asynchronous advantage actor-critic, is a deep reinforcement learning algorithm in which many parallel workers interact with separate copies of an environment and asynchronously update a shared neural network that represents both a policy and a value function. Introduced by researchers at DeepMind in 2016, it showed that parallel on-policy learning on ordinary CPUs could match or exceed the Deep Q-network on Atari games in a fraction of the wall-clock time.
A3C combines three ideas named in its title: asynchronous gradient updates from independent workers, the advantage function as a lower-variance learning signal, and the actor-critic structure in which a policy (the actor) is trained with guidance from a value estimate (the critic).
How A3C trains a shared network
A3C trains by letting each worker collect a short rollout, compute gradients locally and push them to a global parameter set without waiting for the other workers. Each worker then copies the latest global parameters and continues, so the global network receives a steady, unsynchronized stream of updates.
- The worker synchronizes its local network with the global parameters.
- It acts in its own environment copy for up to t_max steps, typically 5 or 20, or until the episode ends.
- It computes n-step returns by bootstrapping from the critic value of the last state.
- It computes policy, value and entropy gradients from the rollout.
- It applies those gradients to the global parameters in a lock-free, Hogwild-style update.
Because different workers explore different parts of the state space at the same moment, their updates are naturally decorrelated. This replaces the experience replay buffer that DQN needs for stability, which in turn allows A3C to remain a purely on-policy method.
The loss function and its components
The A3C objective is a weighted sum of three terms computed on each rollout. The actor and critic usually share convolutional or recurrent layers and split into two output heads.
| Component | Formula idea | Purpose | Typical weight |
|---|---|---|---|
| Policy loss | Minus log probability of action times advantage | Raises probability of better-than-expected actions | 1.0 |
| Value loss | Squared error between n-step return and critic value | Trains the critic to predict returns | 0.5 |
| Entropy bonus | Minus entropy of the action distribution | Prevents premature convergence to a deterministic policy | 0.01 |
The advantage is estimated as the n-step return minus the critic value of the starting state. Subtracting this baseline leaves the expected policy gradient unchanged but substantially reduces its variance, which is the central reason actor-critic methods learn faster than plain REINFORCE.
The rollout length controls the bias-variance balance of this estimate. A one-step advantage leans heavily on the critic and inherits its errors, while a long rollout relies on observed rewards and becomes noisy; later work generalized this choice into generalized advantage estimation, which blends many horizons with a single decay parameter.
Key hyperparameters
A3C is sensitive to learning rate and rollout length, and the original work relied on a broad search over learning rates. The ranges below reflect common published settings for discrete-action benchmarks.
| Hyperparameter | Common range | Effect when increased |
|---|---|---|
| Number of workers | 8 to 32 | Faster data collection, more stale gradients |
| Rollout length t_max | 5 to 20 steps | Less bias, more variance in returns |
| Discount factor | 0.99 | Longer planning horizon, slower credit assignment |
| Learning rate | About 0.0001 to 0.001 | Faster learning until instability appears |
| Entropy coefficient | 0.001 to 0.01 | More exploration, slower policy sharpening |
| Gradient norm clipping | About 40 or lower | Fewer destructive large updates |
The original implementation used shared RMSProp statistics across workers, which proved more robust than per-worker optimizer state. Gradient clipping matters because a single worker with an unusual rollout can otherwise push the shared parameters far from a good region.
A3C compared with related algorithms
A3C sits between early value-based deep RL and the policy optimization methods that later became standard. The comparison shows where it differs from its closest neighbors.
| Algorithm | Policy type | Data reuse | Parallelism | Main stabilizer |
|---|---|---|---|---|
| A3C | Stochastic, on-policy | None | Asynchronous CPU workers | Decorrelated parallel rollouts |
| A2C | Stochastic, on-policy | None | Synchronous batched workers | Large synchronized batches |
| Deep Q-network | Greedy from Q-values, off-policy | Replay buffer | Usually single actor | Replay and target network |
| Proximal policy optimization | Stochastic, on-policy | Several epochs per batch | Synchronous workers | Clipped probability ratio |
| IMPALA | Stochastic, near on-policy | Limited | Distributed actors, central learner | V-trace off-policy correction |
A2C, the synchronous variant, waits for all workers and averages their gradients. In practice it often matches A3C in sample efficiency while using GPUs far more effectively, which is why many libraries implement A2C and treat asynchrony as optional.
Limitations and practical issues
The main weakness of A3C is stale gradients: a worker computes its update with parameters that other workers may already have changed. With many workers, this policy lag adds noise and can destabilize training, and it makes runs hard to reproduce exactly.
- Low sample efficiency: each transition is used once, so A3C needs many environment steps.
- Poor GPU utilization: small per-worker batches underuse accelerators compared with batched methods.
- Run-to-run variance: thread scheduling changes the update order, so results vary between seeds.
- Continuous actions require a Gaussian policy head and careful entropy tuning.
- Sparse reward tasks remain difficult, because the entropy bonus alone gives weak exploration.
For new projects, practitioners usually prefer PPO or IMPALA-style architectures. A3C remains valuable as a clear reference design for asynchronous training and as a baseline in research comparisons.
Understanding it simply
Picture a team of trainees learning the same video game in separate rooms. Each one plays a few moves, notes what went better or worse than expected, and sends a short tip to a shared notebook. Before playing again, each trainee reads the latest version of the notebook.
Because everyone plays different situations at the same time, the notebook improves quickly and does not overfit to one player’s habits. That is the core of A3C: many learners, one shared brain, and constant small corrections based on whether an action turned out better than expected.
Applied in practice
A university robotics group wants a simulated warehouse robot to navigate aisles and reach pick-up points. It has a 32-core server but no spare GPU, so it chooses A3C with 16 worker threads, each running its own copy of the simulator with different random starting positions.
The network shares a small convolutional encoder for a top-down occupancy map and outputs both action probabilities and a state value. With a rollout length of 20 steps, a discount factor of 0.99 and an entropy coefficient of 0.01, the policy learns to reach targets reliably after roughly 40 million environment steps, collected in under a day.
The group then reruns the same task with synchronous A2C on a borrowed GPU and finds similar final performance, which confirms that the gain came mainly from parallel data collection rather than from asynchrony itself.