A3C

Also known as: Asynchronous advantage actor-critic

On this page 7

A3C, short for asynchronous advantage actor-critic, is a deep reinforcement learning algorithm in which many parallel workers interact with separate copies of an environment and asynchronously update a shared neural network that represents both a policy and a value function. Introduced by researchers at DeepMind in 2016, it showed that parallel on-policy learning on ordinary CPUs could match or exceed the Deep Q-network on Atari games in a fraction of the wall-clock time.

A3C combines three ideas named in its title: asynchronous gradient updates from independent workers, the advantage function as a lower-variance learning signal, and the actor-critic structure in which a policy (the actor) is trained with guidance from a value estimate (the critic).

How A3C trains a shared network

A3C trains by letting each worker collect a short rollout, compute gradients locally and push them to a global parameter set without waiting for the other workers. Each worker then copies the latest global parameters and continues, so the global network receives a steady, unsynchronized stream of updates.

  1. The worker synchronizes its local network with the global parameters.
  2. It acts in its own environment copy for up to t_max steps, typically 5 or 20, or until the episode ends.
  3. It computes n-step returns by bootstrapping from the critic value of the last state.
  4. It computes policy, value and entropy gradients from the rollout.
  5. It applies those gradients to the global parameters in a lock-free, Hogwild-style update.

Because different workers explore different parts of the state space at the same moment, their updates are naturally decorrelated. This replaces the experience replay buffer that DQN needs for stability, which in turn allows A3C to remain a purely on-policy method.

The loss function and its components

The A3C objective is a weighted sum of three terms computed on each rollout. The actor and critic usually share convolutional or recurrent layers and split into two output heads.

ComponentFormula ideaPurposeTypical weight
Policy lossMinus log probability of action times advantageRaises probability of better-than-expected actions1.0
Value lossSquared error between n-step return and critic valueTrains the critic to predict returns0.5
Entropy bonusMinus entropy of the action distributionPrevents premature convergence to a deterministic policy0.01
Components of the A3C objective

The advantage is estimated as the n-step return minus the critic value of the starting state. Subtracting this baseline leaves the expected policy gradient unchanged but substantially reduces its variance, which is the central reason actor-critic methods learn faster than plain REINFORCE.

The rollout length controls the bias-variance balance of this estimate. A one-step advantage leans heavily on the critic and inherits its errors, while a long rollout relies on observed rewards and becomes noisy; later work generalized this choice into generalized advantage estimation, which blends many horizons with a single decay parameter.

Key hyperparameters

A3C is sensitive to learning rate and rollout length, and the original work relied on a broad search over learning rates. The ranges below reflect common published settings for discrete-action benchmarks.

HyperparameterCommon rangeEffect when increased
Number of workers8 to 32Faster data collection, more stale gradients
Rollout length t_max5 to 20 stepsLess bias, more variance in returns
Discount factor0.99Longer planning horizon, slower credit assignment
Learning rateAbout 0.0001 to 0.001Faster learning until instability appears
Entropy coefficient0.001 to 0.01More exploration, slower policy sharpening
Gradient norm clippingAbout 40 or lowerFewer destructive large updates
Typical A3C hyperparameters

The original implementation used shared RMSProp statistics across workers, which proved more robust than per-worker optimizer state. Gradient clipping matters because a single worker with an unusual rollout can otherwise push the shared parameters far from a good region.

A3C sits between early value-based deep RL and the policy optimization methods that later became standard. The comparison shows where it differs from its closest neighbors.

AlgorithmPolicy typeData reuseParallelismMain stabilizer
A3CStochastic, on-policyNoneAsynchronous CPU workersDecorrelated parallel rollouts
A2CStochastic, on-policyNoneSynchronous batched workersLarge synchronized batches
Deep Q-networkGreedy from Q-values, off-policyReplay bufferUsually single actorReplay and target network
Proximal policy optimizationStochastic, on-policySeveral epochs per batchSynchronous workersClipped probability ratio
IMPALAStochastic, near on-policyLimitedDistributed actors, central learnerV-trace off-policy correction
A3C and neighboring reinforcement learning algorithms

A2C, the synchronous variant, waits for all workers and averages their gradients. In practice it often matches A3C in sample efficiency while using GPUs far more effectively, which is why many libraries implement A2C and treat asynchrony as optional.

Limitations and practical issues

The main weakness of A3C is stale gradients: a worker computes its update with parameters that other workers may already have changed. With many workers, this policy lag adds noise and can destabilize training, and it makes runs hard to reproduce exactly.

  • Low sample efficiency: each transition is used once, so A3C needs many environment steps.
  • Poor GPU utilization: small per-worker batches underuse accelerators compared with batched methods.
  • Run-to-run variance: thread scheduling changes the update order, so results vary between seeds.
  • Continuous actions require a Gaussian policy head and careful entropy tuning.
  • Sparse reward tasks remain difficult, because the entropy bonus alone gives weak exploration.

For new projects, practitioners usually prefer PPO or IMPALA-style architectures. A3C remains valuable as a clear reference design for asynchronous training and as a baseline in research comparisons.

Understanding it simply

Picture a team of trainees learning the same video game in separate rooms. Each one plays a few moves, notes what went better or worse than expected, and sends a short tip to a shared notebook. Before playing again, each trainee reads the latest version of the notebook.

Because everyone plays different situations at the same time, the notebook improves quickly and does not overfit to one player’s habits. That is the core of A3C: many learners, one shared brain, and constant small corrections based on whether an action turned out better than expected.

Applied in practice

A university robotics group wants a simulated warehouse robot to navigate aisles and reach pick-up points. It has a 32-core server but no spare GPU, so it chooses A3C with 16 worker threads, each running its own copy of the simulator with different random starting positions.

The network shares a small convolutional encoder for a top-down occupancy map and outputs both action probabilities and a state value. With a rollout length of 20 steps, a discount factor of 0.99 and an entropy coefficient of 0.01, the policy learns to reach targets reliably after roughly 40 million environment steps, collected in under a day.

The group then reruns the same task with synchronous A2C on a borrowed GPU and finds similar final performance, which confirms that the gain came mainly from parallel data collection rather than from asynchrony itself.