Soft actor-critic

Also known as: SAC

On this page 7

Soft actor-critic (SAC) is an off-policy actor-critic algorithm for reinforcement learning that trains a stochastic policy to maximize both the expected cumulative reward and the entropy of its own actions. Within artificial intelligence, it belongs to the family of deep reinforcement learning methods for continuous action spaces, such as joint torques of a robot arm or steering and throttle commands.

The entropy term rewards the agent for staying random where randomness costs little, which keeps exploration alive, prevents premature collapse onto a single action and makes training markedly less sensitive to hyperparameters than earlier deterministic methods. Combined with experience replay, this makes SAC one of the most sample-efficient model-free algorithms for continuous control.

The maximum entropy objective

SAC optimizes a maximum entropy objective: at every step the agent receives the environment reward plus a bonus equal to a temperature coefficient alpha multiplied by the entropy of the policy in that state. Standard reinforcement learning maximizes reward alone, so SAC changes the target that the value functions estimate rather than only adding an exploration trick.

The temperature sets the balance between reward and randomness. A high alpha produces broad, exploratory behavior; a low alpha approaches a greedy, nearly deterministic policy. Because the bonus enters the Bellman equation, the resulting values are called soft Q-values, and the optimal policy becomes a distribution proportional to the exponentiated soft Q-function rather than a single best action.

Temperature settingPolicy behaviorTypical consequence
Too highNearly uniform actionsSlow progress, low final return
Well balancedBroad early, focused laterStable learning and robust policies
Too lowAlmost deterministicPremature convergence to poor optima
Automatically tunedEntropy held near a targetLittle manual tuning required
Effect of the entropy temperature

How soft actor-critic works

SAC alternates between collecting experience with the current stochastic policy and updating its networks from minibatches sampled out of a replay buffer. Because updates do not require fresh on-policy data, every transition can be reused many times, which is the main source of its sample efficiency compared with on-policy methods such as proximal policy optimization.

The modern version of the algorithm uses five neural networks or equivalent copies. Each has a narrowly defined role, and the interplay between them stabilizes learning.

ComponentWhat it learnsUpdate target
Actor networkMean and variance of a Gaussian policyMaximize soft Q-value minus alpha times log probability
Two critic networksSoft Q-function estimatesSoft Bellman backup from replayed transitions
Two target criticsSlowly lagging critic copiesPolyak averaging with a small rate
Temperature alphaWeight of the entropy bonusKeep entropy near a target value
Replay bufferStored transitionsFilled continuously, sampled uniformly
Components of soft actor-critic

Two mechanisms deserve emphasis. Clipped double Q-learning takes the minimum of the two critic estimates when forming targets, which counters the overestimation bias that plagues value-based methods. The reparameterization trick samples actions as a deterministic function of the state and independent noise, so the actor gradient flows through the critic with low variance; a tanh squashing function then maps the Gaussian sample into bounded action limits, with a matching correction to the log probability.

SAC is best understood against DDPG, TD3 and PPO, the other common choices for continuous control. DDPG and TD3 learn deterministic policies and add external noise for exploration, while PPO is on-policy and discards data after each update. SAC combines the data reuse of off-policy learning with the exploration and stability benefits of a stochastic policy.

AlgorithmPolicy typeData useExplorationTypical strength
SACStochastic GaussianOff-policy replayBuilt-in entropy bonusRobust and sample-efficient
TD3DeterministicOff-policy replayAdded action noiseStrong, simple, fast updates
DDPGDeterministicOff-policy replayAdded action noiseHistorical baseline, brittle
PPOStochasticOn-policy batchesPolicy variance and entropy termStable with massive simulation
Continuous control algorithms side by side

In practice, SAC and TD3 often perform similarly on standard locomotion benchmarks, and the better choice depends on the task. PPO usually wins when simulation is cheap and highly parallel, because its sample inefficiency matters less than its simplicity and stability at scale. A discrete-action variant of SAC exists, but value-based methods such as Deep Q-network variants remain more common for discrete problems.

Hyperparameters and practical configuration

SAC is valued for reasonable default settings that transfer across many tasks. The values below are widely used starting points for state-based continuous control, not guarantees; image-based tasks, very long horizons or sparse rewards often need adjustments.

HyperparameterCommon defaultWhat it controls
Learning rate0.0003 for all networksSpeed and stability of updates
Discount factor0.99Weight of future rewards
Target smoothing rate0.005How fast target critics follow
Replay buffer size1 million transitionsDiversity of training data
Batch size256Gradient noise per update
Target entropyMinus the action dimensionDesired randomness when alpha is tuned
Common starting hyperparameters

Automatic temperature tuning treats alpha as a learnable variable that is adjusted whenever policy entropy drifts from the target, which removes the most sensitive manual setting. Practitioners also commonly normalize observations, scale rewards to a consistent range and use a short random-action warm-up phase to fill the buffer before learning begins.

Limitations and failure modes

SAC is robust but not universally reliable. Its sample efficiency is still far below what real robots can easily supply, so most successful deployments rely on simulation pretraining or careful data reuse. The entropy objective also means the trained policy is stochastic, and evaluation usually switches to the mean action for deterministic behavior.

  • Sparse rewards: entropy-driven exploration is local and rarely solves tasks with very rare rewards without reward shaping or intrinsic motivation.
  • High update-to-data ratios: many gradient steps per environment step can cause critic overfitting and value divergence.
  • Poorly scaled rewards: the reward scale interacts with alpha, so unnormalized rewards shift the effective balance between reward and entropy.
  • Offline settings: without new interaction, critics overestimate unseen actions; offline reinforcement learning variants add conservative penalties.

Understanding it simply

Soft actor-critic is a way for a computer program to learn a skill, such as moving a robot arm, by trial and error. One part of the program, the actor, decides what to do. Another part, the critic, judges how good each choice is likely to be.

The special twist is that the program is rewarded not only for good results but also for keeping its options open. It is a bit like a new employee who is praised for trying different approaches early on instead of sticking to the first thing that worked. This habit helps it discover better solutions and learn steadily without falling into bad routines.

Applied in practice

A robotics team in a logistics warehouse wants a seven-joint arm to pick irregular parcels from a bin. Programming every grasp by hand is impractical, so the team trains a SAC policy in a physics simulator that randomizes parcel shapes, masses and friction. The policy outputs joint velocity commands every 50 milliseconds, and the reward combines lift height, a success bonus and a small penalty for jerky motion.

With automatic temperature tuning and a 1 million transition replay buffer, the policy reaches a simulated success rate of about 90 percent after roughly 3 million environment steps. Early in training the entropy bonus keeps the arm trying different approach angles, which uncovers side grasps for flat parcels that a deterministic method failed to find.

The team then fine-tunes the policy for a few hours on the real arm, reusing logged transitions off-policy. The outcome is a reliable grasping controller that handles new parcel types without manual reprogramming.