Soft actor-critic (SAC) is an off-policy actor-critic algorithm for reinforcement learning that trains a stochastic policy to maximize both the expected cumulative reward and the entropy of its own actions. Within artificial intelligence, it belongs to the family of deep reinforcement learning methods for continuous action spaces, such as joint torques of a robot arm or steering and throttle commands.
The entropy term rewards the agent for staying random where randomness costs little, which keeps exploration alive, prevents premature collapse onto a single action and makes training markedly less sensitive to hyperparameters than earlier deterministic methods. Combined with experience replay, this makes SAC one of the most sample-efficient model-free algorithms for continuous control.
The maximum entropy objective
SAC optimizes a maximum entropy objective: at every step the agent receives the environment reward plus a bonus equal to a temperature coefficient alpha multiplied by the entropy of the policy in that state. Standard reinforcement learning maximizes reward alone, so SAC changes the target that the value functions estimate rather than only adding an exploration trick.
The temperature sets the balance between reward and randomness. A high alpha produces broad, exploratory behavior; a low alpha approaches a greedy, nearly deterministic policy. Because the bonus enters the Bellman equation, the resulting values are called soft Q-values, and the optimal policy becomes a distribution proportional to the exponentiated soft Q-function rather than a single best action.
| Temperature setting | Policy behavior | Typical consequence |
|---|---|---|
| Too high | Nearly uniform actions | Slow progress, low final return |
| Well balanced | Broad early, focused later | Stable learning and robust policies |
| Too low | Almost deterministic | Premature convergence to poor optima |
| Automatically tuned | Entropy held near a target | Little manual tuning required |
How soft actor-critic works
SAC alternates between collecting experience with the current stochastic policy and updating its networks from minibatches sampled out of a replay buffer. Because updates do not require fresh on-policy data, every transition can be reused many times, which is the main source of its sample efficiency compared with on-policy methods such as proximal policy optimization.
The modern version of the algorithm uses five neural networks or equivalent copies. Each has a narrowly defined role, and the interplay between them stabilizes learning.
| Component | What it learns | Update target |
|---|---|---|
| Actor network | Mean and variance of a Gaussian policy | Maximize soft Q-value minus alpha times log probability |
| Two critic networks | Soft Q-function estimates | Soft Bellman backup from replayed transitions |
| Two target critics | Slowly lagging critic copies | Polyak averaging with a small rate |
| Temperature alpha | Weight of the entropy bonus | Keep entropy near a target value |
| Replay buffer | Stored transitions | Filled continuously, sampled uniformly |
Two mechanisms deserve emphasis. Clipped double Q-learning takes the minimum of the two critic estimates when forming targets, which counters the overestimation bias that plagues value-based methods. The reparameterization trick samples actions as a deterministic function of the state and independent noise, so the actor gradient flows through the critic with low variance; a tanh squashing function then maps the Gaussian sample into bounded action limits, with a matching correction to the log probability.
Soft actor-critic compared with related algorithms
SAC is best understood against DDPG, TD3 and PPO, the other common choices for continuous control. DDPG and TD3 learn deterministic policies and add external noise for exploration, while PPO is on-policy and discards data after each update. SAC combines the data reuse of off-policy learning with the exploration and stability benefits of a stochastic policy.
| Algorithm | Policy type | Data use | Exploration | Typical strength |
|---|---|---|---|---|
| SAC | Stochastic Gaussian | Off-policy replay | Built-in entropy bonus | Robust and sample-efficient |
| TD3 | Deterministic | Off-policy replay | Added action noise | Strong, simple, fast updates |
| DDPG | Deterministic | Off-policy replay | Added action noise | Historical baseline, brittle |
| PPO | Stochastic | On-policy batches | Policy variance and entropy term | Stable with massive simulation |
In practice, SAC and TD3 often perform similarly on standard locomotion benchmarks, and the better choice depends on the task. PPO usually wins when simulation is cheap and highly parallel, because its sample inefficiency matters less than its simplicity and stability at scale. A discrete-action variant of SAC exists, but value-based methods such as Deep Q-network variants remain more common for discrete problems.
Hyperparameters and practical configuration
SAC is valued for reasonable default settings that transfer across many tasks. The values below are widely used starting points for state-based continuous control, not guarantees; image-based tasks, very long horizons or sparse rewards often need adjustments.
| Hyperparameter | Common default | What it controls |
|---|---|---|
| Learning rate | 0.0003 for all networks | Speed and stability of updates |
| Discount factor | 0.99 | Weight of future rewards |
| Target smoothing rate | 0.005 | How fast target critics follow |
| Replay buffer size | 1 million transitions | Diversity of training data |
| Batch size | 256 | Gradient noise per update |
| Target entropy | Minus the action dimension | Desired randomness when alpha is tuned |
Automatic temperature tuning treats alpha as a learnable variable that is adjusted whenever policy entropy drifts from the target, which removes the most sensitive manual setting. Practitioners also commonly normalize observations, scale rewards to a consistent range and use a short random-action warm-up phase to fill the buffer before learning begins.
Limitations and failure modes
SAC is robust but not universally reliable. Its sample efficiency is still far below what real robots can easily supply, so most successful deployments rely on simulation pretraining or careful data reuse. The entropy objective also means the trained policy is stochastic, and evaluation usually switches to the mean action for deterministic behavior.
- Sparse rewards: entropy-driven exploration is local and rarely solves tasks with very rare rewards without reward shaping or intrinsic motivation.
- High update-to-data ratios: many gradient steps per environment step can cause critic overfitting and value divergence.
- Poorly scaled rewards: the reward scale interacts with alpha, so unnormalized rewards shift the effective balance between reward and entropy.
- Offline settings: without new interaction, critics overestimate unseen actions; offline reinforcement learning variants add conservative penalties.
Understanding it simply
Soft actor-critic is a way for a computer program to learn a skill, such as moving a robot arm, by trial and error. One part of the program, the actor, decides what to do. Another part, the critic, judges how good each choice is likely to be.
The special twist is that the program is rewarded not only for good results but also for keeping its options open. It is a bit like a new employee who is praised for trying different approaches early on instead of sticking to the first thing that worked. This habit helps it discover better solutions and learn steadily without falling into bad routines.
Applied in practice
A robotics team in a logistics warehouse wants a seven-joint arm to pick irregular parcels from a bin. Programming every grasp by hand is impractical, so the team trains a SAC policy in a physics simulator that randomizes parcel shapes, masses and friction. The policy outputs joint velocity commands every 50 milliseconds, and the reward combines lift height, a success bonus and a small penalty for jerky motion.
With automatic temperature tuning and a 1 million transition replay buffer, the policy reaches a simulated success rate of about 90 percent after roughly 3 million environment steps. Early in training the entropy bonus keeps the arm trying different approach angles, which uncovers side grasps for flat parcels that a deterministic method failed to find.
The team then fine-tunes the policy for a few hours on the real arm, reusing logged transitions off-policy. The outcome is a reliable grasping controller that handles new parcel types without manual reprogramming.