Stable-Baselines3 (SB3) is an open-source Python library of reliable reinforcement learning algorithms built on PyTorch, offering implementations such as PPO, SAC, TD3 and DQN behind one consistent training interface. Among AI frameworks & tools it occupies a specific niche: it does not build neural networks or serve language models, but trains agents that learn by trial and error in simulated or real environments.
The library is the PyTorch successor to the original Stable Baselines, which was itself a fork of OpenAI Baselines. Its design goal is reproducibility over novelty: each algorithm is benchmarked against published results, covered by unit tests and documented, so that practitioners can treat it as a trusted baseline rather than research code that may silently diverge.
How Stable-Baselines3 structures reinforcement learning
Stable-Baselines3 organizes every experiment around three objects: an environment, a policy and an algorithm. The environment follows the Gymnasium API (reset, step, observation space, action space), the policy is a PyTorch network that maps observations to actions, and the algorithm decides how collected experience updates that network.
A typical run needs only a few lines: create the environment, instantiate a model such as PPO("MlpPolicy", env), call model.learn(total_timesteps=...) and then model.save(). Behind this short interface the library handles rollout collection, advantage estimation, replay buffers, gradient updates and logging, which are the parts where hand-written implementations most often contain subtle bugs.
The following table lists the main building blocks and the role each one plays in a training run.
| Component | Example | Role in training |
|---|---|---|
| Environment | Gymnasium CartPole-v1 | Supplies observations, rewards and episode termination |
| Vectorized environment | DummyVecEnv, SubprocVecEnv | Runs several environment copies in parallel |
| Policy | MlpPolicy, CnnPolicy, MultiInputPolicy | Neural network that selects actions |
| Algorithm | PPO, SAC, DQN | Defines how experience updates the policy |
| Callback | EvalCallback, CheckpointCallback | Evaluates, saves or stops training at set intervals |
| Logger | TensorBoard, CSV, stdout | Records rewards, losses and episode lengths |
Algorithms included in Stable-Baselines3
The core package deliberately contains a small set of well-established algorithms, split into on-policy and off-policy methods. On-policy algorithms such as PPO and A2C learn only from data generated by the current policy, which makes them stable but sample-hungry. Off-policy algorithms such as SAC, TD3 and DQN reuse past experience from a replay buffer, which usually improves sample efficiency at the cost of more tuning.
The table compares the main algorithms by the action spaces they support and their typical use.
| Algorithm | Action space | Policy type | Typical use |
|---|---|---|---|
| PPO | Discrete and continuous | On-policy | Robust default for most tasks |
| A2C | Discrete and continuous | On-policy | Fast, simple synchronous training |
| DQN | Discrete only | Off-policy | Games and discrete control |
| SAC | Continuous only | Off-policy | Robotics and continuous control |
| TD3 | Continuous only | Off-policy | Deterministic continuous control |
| DDPG | Continuous only | Off-policy | Historical baseline, mostly replaced by TD3 |
Hindsight Experience Replay is available as a replay buffer that works with the off-policy algorithms for goal-conditioned tasks. Experimental or less common methods, including RecurrentPPO, MaskablePPO, TQC, QR-DQN, TRPO and ARS, live in the companion package SB3-Contrib, which keeps the core library small and stable.
Stable-Baselines3 compared with related tools
Stable-Baselines3 is a training library, not an environment library or a general deep learning framework. It sits on top of PyTorch and consumes environments written against Gymnasium, while larger systems such as Ray RLlib target distributed training at cluster scale.
The comparison below clarifies how SB3 relates to neighboring tools that practitioners often combine with it.
| Tool | Primary purpose | Relationship to SB3 |
|---|---|---|
| Gymnasium | Standard environment interface | Supplies the environments SB3 trains on |
| PyTorch | Tensor computation and autograd | Backend for all SB3 networks |
| Ray RLlib | Distributed reinforcement learning | Alternative for multi-node and multi-agent scale |
| RL Baselines3 Zoo | Training scripts and tuned hyperparameters | Companion framework built on SB3 |
| SBX | JAX implementations of SB3 algorithms | Faster variant with a near-identical API |
| TensorBoard | Experiment visualization | Default target of SB3 logging |
The practical consequence is that SB3 is the right choice for single-machine experiments and for teaching, benchmarking and prototyping. Teams that need thousands of parallel workers or complex multi-agent setups usually move to a distributed framework.
Configuring and tuning Stable-Baselines3 models
Default hyperparameters in Stable-Baselines3 are reasonable starting points, but reinforcement learning is highly sensitive to hyperparameters, and good settings vary strongly between environments. The RL Baselines3 Zoo publishes tuned configurations for many benchmark tasks and integrates Optuna for automated search.
The parameters in the table have the largest influence on results in most projects.
| Parameter | Applies to | Typical range | Effect |
|---|---|---|---|
| learning_rate | All algorithms | 1e-5 to 1e-3 | Step size of gradient updates |
| n_steps | PPO, A2C | 5 to 4096 per environment | Rollout length before each update |
| gamma | All algorithms | 0.9 to 0.9999 | Weight of future rewards |
| buffer_size | SAC, TD3, DQN | 10,000 to 1,000,000 | Amount of stored past experience |
| ent_coef | PPO, A2C, SAC | 0 to 0.01 or automatic | Strength of exploration pressure |
| clip_range | PPO | 0.1 to 0.3 | Limit on policy change per update |
Observation and reward normalization through the VecNormalize wrapper often matters as much as the algorithm choice for continuous control. Its running statistics must be saved together with the model, because a policy evaluated on unnormalized inputs will behave unpredictably.
Limitations and common mistakes
Stable-Baselines3 removes implementation risk, but it cannot remove the inherent instability of reinforcement learning. Results can vary widely between random seeds, so a single successful run proves little.
The most frequent problems and their fixes are summarized below.
| Symptom | Likely cause | Fix |
|---|---|---|
| Reward never improves | Poorly shaped or sparse reward | Redesign the reward or add HER |
| Training crashes on start | Environment violates the Gymnasium API | Run the built-in environment checker |
| Good training, poor evaluation | Missing normalization statistics | Save and reload VecNormalize |
| Results differ between runs | Seed sensitivity | Report several seeds with variance |
| GPU slower than CPU | Small MLP policies | Train MLP policies on the CPU |
- SB3 is not designed for multi-agent learning; it requires wrappers or another framework.
- Image-based tasks remain slow on a single machine because environment simulation dominates the cost.
- Unbounded or wrongly scaled action spaces degrade continuous control; rescale actions to [-1, 1].
A non-technical view
Reinforcement learning teaches software by reward and penalty, much like training a dog with treats. The program tries an action, sees what happens and slowly learns which choices lead to good outcomes.
Writing this learning process from scratch is difficult and easy to get wrong. Stable-Baselines3 is a ready-made, carefully tested toolbox of these learning methods. A developer describes the task, picks a method from the toolbox and lets the program practice. The value is trust: the methods behave as their published descriptions say, so failures point to the task design rather than hidden bugs.
A concrete example
A small warehouse automation team wants a simulated robotic arm to push packages onto a conveyor. An engineer wraps the physics simulator as a Gymnasium environment with a 12-value observation vector and 4 continuous joint commands, then runs the environment checker to confirm the interface is valid.
She trains SAC with a one-million-step replay buffer on a single workstation across five random seeds, logging to TensorBoard and saving the best model through EvalCallback. After about two million environment steps, four of the five seeds push packages correctly in over 90 percent of evaluation episodes, while one seed stalls.
Because the algorithm itself is a trusted baseline, the team investigates the reward instead of the code and finds that the stalled seed exploited a contact bonus. After the reward is corrected, all five seeds converge, and the policy moves on to tests on the physical arm.