Stable-Baselines3

Also known as: SB3

On this page 7

Stable-Baselines3 (SB3) is an open-source Python library of reliable reinforcement learning algorithms built on PyTorch, offering implementations such as PPO, SAC, TD3 and DQN behind one consistent training interface. Among AI frameworks & tools it occupies a specific niche: it does not build neural networks or serve language models, but trains agents that learn by trial and error in simulated or real environments.

The library is the PyTorch successor to the original Stable Baselines, which was itself a fork of OpenAI Baselines. Its design goal is reproducibility over novelty: each algorithm is benchmarked against published results, covered by unit tests and documented, so that practitioners can treat it as a trusted baseline rather than research code that may silently diverge.

How Stable-Baselines3 structures reinforcement learning

Stable-Baselines3 organizes every experiment around three objects: an environment, a policy and an algorithm. The environment follows the Gymnasium API (reset, step, observation space, action space), the policy is a PyTorch network that maps observations to actions, and the algorithm decides how collected experience updates that network.

A typical run needs only a few lines: create the environment, instantiate a model such as PPO("MlpPolicy", env), call model.learn(total_timesteps=...) and then model.save(). Behind this short interface the library handles rollout collection, advantage estimation, replay buffers, gradient updates and logging, which are the parts where hand-written implementations most often contain subtle bugs.

The following table lists the main building blocks and the role each one plays in a training run.

ComponentExampleRole in training
EnvironmentGymnasium CartPole-v1Supplies observations, rewards and episode termination
Vectorized environmentDummyVecEnv, SubprocVecEnvRuns several environment copies in parallel
PolicyMlpPolicy, CnnPolicy, MultiInputPolicyNeural network that selects actions
AlgorithmPPO, SAC, DQNDefines how experience updates the policy
CallbackEvalCallback, CheckpointCallbackEvaluates, saves or stops training at set intervals
LoggerTensorBoard, CSV, stdoutRecords rewards, losses and episode lengths
Core components of a Stable-Baselines3 training run

Algorithms included in Stable-Baselines3

The core package deliberately contains a small set of well-established algorithms, split into on-policy and off-policy methods. On-policy algorithms such as PPO and A2C learn only from data generated by the current policy, which makes them stable but sample-hungry. Off-policy algorithms such as SAC, TD3 and DQN reuse past experience from a replay buffer, which usually improves sample efficiency at the cost of more tuning.

The table compares the main algorithms by the action spaces they support and their typical use.

AlgorithmAction spacePolicy typeTypical use
PPODiscrete and continuousOn-policyRobust default for most tasks
A2CDiscrete and continuousOn-policyFast, simple synchronous training
DQNDiscrete onlyOff-policyGames and discrete control
SACContinuous onlyOff-policyRobotics and continuous control
TD3Continuous onlyOff-policyDeterministic continuous control
DDPGContinuous onlyOff-policyHistorical baseline, mostly replaced by TD3
Main algorithms and where they fit

Hindsight Experience Replay is available as a replay buffer that works with the off-policy algorithms for goal-conditioned tasks. Experimental or less common methods, including RecurrentPPO, MaskablePPO, TQC, QR-DQN, TRPO and ARS, live in the companion package SB3-Contrib, which keeps the core library small and stable.

Stable-Baselines3 is a training library, not an environment library or a general deep learning framework. It sits on top of PyTorch and consumes environments written against Gymnasium, while larger systems such as Ray RLlib target distributed training at cluster scale.

The comparison below clarifies how SB3 relates to neighboring tools that practitioners often combine with it.

ToolPrimary purposeRelationship to SB3
GymnasiumStandard environment interfaceSupplies the environments SB3 trains on
PyTorchTensor computation and autogradBackend for all SB3 networks
Ray RLlibDistributed reinforcement learningAlternative for multi-node and multi-agent scale
RL Baselines3 ZooTraining scripts and tuned hyperparametersCompanion framework built on SB3
SBXJAX implementations of SB3 algorithmsFaster variant with a near-identical API
TensorBoardExperiment visualizationDefault target of SB3 logging
How SB3 relates to neighboring tools

The practical consequence is that SB3 is the right choice for single-machine experiments and for teaching, benchmarking and prototyping. Teams that need thousands of parallel workers or complex multi-agent setups usually move to a distributed framework.

Configuring and tuning Stable-Baselines3 models

Default hyperparameters in Stable-Baselines3 are reasonable starting points, but reinforcement learning is highly sensitive to hyperparameters, and good settings vary strongly between environments. The RL Baselines3 Zoo publishes tuned configurations for many benchmark tasks and integrates Optuna for automated search.

The parameters in the table have the largest influence on results in most projects.

ParameterApplies toTypical rangeEffect
learning_rateAll algorithms1e-5 to 1e-3Step size of gradient updates
n_stepsPPO, A2C5 to 4096 per environmentRollout length before each update
gammaAll algorithms0.9 to 0.9999Weight of future rewards
buffer_sizeSAC, TD3, DQN10,000 to 1,000,000Amount of stored past experience
ent_coefPPO, A2C, SAC0 to 0.01 or automaticStrength of exploration pressure
clip_rangePPO0.1 to 0.3Limit on policy change per update
Influential hyperparameters and their effect

Observation and reward normalization through the VecNormalize wrapper often matters as much as the algorithm choice for continuous control. Its running statistics must be saved together with the model, because a policy evaluated on unnormalized inputs will behave unpredictably.

Limitations and common mistakes

Stable-Baselines3 removes implementation risk, but it cannot remove the inherent instability of reinforcement learning. Results can vary widely between random seeds, so a single successful run proves little.

The most frequent problems and their fixes are summarized below.

SymptomLikely causeFix
Reward never improvesPoorly shaped or sparse rewardRedesign the reward or add HER
Training crashes on startEnvironment violates the Gymnasium APIRun the built-in environment checker
Good training, poor evaluationMissing normalization statisticsSave and reload VecNormalize
Results differ between runsSeed sensitivityReport several seeds with variance
GPU slower than CPUSmall MLP policiesTrain MLP policies on the CPU
Frequent problems when training with SB3
  • SB3 is not designed for multi-agent learning; it requires wrappers or another framework.
  • Image-based tasks remain slow on a single machine because environment simulation dominates the cost.
  • Unbounded or wrongly scaled action spaces degrade continuous control; rescale actions to [-1, 1].

A non-technical view

Reinforcement learning teaches software by reward and penalty, much like training a dog with treats. The program tries an action, sees what happens and slowly learns which choices lead to good outcomes.

Writing this learning process from scratch is difficult and easy to get wrong. Stable-Baselines3 is a ready-made, carefully tested toolbox of these learning methods. A developer describes the task, picks a method from the toolbox and lets the program practice. The value is trust: the methods behave as their published descriptions say, so failures point to the task design rather than hidden bugs.

A concrete example

A small warehouse automation team wants a simulated robotic arm to push packages onto a conveyor. An engineer wraps the physics simulator as a Gymnasium environment with a 12-value observation vector and 4 continuous joint commands, then runs the environment checker to confirm the interface is valid.

She trains SAC with a one-million-step replay buffer on a single workstation across five random seeds, logging to TensorBoard and saving the best model through EvalCallback. After about two million environment steps, four of the five seeds push packages correctly in over 90 percent of evaluation episodes, while one seed stalls.

Because the algorithm itself is a trusted baseline, the team investigates the reward instead of the code and finds that the stalled seed exploited a contact bonus. After the reward is corrected, all five seeds converge, and the policy moves on to tests on the physical arm.