How agents learn by trial and error from rewards: environments, policies, value functions and the main algorithms.
55 terms
A3C, asynchronous advantage actor-critic, is a reinforcement learning algorithm in which parallel workers update a shared policy and value network.
The action space in reinforcement learning is the set of all actions an agent may choose from in a given state, either discrete or continuous.
Actor-critic is a reinforcement learning architecture in which a policy network acts while a value network evaluates actions to reduce gradient variance.
The advantage function in reinforcement learning measures how much better one action is than the policy's average behavior in a given state.
The Bellman equation expresses the value of a state in reinforcement learning as immediate reward plus the discounted value of the states that follow.
A contextual bandit is a reinforcement learning setting in which an agent sees context, picks one action and learns only from that action's reward.
The credit assignment problem in reinforcement learning is deciding which earlier actions deserve credit or blame for a reward that arrives later.
Cumulative reward is the total, often discounted, sum of rewards an agent collects over time, the quantity that reinforcement learning tries to maximize.
Deep deterministic policy gradient (DDPG) is an off-policy actor-critic algorithm that learns deterministic policies for continuous control tasks.
A Deep Q-network (DQN) is a neural network that approximates the Q-function, letting a reinforcement learning agent choose actions from raw, complex data.
Deep reinforcement learning combines trial-and-error reward learning with deep neural networks so agents can act well in large, complex environments.
The discount factor in reinforcement learning is a number between 0 and 1 that sets how much an agent values future rewards compared with immediate ones.
Double DQN is a deep reinforcement learning algorithm that reduces Q-value overestimation by splitting action selection from action evaluation.
An episode in reinforcement learning is one complete run of agent-environment interaction, from an initial state to a terminal state or a time limit.
Epsilon-greedy is an exploration strategy in reinforcement learning that picks a random action with probability ε and the best-known action otherwise.
Experience replay is a reinforcement learning technique that stores past transitions in a buffer and reuses random samples of them to train an agent.
The exploration-exploitation tradeoff is the reinforcement learning dilemma of choosing between trying new actions and repeating the best known ones.
Generalized advantage estimation (GAE) computes policy-gradient advantages as a weighted sum of TD errors, trading bias against variance via lambda.
Group relative policy optimization (GRPO) is a reinforcement learning method that scores groups of sampled answers against each other instead of using a critic.
Hierarchical reinforcement learning splits a long task into subgoals and temporally extended skills, so an agent plans at several levels of abstraction.
Intrinsic motivation in reinforcement learning is a self-generated reward, such as novelty or prediction error, that drives an agent to explore.
Inverse reinforcement learning (IRL) infers the reward function an expert is optimizing from observed behavior, so an agent can learn the underlying goal.
A Markov decision process (MDP) is the formal model of sequential decisions under uncertainty that underlies nearly every reinforcement learning algorithm.
Model-based reinforcement learning trains an agent that learns or uses a model of its environment to plan actions and improve its policy with less real data.
Model-free reinforcement learning trains an agent to act from sampled rewards alone, without learning a model of environment dynamics.
Monte Carlo methods in reinforcement learning estimate value functions by averaging the actual returns observed over complete episodes of experience.
Monte Carlo tree search (MCTS) is a planning algorithm that grows a search tree from simulated outcomes to choose strong actions in sequential decisions.
Multi-agent reinforcement learning (MARL) studies how several agents learn policies by trial and error in a shared environment where their actions interact.
Off-policy learning lets a reinforcement learning agent evaluate or improve one policy using experience generated by a different behavior policy.
Offline reinforcement learning trains a decision policy entirely from a fixed dataset of logged interactions, without collecting any new experience.
On-policy learning is a reinforcement learning approach in which an agent improves the same policy that it uses to collect its experience.
A partially observable Markov decision process (POMDP) models sequential decisions when an agent cannot see the true state and must act on noisy observations.
A policy in reinforcement learning is the rule that maps each state an agent observes to an action or a probability distribution over actions.
A policy gradient method in reinforcement learning improves a parameterized policy by gradient ascent on expected return, estimated from sampled episodes.
Policy iteration is a reinforcement learning algorithm that alternates policy evaluation and greedy improvement until it reaches an optimal policy.
Proximal policy optimization (PPO) is a reinforcement learning algorithm that improves a policy with clipped, conservative updates for stable training.
The Q-function in reinforcement learning gives the expected discounted return of taking an action in a state and then following a given policy.
Q-learning is a model-free reinforcement learning algorithm that learns the value of each action in each state and derives an optimal policy from it.
REINFORCE is the Monte Carlo policy gradient algorithm in reinforcement learning that raises the probability of actions followed by high returns.
A reward function in reinforcement learning maps states and actions to scalar feedback that defines the goal an agent learns to maximize over time.
Reward shaping adds extra reward signals to a reinforcement learning task so an agent learns faster without changing which behavior is optimal.
An RL environment is the simulated or real world a reinforcement learning agent acts in, returning observations, rewards and episode signals.
Safe reinforcement learning trains agents to maximize reward while respecting explicit safety constraints during both learning and deployment.
SARSA is an on-policy temporal difference algorithm in reinforcement learning that learns action values from the actions an agent actually takes.
Self-play is a reinforcement learning method in which an agent improves by competing against copies of itself instead of fixed opponents or human data.
Soft actor-critic (SAC) is an off-policy reinforcement learning algorithm that maximizes reward plus policy entropy for stable continuous control.
A sparse reward is a reinforcement learning signal that is zero for most steps and nonzero only at rare events such as reaching a goal.
In reinforcement learning, the state space is the set of all situations an agent can encounter, which shapes how its policy and values are learned.
TD3 is an off-policy reinforcement learning algorithm for continuous control that fixes the overestimation bias of DDPG with twin critics and delayed updates.
Temporal difference learning is a reinforcement learning method that updates value estimates from the gap between successive predictions.