Value function

Also known as: state-value function

On this page 7

A value function in reinforcement learning is a function that assigns to each state the expected cumulative discounted reward an agent will obtain from that state onward while following a particular policy. The state-value function, usually written V(s) or V_pi(s), turns a stream of future rewards into a single number that tells the agent how good it is to be in a state.

Value functions are the backbone of most reinforcement learning algorithms. They let an agent compare situations without simulating the whole future, they appear in the Bellman equation that links the value of a state to the values of its successors, and they serve as the critic in actor-critic methods.

How a value function is defined

Formally, for a Markov decision process and a fixed policy pi, the state-value function is V_pi(s) = E_pi[G_t | S_t = s], where the return G_t is the sum of future rewards weighted by powers of the discount factor gamma: G_t = R_{t+1} + gamma * R_{t+2} + gamma^2 * R_{t+3} + .... The expectation runs over both the randomness of the policy and the randomness of the environment dynamics.

The value is therefore a property of a state and a policy together. The same state can be valuable under a skilled policy and nearly worthless under a random one. The optimal value function V*(s) is the highest value achievable from a state by any policy, and any policy that acts greedily with respect to V* together with a model of the environment is optimal.

The value function satisfies a recursive relation known as the Bellman expectation equation: the value of a state equals the expected immediate reward plus the discounted value of the next state. This recursive structure is what makes efficient learning possible, because an estimate can be improved by looking only one step ahead.

Types of value functions

Reinforcement learning uses several closely related value functions, each answering a slightly different question. The table below compares the main forms.

FunctionNotationQuestion it answers
State-value functionV_pi(s)How good is this state under the policy
Action-value functionQ_pi(s, a)How good is this action in this state
Advantage functionA_pi(s, a) = Q - VHow much better is this action than average
Optimal state valueV*(s)Best achievable return from this state
Optimal action valueQ*(s, a)Best return after taking this action
Main forms of value functions

The key practical difference between V and Q lies in control. Selecting an action from V alone requires a model of transitions and rewards, because the agent must predict which state each action leads to. The Q-function stores action values directly, which is why model-free control methods such as Q-learning and SARSA learn Q rather than V.

How value functions are estimated

Agents rarely know the true value function and must estimate it from experience or compute it from a model. Estimation methods differ in how far they look ahead before updating, which determines their bias-variance trade-off.

MethodUpdate targetNeeds a modelBias and variance
Dynamic programmingFull expected backupYesNo sampling noise, exact model required
Monte Carlo methodsComplete episode returnNoUnbiased, high variance
TD(0) learningReward plus next-state estimateNoBiased, low variance
TD(lambda) and n-stepMix of short and long returnsNoTunable between the two
Estimation methods for value functions

Temporal difference learning is the most influential idea here. After each step the agent computes the TD error, delta = r + gamma * V(s') - V(s), and moves V(s) a small step toward the target. This bootstrapping reuses existing estimates, so learning proceeds online without waiting for episodes to end.

In small problems the value function is stored as a lookup table with one entry per state. Large or continuous state spaces require function approximation, typically a linear model over features or a deep neural network, which generalizes across similar states but can also introduce instability.

Role in reinforcement learning algorithms

Value functions appear in nearly every family of reinforcement learning algorithms, though in different roles. The following table shows where each family uses them.

Algorithm familyValue function usedRole
Value iteration and policy iterationV or Q with a modelComputes the optimal policy directly
Q-learning and deep Q-networkQDrives greedy action selection
Actor-critic and PPOV as the criticReduces variance of policy gradients
Generalized advantage estimationVBuilds advantage estimates from TD errors
Monte Carlo tree searchLearned V at leaf nodesEvaluates positions without full rollouts
Value functions across algorithm families

In policy gradient methods, the value function is a baseline subtracted from the return. It does not change the expected gradient, but it sharply reduces its variance, which is the main reason actor-critic methods learn faster than plain REINFORCE.

Limitations and common pitfalls

Learned value functions are approximations, and their errors can mislead the agent in systematic ways. Practitioners watch for several well-known failure modes.

  • Overestimation bias arises when the agent takes a maximum over noisy action values; Double DQN and TD3 were designed to counter it.
  • The deadly triad of function approximation, bootstrapping and off-policy learning can make estimates diverge instead of converge.
  • A discount factor set too low makes the agent short-sighted, while a value close to 1 slows learning and increases variance.
  • Sparse rewards leave most states with nearly identical values, so the function offers little guidance.
  • Distribution shift in offline reinforcement learning causes overly optimistic values for actions absent from the dataset.

A further subtlety is that a value function is only as meaningful as its reward function. If the reward is misspecified, an accurate value function faithfully guides the agent toward unintended behavior, a pattern often discussed as reward hacking.

In plain words

A value function is like an experienced player’s sense of a position in a board game. Looking at the board, the player cannot see every future move, yet can still say whether the position looks promising or dangerous.

For a learning machine, the value function is that sense expressed as a number. It summarizes all the rewards likely to come from a situation, so the agent can steer toward good situations and away from bad ones without planning every step to the end.

A real-world example

A warehouse operator trains a reinforcement learning agent to route autonomous carts between picking stations. Each delivery earns a reward of plus 1, every second of travel costs minus 0.01, and collisions cost minus 10. The engineers set the discount factor to 0.99, which corresponds to an effective planning horizon of roughly 100 steps.

The team uses proximal policy optimization, in which a neural network critic learns the state-value function from simulated shifts. Early in training, the critic assigns similar values to all aisles. After several million simulated steps, it gives clearly lower values to congested intersections near packing stations during peak hours.

Those value estimates feed the advantage calculations that update the routing policy. The resulting carts learn to avoid predictable bottlenecks before they form, and in simulation the operator measures shorter average delivery times and fewer near-collisions than with the previous rule-based router.