Multi-agent reinforcement learning

Also known as: MARL

On this page 7

Multi-agent reinforcement learning (MARL) is the branch of reinforcement learning in which two or more agents learn simultaneously in a shared environment, each choosing actions from its own observations while the outcome depends on the joint behavior of all agents. Within artificial intelligence, MARL extends the single-agent framework of rewards, policies and value functions to settings such as robot teams, traffic control, trading and competitive games.

The defining difficulty is that every agent is part of every other agent’s environment. As the others learn, the world each agent faces keeps changing, so convergence guarantees from single-agent learning no longer hold in general.

How multi-agent reinforcement learning is formalized

MARL is usually modeled as a Markov game, also called a stochastic game, which generalizes the Markov decision process to many agents. At each step every agent selects an action, the joint action determines the next state, and each agent receives its own reward. When agents see only part of the state, the model becomes a decentralized partially observable Markov decision process (Dec-POMDP) for cooperative tasks or a partially observable stochastic game in general.

The solution concept also changes. Instead of a single optimal policy, MARL seeks equilibria, most often a Nash equilibrium, in which no agent can improve its expected return by changing its policy alone. Many games have several equilibria with very different payoffs, so which one learning reaches depends on initialization, exploration and the algorithm.

The table below summarizes the main formal components and how they differ from the single-agent case.

ComponentSingle-agent versionMulti-agent version
StateEnvironment stateShared state, often partially observed per agent
ActionOne action per stepJoint action of all agents
RewardOne reward signalIndividual or shared team reward
PolicyMaps states to actionsOne policy per agent, possibly shared parameters
SolutionOptimal policyEquilibrium or best team policy
Components of a multi-agent learning problem

Cooperative, competitive and mixed settings

MARL problems are classified by how the agents’ rewards relate. In fully cooperative tasks all agents share one reward, in fully competitive tasks one agent’s gain is another’s loss, and mixed or general-sum tasks combine shared and conflicting interests, as in negotiation or traffic.

SettingReward relationshipTypical exampleMain challenge
Fully cooperativeIdentical team rewardWarehouse robot fleetsMulti-agent credit assignment
Fully competitiveZero-sum rewardsTwo-player board gamesExploitable strategies
Mixed motivePartly aligned rewardsAutonomous driving at intersectionsBalancing cooperation and self-interest
Team versus teamCooperative inside, competitive acrossMultiplayer strategy gamesCoordination under adversarial pressure
Reward structures in multi-agent reinforcement learning

In competitive settings, self-play and population-based training are standard tools: agents train against copies or archived versions of themselves so that strategies improve without a fixed opponent. In cooperative settings, the central problem is deciding which agent caused a team reward, an extension of the credit assignment problem.

Training architectures and key algorithms

The most influential design choice in MARL is where information is available during training and during execution. The dominant paradigm is centralized training with decentralized execution (CTDE): during training a critic or mixing network may see all observations and actions, while at deployment each agent acts only on its local observation.

ParadigmTraining informationExecution informationRepresentative methods
Independent learnersLocal onlyLocal onlyIndependent Q-learning, independent PPO
Centralized training, decentralized executionGlobal state and joint actionsLocal onlyMADDPG, MAPPO, QMIX, COMA
Fully centralizedGlobalGlobalSingle controller over the joint action
Communication learningLocal plus learned messagesLocal plus messagesCommNet, differentiable communication channels
Main training paradigms in MARL

Value decomposition methods address cooperative tasks. VDN represents the team value as a sum of per-agent values, and QMIX combines them with a monotonic mixing network so that each agent’s greedy action is consistent with the best joint action. Actor-critic methods such as MADDPG and multi-agent PPO use a centralized critic to reduce the variance caused by other agents’ changing behavior.

Fully centralized control is rarely practical, because the joint action space grows exponentially with the number of agents. Ten agents with five actions each already produce nearly ten million joint actions.

Core challenges of learning with many agents

MARL inherits every difficulty of single-agent reinforcement learning and adds several of its own. The most fundamental is non-stationarity: from any one agent’s perspective, the transition and reward dynamics change as other agents update their policies, which undermines experience replay and value estimates.

ProblemCauseCommon mitigation
Non-stationarityOther agents keep changing their policiesCentralized critics, opponent modeling
ScalabilityExponential joint action spaceParameter sharing, mean-field approximations
Credit assignmentShared reward hides individual contributionCounterfactual baselines, value decomposition
Equilibrium selectionSeveral stable but unequal outcomesCoordination signals, communication, reward shaping
Partial observabilityAgents see only local informationRecurrent policies, learned communication
Characteristic problems in multi-agent learning

Evaluation is also harder than in the single-agent case. A policy that defeats one opponent may lose badly to another, so competitive agents are judged against diverse populations rather than a single benchmark score.

Limitations and common misconceptions

  • More agents do not mean faster learning: sample requirements typically grow with the number of interacting agents.
  • Independent learners are not always inadequate: with parameter sharing they are strong baselines on many cooperative benchmarks.
  • Simulation success rarely transfers directly: real robots and vehicles face communication delays, sensor noise and unmodeled agents.
  • Emergent behavior can be undesirable: agents may discover collusion or exploit simulator flaws instead of solving the intended task.
  • Safety guarantees are limited, so deployment in physical systems usually requires constraints, monitoring and human oversight.

Practitioners therefore start with the simplest architecture that captures the interaction, add centralized information only when independent learning fails, and test against varied partners and opponents before drawing conclusions.

In simple terms

Ordinary reinforcement learning teaches one learner by reward and trial and error, much like training a single dog with treats. Multi-agent reinforcement learning teaches a whole group at once, and every member’s success depends on what the others do.

Think of a youth soccer team learning to play together. Each player improves, but the team changes as they all improve, so yesterday’s good move may not work today. The hard part is learning while everyone else is learning too, and figuring out who deserves credit when the team scores.

A practical example

A logistics operator runs a fulfillment center where 24 mobile robots carry shelving units to packing stations. Fixed routing rules cause congestion at busy aisles, so the engineering team trains the robots with a cooperative MARL approach in a simulator of the building layout.

Each robot observes only its local surroundings and nearby robots, while a centralized critic sees the full warehouse state during training. The robots share policy parameters, and the team reward combines delivered orders per hour with a penalty for collisions and blocked aisles. After several million simulated steps, the learned policies yield to each other at intersections and spread traffic across parallel aisles.

Before deployment, the team tests the policies with randomly disabled robots and unexpected human workers in the simulation. In a two-week pilot, throughput rises by roughly 12 percent compared with the rule-based router, and a safety layer that stops robots near people remains active at all times.