Federated averaging (FedAvg) is the core optimization algorithm of federated learning: a central server sends a global model to many client devices, each client trains it for several steps on its local data, and the server combines the returned weights into a weighted average that becomes the next global model. In Edge AI, it is the standard way to improve a model with data that stays on phones, PCs and embedded devices.
The algorithm was introduced in 2016 as a generalization of federated stochastic gradient descent (FedSGD). Its central insight is that letting each device run multiple local epochs before communicating cuts the number of network rounds by one or two orders of magnitude, which matters because communication, not computation, is the bottleneck on mobile and wireless links.
How federated averaging works round by round
Federated averaging proceeds in synchronous communication rounds, each of which turns many local training runs into one global update. A round follows a fixed sequence of selection, broadcast, local training, upload and aggregation.
- The server selects a fraction C of eligible clients, typically devices that are idle, charging and on an unmetered network.
- Each selected client downloads the current global weights.
- Each client runs E local epochs of stochastic gradient descent over its data in minibatches of size B.
- Each client uploads its new weights or the difference from the starting weights, called the model delta.
- The server computes the average of the updates, weighted by each client’s number of samples, and applies it to the global model.
Weighting by sample count makes the result an unbiased estimate of training on the union of all client data, provided the clients were sampled fairly. In formula terms, the new weights equal the sum over clients of n_k divided by n times w_k, where n_k is the sample count of client k and n is the total across participating clients.
The table below lists the three hyperparameters that define the algorithm and how each one shifts the balance between communication cost and model quality.
| Parameter | Meaning | Typical range | Effect of increasing it |
|---|---|---|---|
| Client fraction C | Share of clients selected per round | 0.01 to 0.1 of eligible devices | Lower variance, higher server load |
| Local epochs E | Passes over local data per round | 1 to 5 | Fewer rounds, more client drift |
| Local batch size B | Minibatch size on the device | 10 to 64 | Faster steps, fewer updates per epoch |
| Server learning rate | Scale applied to the averaged delta | Often 1.0 in plain FedAvg | Faster progress, risk of oscillation |
Federated averaging compared with related training methods
FedAvg differs from its neighbors mainly in how often devices communicate and what they send. FedSGD is the special case with one local step and the full local dataset as a single batch, so FedAvg with E equal to 1 and B equal to the dataset size reduces to it.
The comparison below places FedAvg next to methods that are frequently confused with it in edge deployments, including split learning and on-device training.
| Method | What leaves the device | Communication frequency | Main use |
|---|---|---|---|
| FedSGD | One gradient per round | Every local step | Baseline, simple analysis |
| FedAvg | Weights or model delta | After several local epochs | Cross-device federated learning |
| FedProx | Weights with proximal regularization | After local epochs | Highly heterogeneous clients |
| Split learning | Intermediate activations | Every forward and backward pass | Very small devices |
| On-device training | Nothing by default | None | Purely local personalization |
Unlike data-parallel training in a data center, FedAvg cannot assume identical data distributions, reliable workers or fast interconnects. That difference explains most of its specific failure modes and most of the variants built on top of it.
Non-IID data and client drift
The main weakness of federated averaging is sensitivity to non-IID data, meaning data that is not independent and identically distributed across clients. One user types mostly in Spanish, another photographs only pets, and a sensor sees one climate; each local model is pulled toward its own local optimum.
This effect is called client drift. The more local epochs a client runs, the further its weights move away from the global objective, and the average of drifted models can converge slowly, to a worse optimum, or oscillate. Practitioners counter it with fewer local epochs, learning rate decay and drift-correcting variants.
| Symptom | Likely cause | Common remedy |
|---|---|---|
| Accuracy plateaus below centralized baseline | Strongly non-IID data | Reduce E, use FedProx or SCAFFOLD |
| Loss oscillates between rounds | Server step too large | Add server momentum or adaptive optimizer |
| Model favors frequent device types | Biased client sampling | Stratify selection, reweight updates |
| Rounds stall waiting for uploads | Stragglers on slow links | Over-select clients, drop late ones |
| Batch normalization degrades | Local statistics differ | Use group or layer normalization |
Adaptive server optimizers such as FedAdam and FedYogi treat the averaged delta as a pseudo-gradient and apply momentum or per-parameter scaling to it. They often reach a target accuracy in noticeably fewer rounds than plain averaging, at the cost of extra server-side hyperparameters.
Privacy and security properties
Federated averaging keeps raw data on the device, but model updates can still leak information. Gradient inversion and membership inference attacks have reconstructed training samples or confirmed their presence from individual updates, especially with small batches and large models.
Production systems therefore combine FedAvg with secure aggregation, a cryptographic protocol that lets the server see only the sum of many updates, and with differential privacy, which clips each update and adds calibrated noise. The privacy budget, expressed as epsilon, is spent across rounds, so the number of rounds becomes a privacy parameter as well as a quality parameter.
- Update clipping bounds the influence of any single client on the global model.
- Secure aggregation hides individual updates but requires a minimum number of participants per round.
- Robust aggregation rules, such as coordinate-wise median or trimmed mean, limit damage from poisoned updates but weaken the averaging guarantees.
Deploying federated averaging on edge devices
Running FedAvg in production is an engineering problem as much as an algorithmic one. Devices must train within tight memory, battery and thermal limits, and only a small fraction is available at any moment, which favors compact models and compressed updates.
The decision table below summarizes how practical constraints change the recommended configuration.
| Situation | Recommended choice | Reason |
|---|---|---|
| Millions of phones, few online at once | Small C, many rounds | Availability is low and uneven |
| Metered or slow uplink | Quantized or sparse deltas | Upload size dominates cost |
| Highly personal data | Fine-tune locally after FedAvg | Global model underfits individuals |
| Few reliable institutional clients | Larger E, all clients each round | Cross-silo setting tolerates heavier rounds |
| Sensitive data with legal limits | Differential privacy with secure aggregation | Updates alone are not anonymous |
Update compression through 8-bit quantization, sparsification or low-rank deltas can shrink each upload by roughly 4 to 100 times, depending on the method and the tolerated accuracy loss. Evaluation is also harder than in centralized training, because the true test distribution is spread across devices and is usually measured with federated evaluation rounds.
Explained without jargon
Imagine a class of students who each study a few pages of a textbook at home. Nobody hands in their notebook. Instead, each student writes down how their understanding changed, and the teacher combines everyone’s corrections into one improved summary that goes back to the whole class.
Federated averaging works the same way for AI models. Your phone learns a little from your own typing or photos, sends back only the changes to the model, and a server averages the changes from many phones. The result is a smarter shared model without collecting personal data in one place.
A typical use case
A company that builds a mobile keyboard wants better next-word suggestions for regional slang without uploading what people type. It deploys a small language model of about 1.4 million parameters and runs federated averaging only on phones that are charging, idle and on Wi-Fi.
Each round selects several hundred devices, and each device trains for one local epoch on its recent typing history. Updates are clipped, protected by secure aggregation and noised for differential privacy. After about 3,000 rounds over several weeks, the new model is tested with federated evaluation and shows a measurable gain in prediction accuracy over the server-trained baseline, while no typed text has left any phone.