Model merging is the technique of combining the parameters of two or more trained neural networks into a single model by arithmetic on their weights, without additional gradient-based training on the combined data. In artificial intelligence, it has become a standard step of the training pipeline for open language models, because it lets teams join separately fine-tuned skills at the cost of a few minutes of computation instead of a new training run.
Merging works reliably only when the source models share an architecture and a common pretrained ancestor. Under that condition, the differences between fine-tuned checkpoints behave like roughly additive edits to the base model, which makes averaging, adding and pruning those edits a practical alternative to multi-task fine-tuning.
How model merging combines weights
Model merging operates directly on parameter tensors: for each layer, the merged weight is computed from the corresponding weights of the source models. The simplest form is linear weight averaging, in which every parameter of the result equals a weighted mean of the same parameter across checkpoints.
The method succeeds because fine-tuning from a shared base usually moves the weights only a short distance. Checkpoints that start from the same pretrained model tend to lie in one connected low-loss basin, a property called linear mode connectivity, so points on the straight line between them also perform well. Two networks trained from different random initializations do not share this property, and averaging them typically produces a model that performs close to chance.
Many modern methods express each fine-tuned model as a task vector, the element-wise difference between its weights and the base weights. Merging then means adding scaled task vectors back onto the base, which is why this family of methods is often described as task arithmetic.
The following table lists the inputs that every merge configuration must specify.
| Input | Role | Typical setting |
|---|---|---|
| Base model | Reference point for task vectors | The shared pretrained checkpoint |
| Source models | Checkpoints that contribute skills | 2 to 5 fine-tuned variants |
| Weights or scaling factor | Strength of each contribution | 0.3 to 1.0 per model |
| Density | Share of task-vector entries kept | 0.2 to 0.7 for pruning methods |
| Merge method | Rule that combines the tensors | Linear, SLERP, TIES or DARE |
Main merging methods and their mechanics
Merging methods differ mainly in how they handle interference, the situation in which two task vectors push the same parameter in opposite directions. Simple averaging dilutes every skill, while newer methods prune or resolve conflicting entries before adding them.
| Method | Mechanism | Strength | Weakness |
|---|---|---|---|
| Linear averaging | Weighted mean of all parameters | Simple and stable | Dilutes distinct skills |
| Model soup | Average of runs with different hyperparameters | Improves robustness | Needs one shared fine-tuning task |
| SLERP | Spherical interpolation between two models | Preserves weight norms | Limited to two models at a time |
| Task arithmetic | Adds scaled task vectors to the base | Combines several skills | Sensitive to interference |
| TIES-merging | Trims small values, elects a sign, merges agreeing entries | Reduces sign conflicts | Extra density and sign settings |
| DARE | Randomly drops task-vector entries and rescales the rest | Sparse, low interference | Results vary with the random seed |
| Fisher merging | Weights parameters by Fisher information | Respects parameter importance | Requires gradient statistics |
TIES-merging and DARE rest on one empirical observation: most entries of a task vector are small and redundant, and removing 50 to 90 percent of them often leaves task performance nearly unchanged. Pruning before merging therefore reduces collisions between models without discarding the changes that carry each skill.
A separate family, sometimes called layer stacking or passthrough merging, concatenates layers from different models to build a deeper network. These results are less predictable than weight averaging and usually need further training to recover quality.
Model merging compared with related training techniques
Model merging competes with other ways to combine knowledge, and the right choice depends on compute budget and data access. Merging needs no training data at all, which is its decisive advantage when the original datasets are private or lost.
| Approach | Needs training data | Inference cost | Typical use |
|---|---|---|---|
| Model merging | No | One model | Combining fine-tuned variants cheaply |
| Multi-task fine-tuning | Yes, all tasks | One model | Best quality when data is available |
| Ensembling | No | Several models | Accuracy gains at higher serving cost |
| Distillation | Yes, transfer set | One smaller model | Compressing a teacher into a student |
| LoRA adapter switching | No at merge time | Base plus adapter | Serving many tasks from one base |
Unlike an ensemble, a merged model costs the same to serve as a single model, since only one set of weights runs at inference. Unlike multi-task fine-tuning, merging gives no guarantee of optimality, so a merged model usually trails a jointly trained model by a few points on each task.
Where model merging is used in practice
Model merging appears at several stages of post-training, from combining skill-specific checkpoints to stabilizing the final release. Frontier and open model developers have described merging checkpoints from different runs or data mixtures as part of their recipes.
- Skill combination: joining a coding fine-tune and a math fine-tune of the same base into one assistant.
- Checkpoint averaging: averaging the last several checkpoints of one run to obtain smoother, better-generalizing weights.
- Recovering forgotten abilities: interpolating a fine-tuned model back toward its base to reduce catastrophic forgetting.
- Merging LoRA adapters: folding low-rank updates into full weights or combining several adapters.
- Evolutionary merging: searching merge weights automatically against a validation score.
The table below gives a practical decision guide for choosing a method.
| Situation | Recommended method | Reason |
|---|---|---|
| Two similar fine-tunes | SLERP or linear | Low conflict, smooth result |
| Several runs of one task | Model soup | Averaging improves robustness |
| Three or more distinct skills | TIES or DARE | Pruning limits interference |
| Fine-tune lost general ability | Interpolation toward base | Restores base behavior partially |
| Unclear best weights | Grid or evolutionary search | Validation guides the coefficients |
Limitations and common mistakes in model merging
The most frequent failure is merging incompatible models. Checkpoints must share the same architecture, tokenizer and vocabulary, and ideally the same base checkpoint; models with different tokenizers cannot be merged by weight arithmetic without additional alignment work.
Merged models also need full evaluation before release. A merge can score well on the benchmarks used to tune it while quietly losing safety behavior, instruction following or long-context accuracy. Public leaderboards have seen heavily merged models whose scores reflect benchmark contamination inherited from a source model rather than genuine ability.
| Symptom | Likely cause | Fix |
|---|---|---|
| Output is incoherent | Sources lack a shared base | Merge only descendants of one checkpoint |
| One skill disappears | Weight too low or sign conflicts | Raise its weight or use TIES |
| Repetitive or broken text | Scaling factor too high | Lower the scaling coefficient |
| Safety refusals vanish | Task vectors override alignment | Include the aligned model and re-test |
| Scores fall after quantization | Merged weights have outliers | Evaluate the quantized model separately |
Licensing is a practical constraint as well: a merged model inherits the license terms of every source, so one restrictive component can limit the use of the whole result.
The idea in brief
Imagine one student who took two separate summer courses, one in chemistry and one in history. Model merging is like blending what both courses changed in that student into a single person who knows both subjects, without sending anyone back to school.
It only works when every version started from the same original model. The key point is simple: merging is cheap and fast, but the result must always be tested, because the blend can lose something along the way.
A practical scenario
A small research team maintains an open 7-billion-parameter language model. One engineer has fine-tuned it for Python coding, another for customer-support dialogue, and both variants started from the same base checkpoint. Running a joint fine-tune would require combining two datasets, one of which is under a restrictive data agreement.
The team instead applies TIES-merging with a density of 0.5 and equal weights, then evaluates the result on a coding benchmark, a dialogue test set and a safety suite. The first merge loses some safety refusals, so the team adds the original aligned chat model as a third source with a weight of 0.3.
The final merged model keeps about 95 percent of each specialist’s score, passes the safety suite and runs as a single model. The whole process takes under an hour on one GPU, compared with days for a new training run.