Model merging

On this page 7

Model merging is the technique of combining the parameters of two or more trained neural networks into a single model by arithmetic on their weights, without additional gradient-based training on the combined data. In artificial intelligence, it has become a standard step of the training pipeline for open language models, because it lets teams join separately fine-tuned skills at the cost of a few minutes of computation instead of a new training run.

Merging works reliably only when the source models share an architecture and a common pretrained ancestor. Under that condition, the differences between fine-tuned checkpoints behave like roughly additive edits to the base model, which makes averaging, adding and pruning those edits a practical alternative to multi-task fine-tuning.

How model merging combines weights

Model merging operates directly on parameter tensors: for each layer, the merged weight is computed from the corresponding weights of the source models. The simplest form is linear weight averaging, in which every parameter of the result equals a weighted mean of the same parameter across checkpoints.

The method succeeds because fine-tuning from a shared base usually moves the weights only a short distance. Checkpoints that start from the same pretrained model tend to lie in one connected low-loss basin, a property called linear mode connectivity, so points on the straight line between them also perform well. Two networks trained from different random initializations do not share this property, and averaging them typically produces a model that performs close to chance.

Many modern methods express each fine-tuned model as a task vector, the element-wise difference between its weights and the base weights. Merging then means adding scaled task vectors back onto the base, which is why this family of methods is often described as task arithmetic.

The following table lists the inputs that every merge configuration must specify.

InputRoleTypical setting
Base modelReference point for task vectorsThe shared pretrained checkpoint
Source modelsCheckpoints that contribute skills2 to 5 fine-tuned variants
Weights or scaling factorStrength of each contribution0.3 to 1.0 per model
DensityShare of task-vector entries kept0.2 to 0.7 for pruning methods
Merge methodRule that combines the tensorsLinear, SLERP, TIES or DARE
Core inputs of a merge configuration

Main merging methods and their mechanics

Merging methods differ mainly in how they handle interference, the situation in which two task vectors push the same parameter in opposite directions. Simple averaging dilutes every skill, while newer methods prune or resolve conflicting entries before adding them.

MethodMechanismStrengthWeakness
Linear averagingWeighted mean of all parametersSimple and stableDilutes distinct skills
Model soupAverage of runs with different hyperparametersImproves robustnessNeeds one shared fine-tuning task
SLERPSpherical interpolation between two modelsPreserves weight normsLimited to two models at a time
Task arithmeticAdds scaled task vectors to the baseCombines several skillsSensitive to interference
TIES-mergingTrims small values, elects a sign, merges agreeing entriesReduces sign conflictsExtra density and sign settings
DARERandomly drops task-vector entries and rescales the restSparse, low interferenceResults vary with the random seed
Fisher mergingWeights parameters by Fisher informationRespects parameter importanceRequires gradient statistics
Widely used model merging methods

TIES-merging and DARE rest on one empirical observation: most entries of a task vector are small and redundant, and removing 50 to 90 percent of them often leaves task performance nearly unchanged. Pruning before merging therefore reduces collisions between models without discarding the changes that carry each skill.

A separate family, sometimes called layer stacking or passthrough merging, concatenates layers from different models to build a deeper network. These results are less predictable than weight averaging and usually need further training to recover quality.

Model merging competes with other ways to combine knowledge, and the right choice depends on compute budget and data access. Merging needs no training data at all, which is its decisive advantage when the original datasets are private or lost.

ApproachNeeds training dataInference costTypical use
Model mergingNoOne modelCombining fine-tuned variants cheaply
Multi-task fine-tuningYes, all tasksOne modelBest quality when data is available
EnsemblingNoSeveral modelsAccuracy gains at higher serving cost
DistillationYes, transfer setOne smaller modelCompressing a teacher into a student
LoRA adapter switchingNo at merge timeBase plus adapterServing many tasks from one base
Merging versus neighboring approaches

Unlike an ensemble, a merged model costs the same to serve as a single model, since only one set of weights runs at inference. Unlike multi-task fine-tuning, merging gives no guarantee of optimality, so a merged model usually trails a jointly trained model by a few points on each task.

Where model merging is used in practice

Model merging appears at several stages of post-training, from combining skill-specific checkpoints to stabilizing the final release. Frontier and open model developers have described merging checkpoints from different runs or data mixtures as part of their recipes.

  • Skill combination: joining a coding fine-tune and a math fine-tune of the same base into one assistant.
  • Checkpoint averaging: averaging the last several checkpoints of one run to obtain smoother, better-generalizing weights.
  • Recovering forgotten abilities: interpolating a fine-tuned model back toward its base to reduce catastrophic forgetting.
  • Merging LoRA adapters: folding low-rank updates into full weights or combining several adapters.
  • Evolutionary merging: searching merge weights automatically against a validation score.

The table below gives a practical decision guide for choosing a method.

SituationRecommended methodReason
Two similar fine-tunesSLERP or linearLow conflict, smooth result
Several runs of one taskModel soupAveraging improves robustness
Three or more distinct skillsTIES or DAREPruning limits interference
Fine-tune lost general abilityInterpolation toward baseRestores base behavior partially
Unclear best weightsGrid or evolutionary searchValidation guides the coefficients
Choosing a merge strategy

Limitations and common mistakes in model merging

The most frequent failure is merging incompatible models. Checkpoints must share the same architecture, tokenizer and vocabulary, and ideally the same base checkpoint; models with different tokenizers cannot be merged by weight arithmetic without additional alignment work.

Merged models also need full evaluation before release. A merge can score well on the benchmarks used to tune it while quietly losing safety behavior, instruction following or long-context accuracy. Public leaderboards have seen heavily merged models whose scores reflect benchmark contamination inherited from a source model rather than genuine ability.

SymptomLikely causeFix
Output is incoherentSources lack a shared baseMerge only descendants of one checkpoint
One skill disappearsWeight too low or sign conflictsRaise its weight or use TIES
Repetitive or broken textScaling factor too highLower the scaling coefficient
Safety refusals vanishTask vectors override alignmentInclude the aligned model and re-test
Scores fall after quantizationMerged weights have outliersEvaluate the quantized model separately
Symptoms, causes and fixes after a merge

Licensing is a practical constraint as well: a merged model inherits the license terms of every source, so one restrictive component can limit the use of the whole result.

The idea in brief

Imagine one student who took two separate summer courses, one in chemistry and one in history. Model merging is like blending what both courses changed in that student into a single person who knows both subjects, without sending anyone back to school.

It only works when every version started from the same original model. The key point is simple: merging is cheap and fast, but the result must always be tested, because the blend can lose something along the way.

A practical scenario

A small research team maintains an open 7-billion-parameter language model. One engineer has fine-tuned it for Python coding, another for customer-support dialogue, and both variants started from the same base checkpoint. Running a joint fine-tune would require combining two datasets, one of which is under a restrictive data agreement.

The team instead applies TIES-merging with a density of 0.5 and equal weights, then evaluates the result on a coding benchmark, a dialogue test set and a safety suite. The first merge loses some safety refusals, so the team adds the original aligned chat model as a third source with a weight of 0.3.

The final merged model keeps about 95 percent of each specialist’s score, passes the safety suite and runs as a single model. The whole process takes under an hour on one GPU, compared with days for a new training run.