What actually happens if every large language model is merged into one

Ask a room of engineers what would happen if you combined every large language model on earth into one system, and you get two reactions. The first is excitement: a single model that inherits the coding strength of one system, the reasoning of another, the multilingual reach of a third, and the raw knowledge of them all. The second is a wince, because the people who have actually tried to merge models know the result is rarely the sum of the parts. More often it is a muddle that does each thing slightly worse than the specialist it was built from.

The question hidden inside a deceptively simple idea

The idea sounds like straightforward addition. The reality is closer to mixing paint. Combine enough distinct colours and you do not get every colour at once. You get brown. The core surprise of model combination is that averaging strong systems tends to produce something mediocre rather than something superhuman, and the reasons for that reach into the mathematics of how these networks store what they know.

There is also a definitional trap in the question. “Combining all models” can mean at least three technically different things, and they lead to completely different answers. It can mean literally fusing the numerical weights of separate networks into one set of parameters. It can mean building a system that keeps the models intact and orchestrates them, sending each request to whichever model handles it best. Or it can mean the corporate reading, where every lab that trains these systems merges into a single company that controls the frontier. Each version is worth taking seriously, because each is already happening in a partial form somewhere in the industry.

The most literal version, weight fusion, is a real and active research field. Techniques with names like model soup, task arithmetic, TIES, and DARE let practitioners blend the parameters of related models, and merged models have topped open leaderboards. But every one of these methods carries a quiet precondition that most casual versions of the question ignore: the models being merged usually have to share the same architecture and often the same starting point. That single constraint is enough to rule out the fantasy of stirring GPT, Claude, Gemini, Llama, and DeepSeek into one pot.

The orchestration version is where the practical action sits in 2026. Routers, ensembles, and mixture-of-agents systems combine the outputs of many models without ever touching their internals, and these approaches are now production infrastructure at scale. They deliver much of what people actually want from combination, cost control and higher answer quality, without the mathematical impossibility of fusing incompatible networks.

The corporate version is the one regulators lose sleep over. A market where a handful of firms already control the compute, the data pipelines, and the most capable systems does not need a formal merger to concentrate power. The consolidation is happening quietly through talent deals and licensing structures designed to slip past merger review.

This article takes all three readings apart in detail: what merging weights actually requires, why heterogeneous models resist fusion, how mixture of experts and routing achieve a smarter kind of combination, what a literal corporate merger would mean, and what the honest answer is to the question of whether one combined model would be smarter than the best model we already have. The short version is that the interesting outcomes come not from fusion but from orchestration, and the frightening outcomes come not from technology but from concentration.

Three very different things people mean by combining models

Precision matters here because the same sentence can describe three unrelated engineering realities. Separating them early prevents most of the confusion that surrounds the topic.

Weight-space merging treats a trained model as a giant list of numbers, its parameters, and asks what happens when you mathematically blend those numbers across several models. The simplest form averages the weights element by element. More refined forms selectively keep, trim, or rescale parameters to reduce conflict. This is the field that produced model soups, spherical interpolation, task arithmetic, TIES-merging, and DARE, and the open-source toolkit MergeKit put it within reach of anyone with a laptop and a Hugging Face account. Weight merging is cheap because it never runs the model during the merge itself; it only manipulates parameters. That efficiency is the whole appeal, and it is also why the technique is so tightly constrained.

Output-space combination leaves every model intact and combines what they produce. A router picks the best model for each query. An ensemble sends a prompt to several models and fuses or votes on their answers. A mixture-of-agents system stacks models in layers, letting later models refine the drafts of earlier ones. None of these methods share a single parameter across models. They treat each model as a sealed box and coordinate the boxes. This is the version that scales across vendors, because it does not care whether the models inside share an architecture, a tokenizer, or even a company.

Organizational combination is the business-and-policy reading. It asks what happens if the companies training frontier models stop competing and consolidate, whether through acquisition, licensing, joint safety bodies, or simple market gravity. The technical questions here dissolve into questions about market power, innovation, safety coordination, and the risk of a single point of failure for a technology the economy increasingly depends on.

These three readings do not blur into one another; they lead to opposite conclusions. Weight merging across the whole industry is close to impossible for technical reasons. Output combination across the whole industry is not only possible but already routine. Organizational combination is possible, partly underway, and the one with the largest stakes for competition and safety.

A fourth, subtler mechanism sits between weight merging and organizational combination: knowledge distillation. Distillation trains a new model to imitate the behaviour of an existing one, transferring capability without copying a single weight directly. It is how a small model can inherit much of a large model’s skill, and it is also how one lab can absorb the capabilities of a rival’s system by learning from its outputs. Distillation is a form of combination through imitation rather than fusion, and it has become a flashpoint in export-control and intellectual-property disputes precisely because it lets capability leak across the boundaries that weight merging cannot cross.

Keeping these four mechanisms distinct, weight merging, output combination, distillation, and organizational consolidation, is the single most useful habit for thinking clearly about the question. Most breathless predictions about a combined super-model quietly assume weight merging works across incompatible systems, which it does not, while ignoring the orchestration methods that actually deliver combined capability every day. The rest of this analysis follows that structure, starting with the current shape of the frontier, because the answer to “what if we combined them all” depends heavily on what “all” currently contains.

The frontier as it actually stands in 2026

Any answer to the combination question has to start with an inventory, because the set of models being combined is not a tidy list of five names. By mid-2026 the frontier is crowded and no single system wins everything. The old framing of a two-horse race between OpenAI and Google no longer describes reality. At least five proprietary labs field frontier-class systems that trade blows benchmark by benchmark, and a strong open-weight ecosystem sits close behind.

On the proprietary side, the leading families in 2026 include OpenAI’s GPT-5 line, Anthropic’s Claude Opus and Sonnet models, Google’s Gemini 3 series, and xAI’s Grok 4 family. Independent leaderboards through the first half of 2026 showed these systems clustered within a few points of one another on the benchmarks that discriminate hardest, with the lead changing hands as new versions shipped. On graduate-level science reasoning, several systems reached the low-to-mid nineties on GPQA Diamond. On real-world software engineering, seven models landed within roughly three points of each other on SWE-bench Verified, a spread so tight that the evaluation scaffold often matters more than the underlying model.

The open-weight side has closed most of the gap that once separated it from the proprietary frontier. Meta’s Llama 4 family, DeepSeek’s V3 and V4 systems, Alibaba’s Qwen line, Moonshot’s Kimi models, Z.AI’s GLM series, and MiniMax’s M2 family now compete directly with closed systems on developer benchmarks, often at a fraction of the API cost. DeepSeek’s V4 Pro landed within a point of the best proprietary coders on SWE-bench Verified while costing dramatically less per output token. The practical consequence is that “all available models” in 2026 numbers in the hundreds, not the handful. One tracker logged 255 model releases from major organizations in the first quarter of 2026 alone.

Three structural facts about this field shape everything that follows.

No model dominates across tasks. One system leads on coding-agent workflows, another on abstract reasoning, another on long-context multimodal work, another on real-time information through live data access. A model can top a math benchmark and write flat prose, or lead on science questions and trail on production coding. This spread is exactly what makes combination tempting, and exactly why naive averaging destroys it: the strengths live in different regions of each model’s parameters, and blending them dilutes rather than concentrates.

The models are architecturally diverse. Nearly every frontier system now uses some form of sparse mixture of experts, but the specifics differ wildly, expert counts, routing strategies, attention mechanisms, context lengths, and above all tokenizers. Two systems that both call themselves transformers can be as internally different as two engines that both burn fuel. That diversity is the technical wall that stops cross-family weight merging.

Capability is increasingly a moving target. With flagships updating every few weeks and open-weight models chasing them within months, any snapshot of “the best combination” is stale almost immediately. This is why the winning production strategy in 2026 is not to pick one model or to merge them, but to stay model-agnostic and route by task, so that each new flagship becomes an upgrade rather than a migration.

The picture, then, is a broad field of strong, distinct, fast-moving systems, none supreme, most mutually incompatible at the parameter level. That is the raw material anyone hoping to build a single combined model would be working with, and it explains why the literal fusion of the whole field is a non-starter before a single line of merging code is written.

Weight merging and what it really requires

To understand why fusing the whole frontier fails, it helps to understand when weight merging actually works, because it genuinely does work under the right conditions. Merged models have reached top rankings on competitive leaderboards, and strategic combination has produced systems that outperform the individual fine-tuned variants they were built from. The field went from a niche trick to a mainstream component of model development in about two years.

Weight merging rests on a geometric intuition about how neural networks store what they learn. A trained model corresponds to a point in an enormous space with one dimension per parameter, often billions of them. Training finds a point in a low-loss region of that space, a valley in the loss surface where the model performs well. The central discovery behind merging is that models fine-tuned from the same starting point tend to land in the same broad valley, connected by paths along which performance stays high. This property, studied under the name linear mode connectivity, means you can move between two such models by averaging their weights and stay in good territory the whole way.

That single condition, a shared origin, is doing enormous work. When two models start from the same pre-trained base and are then fine-tuned on different tasks, their parameters remain aligned. Parameter number 4,000,000,001 means roughly the same thing in both models, so averaging them combines two compatible adjustments to the same underlying representation. The merge inherits both fine-tunings because both were edits to a shared foundation.

Remove the shared origin and the alignment vanishes. Two models trained independently from different random initializations, on different data, with different architectures, place their learned features in different coordinates. The same concept might live in one set of neurons in one model and a completely different set in the other. Averaging their weights then combines unrelated things, like overlaying two photographs taken from different angles and expecting a sharper image. What you get is blur.

This is why the practical literature on merging is almost entirely about combining models within a family. The canonical demonstrations merge fine-tuned variants of the same base, several Llama-3-8B derivatives, a set of Qwen fine-tunes, checkpoints from the same pre-training run. The open leaderboards that merged models climbed were dominated by same-family blends. Even the more advanced techniques that follow, which reduce interference and let you merge more models at once, still assume this compatibility. They are tools for cleaning up conflicts between aligned parameters, not for bridging models that were never aligned to begin with.

There is a second, quieter requirement: the models must be roughly the same size and shape. Averaging assumes a one-to-one correspondence between parameters. Two models with different layer counts, different hidden dimensions, or different numbers of attention heads have no such correspondence. You cannot average a 70-billion-parameter dense model with a 671-billion-parameter sparse one, because there is no coherent way to line up their parameters at all.

Merging is a precision instrument for a narrow job: combining related models cheaply, without retraining, to recycle the effort spent on separate fine-tunes. It is genuinely useful, and it is nothing like the universal blender the combination question imagines. Understanding the specific techniques makes the constraint concrete, so the next sections walk through them in turn.

Model soup and the averaging baseline

The plainest merging method is also the one that named the field. Model soup, introduced in 2022, does exactly what it sounds like: it averages the weights of several fine-tuned models to produce a single set of parameters. No retraining, no gating, no cleverness. You take the numbers, add them up, divide, and ship the result. The surprise was that this crude operation often improved accuracy without adding any inference cost, because the averaged model kept the strengths of its ingredients while smoothing out some of their individual quirks.

Two variants exist. A uniform soup averages every model equally. A greedy soup adds models one at a time, keeping each new ingredient only if it improves validation accuracy, so the final blend contains just the models that actually help. The greedy version tends to win, because it quietly discards ingredients that would drag the mixture down, which is itself an early hint of the interference problem that haunts merging.

Model soup works because of the shared-basin property described earlier. When all the ingredients were fine-tuned from one base, they sit in the same low-loss valley, and their average sits there too. The method has stayed a reliable baseline precisely because it is so simple; more sophisticated techniques are usually measured against how much they beat plain averaging.

Its weakness is equally simple. Averaging treats every parameter as equally worth keeping, which means it has no way to resolve conflicts. If one model pushed a parameter strongly positive to encode one behaviour, and another pushed the same parameter strongly negative to encode a different one, the average lands near zero and encodes neither. Unresolved parameter conflicts are the fundamental reason naive averaging degrades performance once the ingredients diverge, and they are the specific problem every later method tries to solve. Model soup is the honest baseline: it shows both that merging can help and exactly where it breaks.

The soup metaphor also captures the ceiling. A soup made from good ingredients is decent, but it does not taste better than the best single dish those ingredients could have made separately. Averaging pulls toward a competent middle. That is fine when the goal is a strong generalist assembled from near-identical fine-tunes. It is useless when the goal is to stack distinct peak capabilities into one system, because peaks are exactly what averaging erodes.

Spherical interpolation and the two-model ceiling

Simple averaging moves along a straight line between two models. Spherical linear interpolation, universally abbreviated SLERP, moves along a curved path instead, following the surface of a sphere rather than cutting through its interior. The reason is geometric. In the high-dimensional space where model weights live, the information that matters often sits in the direction of the weight vectors more than their raw magnitude. A straight-line average can shrink those magnitudes and land in a region that neither original model would recognize. SLERP preserves the geometric relationship between the two models by interpolating along the arc that connects them, which tends to keep the blend in healthy territory.

In practice SLERP is one of the most widely used merging methods, and many of the strongest community merges on open leaderboards were built with it. It works well and is well understood. Toolkits expose it with a small number of knobs: an interpolation factor that decides how much of each model to keep, sometimes varied layer by layer so that early layers lean toward one model and later layers toward another.

SLERP carries a hard structural limitation that matters enormously for the combination question: it can only merge two models at a time. The spherical path is defined between two points. There is no clean single-step spherical interpolation among fifty models. You can chain SLERP, merging two, then merging the result with a third, and so on, but each step compounds the loss from the previous one, and the order of merging changes the outcome. Chaining is not the same as combining everything at once, and the drift accumulates.

This two-model ceiling is why SLERP alone cannot answer the fantasy version of the question. Combining every available model is inherently a many-model problem, and the method best suited to careful pairwise blending is structurally the wrong tool for it. The methods that follow, task arithmetic, TIES, and DARE, were developed in part to break past this ceiling and merge many models in one operation, which is why they became the backbone of large-scale merging.

Normalized variants such as NuSLERP refine the technique further, adjusting how the vectors are scaled before interpolation, but they inherit the same core character: a precise, geometry-aware way to blend a pair of compatible models, prized for quality on the two-model case and quietly unsuited to the crowd.

Task arithmetic and the idea of a task vector

The most conceptually elegant idea in the merging literature is that a model’s skills can be treated like vectors you can add and subtract. Task arithmetic starts by defining a task vector: take a model fine-tuned on some task, subtract the weights of the base model it started from, and the difference is the set of parameter changes that fine-tuning produced. That difference vector encodes, in weight space, what the model learned to do the task.

Once skills are expressed as vectors, arithmetic becomes possible. Add a coding task vector and a translation task vector to the same base, and in principle you get a model that can do both. Subtract a task vector, and you can try to remove a behaviour, a technique researchers have used to reduce toxic outputs by negating a “toxicity” vector. Representing fine-tuning as an additive, composable vector is the insight that turned merging from a curiosity into a general method for combining specialized models.

Task arithmetic clarifies exactly why the shared-base requirement is non-negotiable. A task vector has meaning only relative to the base it was subtracted from. Two task vectors can be added coherently only if they are edits to the same base, expressed in the same coordinate system. Vectors computed against different bases are simply not addable in any coherent sense; the operation produces numbers, but the numbers do not correspond to any coherent model. This is the algebraic form of the compatibility constraint that runs through the whole field.

The method also exposes the interference problem in a precise way. When you add several task vectors, they overlap. Two tasks might want to change the same parameter in opposite directions. The naive sum lets these conflicts partially cancel, so each added task degrades the others. The more tasks you stack, the worse the mutual interference, and performance on every individual task drifts downward from what a dedicated fine-tune would achieve. This is the mathematical core of the pull toward the mean: adding many task vectors does not accumulate many peak skills; it accumulates conflicts that erode all of them.

Task arithmetic is powerful because it makes combination explicit and controllable. You can weight each task vector, scaling how much of each skill enters the blend. You can compose and decompose behaviours. But it does not escape the fundamental tension. Every technique that follows is essentially a smarter way of adding task vectors while suppressing the interference between them. The realization that interference, not capacity, is the binding constraint reframes the entire question. The problem with combining models is not that there is no room for all the skills. It is that the skills fight each other for the same parameters.

That reframing is worth holding onto. The combination fantasy imagines skills as items you drop into a bag, where more items simply means a fuller bag. The reality is closer to a negotiation in which every new participant renegotiates settings the others depend on. TIES and DARE are, at heart, negotiation referees.

Trimming sign conflicts with TIES

TIES-merging tackles interference head-on, and its name spells out its three steps: Trim, Elect Sign, and Merge. The method starts from task vectors, the same difference-from-base representation task arithmetic uses, and then cleans them up before combining, rather than blindly summing them.

The first step, trimming, recognizes that most of the parameter changes in a task vector are small and unimportant. Fine-tuning nudges billions of parameters, but only a fraction carry the real signal for the new task. TIES keeps the parameters with the largest magnitude changes and zeroes out the rest. This immediately reduces interference, because two task vectors that have been trimmed to their essentials overlap far less than two dense ones. Most of the potential conflicts simply disappear when the trivial changes are dropped.

The second step, electing a sign, resolves the conflicts that remain. For each parameter, TIES looks across all the task vectors being merged and asks which direction the majority of them want to move it, weighted by magnitude. It then keeps only the contributions that agree with that elected direction and discards the ones pulling the opposite way. This is the decisive move. Instead of letting a positive push and a negative push cancel into a useless average, TIES picks a winner and preserves a decisive value. Resolving sign conflicts by majority vote is what lets TIES merge many models without collapsing every parameter toward zero.

The third step, merging, averages only the surviving, sign-aligned values. Because the conflicts have been trimmed and resolved first, the final average encodes coherent skills rather than smeared compromises.

The payoff is that TIES scales to more models than pairwise methods like SLERP. By systematically suppressing interference, it can combine many fine-tuned variants while retaining more of each one’s capability than naive averaging would. It became a workhorse of the merging community for exactly this reason, and it appears constantly in the recipes behind high-ranking merged models.

Its limits still trace back to the same source. TIES reduces interference; it does not eliminate the underlying incompatibility. It still assumes the task vectors are edits to a shared base in a shared coordinate system, so it still cannot bridge models from different families. And the sign election, while clever, is a heuristic. When genuinely different capabilities want opposite parameter values for good reasons, forcing a single elected sign discards real information from the losing models. TIES makes many-model merging viable within a family; it does not make it lossless, and it does not make it universal.

Dropping and rescaling parameters through DARE

DARE takes the trimming intuition to an extreme and adds a mathematical correction that makes the extremity safe. Its name stands for Drop And REscale, and the method has a striking headline result: task vectors can survive having 90 percent, and in some cases 99 percent, of their parameter changes deleted at random, as long as the survivors are rescaled to compensate.

The mechanism has two parts. First, DARE randomly drops a large fraction of the delta parameters, the changes relative to the base, setting them to zero. This is more aggressive than TIES, which keeps the largest-magnitude changes deliberately; DARE prunes at random. Second, it rescales the surviving parameters upward by a factor that accounts for how many were dropped. If you keep only a tenth of the changes, you multiply them to preserve the total expected contribution. The rescaling is what keeps the sparsified task vector faithful to the original despite the massive deletion.

The reason this works reveals something about how fine-tuning stores information. Fine-tuning is highly redundant. The changes it makes are spread thinly across enormous numbers of parameters, so most individual changes are near-duplicates of the signal carried by others. Deleting most of them at random loses little, because the surviving changes still carry the core adjustment. The redundancy of fine-tuning is what makes extreme sparsification survivable, and DARE exploits that redundancy to shrink each task vector to a sliver of its original density before merging.

The practical value for combination is direct. Sparser task vectors overlap far less, so they interfere far less when added. Two task vectors that each retain only a few percent of their changes are unlikely to fight over the same parameters, simply because they touch so few of the same ones. DARE therefore composes naturally with the other methods. The common recipe DARE-TIES first sparsifies each task vector with DARE, then resolves the remaining sign conflicts with TIES, combining aggressive pruning with principled conflict resolution. This pairing is one of the most reliable ways to merge several fine-tuned models at once.

Even DARE, though, lives inside the family constraint. Random dropping and rescaling operate on delta parameters defined relative to a shared base. The method assumes the deltas are edits to the same foundation, expressed in the same coordinates. It reduces interference dramatically; it does not create compatibility where none exists. Applied across different architectures with different bases, there are no coherent deltas to drop and rescale in the first place.

DARE completes the core toolkit. Model soup gives the baseline, SLERP gives geometric care for pairs, task arithmetic gives the composable vector framework, TIES gives conflict resolution, and DARE gives extreme sparsification. Every one of them is a variation on the same theme: combine aligned parameter edits while fighting the interference that combination creates. And every one shares the same wall. They are extraordinary tools for recycling and blending related models, and they are structurally incapable of fusing the incompatible systems that make up the actual frontier.

Evolutionary merging and the move toward automation

The methods so far all demand manual choices. How much to trim, what dropout rate to use, which interpolation factor, which layers lean toward which model, these knobs multiply fast, and tuning them by hand is slow guesswork. A newer line of work automates the search, treating the merge configuration itself as something to search rather than to guess.

Evolutionary approaches borrow from biology. They generate a population of candidate merge recipes, evaluate how well each merged model performs, keep the best, and breed variations of them across many generations. The search can operate in two spaces at once: the parameter space, deciding how to blend weights, and the data-flow space, deciding how to route information through layers drawn from different models. Letting a search discover merge recipes has produced combinations that outperform hand-designed merges and, in some cases, the individual source models on the target tasks. The appeal is that the algorithm can find non-obvious blends a human would never try, including recipes that mix models across different domains.

Other automation efforts make the merge itself differentiable, so the blending coefficients can be learned by gradient descent rather than searched blindly. Differentiable adaptive merging learns how much weight to give each model per layer or per component, tuning the recipe with a small amount of data instead of a brute-force sweep. Related ideas learn to combine a bank of expert parameters on the fly, instantiating a specialist at test time without retraining, and some work shows performance can keep improving by extrapolating beyond the source models in weight space rather than merely interpolating between them.

These advances matter because they push merging from a craft toward a pipeline. Automated search makes it feasible to combine more models and to squeeze more capability out of a given set of ingredients. Toolkits have folded many of these strategies into a single interface, and surveys now catalogue dozens of distinct merging strategies, from spherical interpolation to Fisher-weighted averaging to evolutionary search.

Automation does not repeal the compatibility law. Evolutionary and differentiable methods search more cleverly over the space of merges, but they still search within the space that alignment allows. When the source models share a base and an architecture, a search can find excellent recipes. When they do not, the search runs over a space where no good merge exists, and no amount of clever search invents one. Automation improves how well we combine compatible models; it does not extend combination to incompatible ones.

The trajectory is nonetheless important for the larger question. It suggests that within families, the ceiling on merged capability is still rising, and that the manual craft of the early merging era is giving way to systematic search. If a future arrives in which the whole frontier converges on a shared architecture and tokenizer, these automated methods are what would make broad merging practical. Until then, they sharpen a tool that remains bounded by the two constraints the next sections examine directly: the tokenizer and the architecture.

The tokenizer problem that stops cross-family merges

Before a language model sees text, it sees numbers. A tokenizer chops raw text into units, tokens, and assigns each a numerical identity. The model’s very first layer, the embedding table, maps each token id to a vector, and everything the model does downstream operates on those vectors. Two models with different tokenizers do not even agree on what the input is, and that disagreement alone makes weight merging between them incoherent before any deeper architectural issue arises.

Consider what a tokenizer does concretely. One model might split an uncommon word into three sub-word pieces; another might have that whole word as a single token; a third might break it differently again. The word “tokenization” could be one token in one vocabulary and four in another. Each tokenizer has its own vocabulary size, its own set of tokens, and its own ordering. Token id 5,000 might mean one fragment in DeepSeek’s vocabulary and something entirely unrelated in Llama’s.

The embedding table encodes this. If model A has a 100,000-token vocabulary and model B has a 150,000-token vocabulary, their embedding tables are different shapes, and there is no correspondence between row 5,000 in one and row 5,000 in the other. Averaging embedding tables across models with different tokenizers means averaging vectors that represent different things. The result is meaningless. This is not a subtle degradation; it is a category error, like averaging the pages of two books that use different alphabets and expecting readable text.

Tokenizer mismatch cascades through the entire network. Because every internal representation is built from token embeddings, the meaning of a given neuron’s activation depends on the tokenization that produced it. Two models trained with different tokenizers develop internal representations tuned to their own input scheme. Even if every other part of their architecture were identical, their weights would be organized around incompatible foundations. The features are in different coordinates because the inputs were in different coordinates.

This is a large part of why the merging literature stays within families. Models in the same family, all the Llama derivatives, all the Qwen fine-tunes, share a tokenizer by construction, because they descend from a common base. Their embedding tables line up. Their internal representations are built on the same tokenization. That shared foundation is precisely what makes their weights mergeable. Cross-family merging throws that foundation away.

Vocabulary alignment across models with different tokenizers is one of the hardest open problems in model fusion, and the partial solutions that exist, mapping tokens between vocabularies, re-embedding one model’s vocabulary into another’s space, are lossy and complex, closer to translation than to merging. They are active research, not a solved capability. Some cross-tokenizer fusion methods work at the level of outputs or logits rather than weights, precisely because the weight-level correspondence does not exist.

The tokenizer problem is worth dwelling on because it is so easy to overlook. People picture models as interchangeable brains that merely need to be wired together. But two models with different tokenizers are not two dialects of one language; they are two languages with no shared dictionary. You cannot average your way from one to the other. And since the frontier’s models use a range of different tokenizers, this single issue already rules out the literal fusion of the whole field, independent of every other incompatibility. The architecture problem, examined next, only deepens the wall.

Architecture mismatch and the limits of averaging weights

Even if two models somehow shared a tokenizer, weight merging would still require them to share a shape. Averaging parameters assumes a one-to-one map: parameter here corresponds to parameter there. Frontier models violate that assumption in almost every dimension that matters.

Start with depth and width. Models differ in the number of layers, the size of the hidden dimension, the number of attention heads, and the dimension of each head. A model with 80 layers and a model with 64 layers have no natural way to align their layers. A model with a hidden size of 8,192 and one with 12,288 have weight matrices of different shapes that cannot be added at all. There is no arithmetic that averages a matrix of one size with a matrix of another. The operation is undefined, not merely inadvisable.

Then there is the dense-versus-sparse divide, which by 2026 splits the field cleanly. Nearly every frontier model now uses a sparse mixture-of-experts architecture, but a dense model and a sparse model are structurally incomparable. A dense model has one feed-forward network per layer that processes every token. A sparse model has dozens or hundreds of expert sub-networks per layer and a router that sends each token to a few of them. There is no correspondence between “the feed-forward weights” of a dense model and “the 256 experts” of a sparse one. You cannot average them because there is nothing to average against.

Sparse models differ from each other too. One frontier system might have 256 experts with top-8 routing; another might have a different count with a different number active per token; a third might use a fundamentally different routing scheme or shared-expert design. Their routers are trained to their own expert populations. Transplanting an expert from one into another, or averaging their routers, breaks the delicate balance the router learned. The parts are not modular in the way the combination fantasy assumes.

Attention mechanisms diverge as well. Some models use grouped-query attention, others multi-head latent attention, others sparse attention variants introduced specifically to handle very long contexts. These are different computations, not different settings of the same computation. Their weights are not interchangeable.

The following comparison shows how far apart even a small sample of well-documented systems sit on the dimensions that merging requires to match.

Architectural spread across a sample of documented 2026-era models

These figures come from public model reports and technical write-ups, and they represent a tiny slice of the field, yet no two rows share the total size, the active size, and the expert configuration all at once. Averaging across these rows is not a hard engineering task; it is an undefined operation, because the parameters have no shared coordinate system to be averaged in.

The lesson is blunt. Weight merging is a within-shape, within-tokenizer, within-family technique. The frontier is a collection of different shapes, different tokenizers, and different families. Fusing the weights of every available model is not difficult in 2026; it is mathematically ill-defined, and no advance in merging algorithms changes that, because the problem is the absence of correspondence, not the cleverness of the blend. That leaves the deeper conceptual point, which holds even in the rare cases where merging is possible: the pull toward the mean.

Interference and the quiet pull toward the mean

Suppose the technical obstacles vanished. Imagine every frontier model shared one tokenizer, one architecture, one base, so that weight merging was well-defined across the whole field. Would combining them all produce a system smarter than the best of them? The evidence from within-family merging says no, and understanding why is the most important insight in this whole topic.

Merging averages. Even the sophisticated methods, after all their trimming and sign-electing and rescaling, still blend parameters toward a shared value. Averaging is a smoothing operation, and smoothing removes extremes. The very thing that makes a model exceptional at a task is usually an extreme, a set of parameters pushed hard in a particular direction to encode a sharp capability. When you average many models, you pull those extremes toward a middle, and the middle is competent at everything and exceptional at nothing.

This is why greedy soup exists: it discards ingredients that would drag the blend down, an admission that adding more models often hurts. It is why stacking many task vectors degrades performance on each task through interference. It is why the practical merging community merges a handful of carefully chosen, complementary models rather than throwing in everything available. More ingredients means more conflict, and more conflict means more of every capability sanded off.

The paint analogy from the opening is exact, not decorative. Each model is a pigment with a strong, specific hue. Two or three chosen for how they complement each other can blend into something richer. But keep adding pigments and the mixture marches inexorably toward a muddy grey-brown, because every colour partially cancels the others. There is no combination of enough distinct paints that yields brilliant white or vivid red. The mathematics of averaging forbids it. A merge of everything does not concentrate all the peaks; it averages them into a plateau.

There is a deeper reason rooted in how these models represent knowledge. Two strong models often disagree not because one is wrong, but because they encode the same capability differently, or make different trade-offs. One model might be calibrated to be cautious, another to be creative. Averaging their parameters does not give you a model that is cautious when caution is needed and creative when creativity is needed. It gives you a model that is uniformly, blandly halfway between the two in every situation, which is worse than either in the situations where their distinct dispositions were the right call.

This is the central gap between weight merging and the orchestration methods examined next. Merging forces one blended answer for all inputs. Orchestration keeps the models intact and picks the right one for each input. The first destroys the specialization; the second exploits it. That difference is why the future of combination runs through routers and ensembles rather than through weight fusion.

The pull toward the mean also reframes what “smarter” would even mean for a combined model. Intelligence in these systems is not a single scalar quantity you can pour into one bucket. It is a profile of strengths and weaknesses shaped by architecture, data, and training choices. You cannot add the profiles together and get a taller profile. You can only average them into a flatter one, or, if you are clever, keep them separate and choose among them. The fantasy of combination assumes intelligence is additive. The evidence says it is not, at least not through fusion.

None of this means merging is useless. Within a family, blending complementary fine-tunes produces genuinely stronger generalists, and that is valuable. But the honest answer to “would merging everything make it smarter” is that even in the impossible case where merging everything were technically allowed, it would most likely make it more average. The gains people imagine come from a different mechanism entirely, and that mechanism is a model that combines many experts inside one architecture, which is exactly what mixture of experts already does.

Mixture of experts as combination done properly

The most successful form of model combination in 2026 is not a merge of separate models at all. It is a single model built from the ground up to contain many specialists, and it powers nearly the entire frontier. Mixture of experts, or MoE, is what combination looks like when it is designed in rather than bolted on.

The idea is old, dating to 1991, but the last two years turned it from a research curiosity into the default architecture for frontier and open-weight systems alike. By early 2026, essentially every frontier model, DeepSeek’s V-series, Qwen3, the Llama 4 family, Mistral Large 3, Kimi K2, and the GPT-5 line, relies on sparse mixture of experts. The convergence is close to total.

The mechanism replaces the single feed-forward network at each layer with many smaller sub-networks called experts, plus a small routing network, the gate, that decides which experts handle each token. Only a few experts activate for any given token, so the model uses a fraction of its parameters on each step. The numbers make the point vivid. DeepSeek-V3 stores around 671 billion parameters but activates only about 37 billion per token, roughly five percent of the model doing all the work on each step. With 256 experts and top-8 routing, it runs a small slice of its total expert compute per token while retaining the knowledge capacity of the full parameter count.

This is genuine combination, and it works because it inverts the merging approach. Instead of averaging many models into one blurred set of weights, MoE keeps many specialists distinct inside one model and learns a router that sends each token to the right ones. The specialization is preserved rather than averaged away, and the router exploits it token by token. A token that needs mathematical reasoning can be routed to experts that specialize in it; a token that needs multilingual handling can go elsewhere. The model gets breadth from having many experts and depth from routing to the relevant ones, without the interference that plagues weight merging.

The economics are why the whole industry adopted it. Because only a fraction of parameters activate per token, a sparse model’s compute cost per token is closer to a small dense model than to its own total size. Research showed a trillion-parameter sparse model beating a dense 175-billion-parameter model while using half the inference compute. At the frontier, MoE is not merely cheaper to run; at equal compute it can reach higher quality than a dense architecture, because it can hold far more total capacity than a dense model could train under the same budget. The dramatic fall in frontier-model prices through 2025 and 2026 is in large part an MoE story.

The connection to the combination question is direct and slightly ironic. The thing people imagine when they ask about combining all models, one system with many specialties, already exists, but it is achieved by building the experts together inside one architecture, not by merging separately trained models afterward. The experts in an MoE are trained jointly, so they share a tokenizer, a coordinate system, and a router calibrated to their exact population. That shared foundation is what merging separate frontier models fatally lacks. MoE gets the benefit of combination precisely by avoiding the incompatibility that fusion runs into. It is also, as the next section shows, more fragile than its success suggests.

Expert collapse and the fragility of sparse routing

Mixture of experts is powerful, but it is not a stable machine you can casually rearrange. The router that makes it work is also its weakest point, and the failure modes reveal why even this well-designed form of combination resists the plug-and-play modularity the combination fantasy assumes.

The first problem is load balancing. If the router learns to send most tokens to a few favourite experts, the rest sit idle and the model wastes its capacity. Worse, the favoured experts get overworked and the neglected ones never specialize, so the model effectively shrinks to a fraction of its trained size. Classic MoE designs fight this with an auxiliary loss that penalizes unbalanced routing, and 2026-era designs have moved toward auxiliary-loss-free load balancing supplemented by additional objectives that push the router toward genuine specialization rather than artificial uniformity. Getting this balance right is delicate, and it is tuned jointly with the experts during training.

The second problem is expert collapse, and recent work made its severity concrete. Researchers found that a tiny subset of experts, sometimes just a handful out of thousands, dominate the model’s behaviour by producing the extreme activations that flow between layers. Pruning as few as three experts out of thousands in one open MoE model caused catastrophic collapse, with the model degenerating into repetitive, uninformative output and losing much of its math and reasoning ability. The experts are not equal and interchangeable parts. A small number of them are load-bearing in a way that is not obvious from the outside, and removing them breaks the whole system.

This fragility matters directly for anyone imagining that you could combine models by treating their experts as modular components, harvesting the best experts from many MoE systems and assembling a super-model from them. The experts are not portable. Each one is trained in the context of a specific router, a specific set of sibling experts, and a specific coordinate system. An expert extracted from one model has no meaning inside another, because the router it depended on and the representations it expected are gone. Experts are specialists within a team, not free agents you can trade between teams.

There is a memory reality too. Even though only a fraction of experts activate per token, all of them must sit in memory, because any token might need any expert. A large sparse model can require enormous amounts of GPU memory just to hold its full expert population, even though each forward pass touches only a slice. This is why sparse models are cheap in compute but expensive in memory, and why deploying them demands purpose-built infrastructure for expert parallelism and routing overhead. Combining more experts from more models would multiply this memory burden without any guarantee the combined router could coordinate them.

The deeper point is that MoE’s success comes from tight integration, not loose assembly. The experts, the router, the load-balancing objectives, and the training data are co-designed and co-trained. That integration is exactly what a post-hoc combination of separate models lacks. You cannot get MoE’s benefits by gluing existing models together, because the glue, the jointly trained router, is the hard part, and it only works when it grows up with the experts it coordinates. This is the same lesson merging taught, from a different direction: combination that works is combination that was designed together from the start. The methods that combine already-separate models therefore have to work at a different level entirely, which is where sparse upcycling and output-level orchestration come in.

Sparse upcycling and branch-train-mix

There is a bridge between the world of separate models and the world of integrated mixture of experts, and it is worth understanding because it is the closest thing to a legitimate way of turning existing models into a combined one. The techniques are sparse upcycling and branch-train-mix, and they show both what is possible and what it costs.

Sparse upcycling converts a dense model into a mixture of experts by duplicating its feed-forward networks into several identical experts, then continuing training so the copies diverge and specialize. Instead of training a sparse model from scratch, you take a finished dense model, expand each feed-forward layer into a set of experts initialized as copies of the original, add a router, and keep training. The model starts as a redundant version of its dense self and gradually differentiates its experts as training continues. This recycles the enormous investment in the original dense model rather than throwing it away, and it is a real, used technique for building MoE systems more cheaply.

Branch-train-mix goes further and speaks more directly to the combination question. The recipe branches a base model into several copies, trains each copy separately on a different domain so it becomes a specialist, and then mixes the specialists back together into a single mixture-of-experts model, using the specialists as the experts and training a router to coordinate them. This is combination in the honest sense: several separately trained specialists brought into one model. It works because all the branches start from the same base, so they remain compatible, and because the final mixing step trains a router to coordinate them rather than averaging them.

Both techniques underline the same two conditions that run through this entire analysis. The pieces must share a common origin, and there must be a training step that learns to coordinate them. Sparse upcycling starts from one dense model, so compatibility is guaranteed. Branch-train-mix branches from one base, so the specialists stay aligned, and it invests in training a router rather than hoping an average will do. Neither technique can absorb a model from a different family, because a foreign model shares neither the origin nor the coordinate system, and the coordinating router would have nothing coherent to coordinate.

The existence of these methods is the strongest available answer to “can you combine separately trained models into one.” The answer is a qualified yes, but only when they were branched from a shared base and only with additional training to knit them together. You can combine models you deliberately grew apart from a common root; you cannot combine models that were never related. That distinction, common root plus coordinating training, is the closest the field comes to real fusion, and it is a world away from stirring the whole frontier into one pot. For genuinely unrelated models, the only combination that works keeps them separate and coordinates their outputs, which is the subject of the next several sections.

Ensembles instead of merges

If you cannot fuse the weights of unrelated models, you can still combine what they produce. Output-level combination sidesteps every compatibility problem, because it never touches parameters, tokenizers, or architectures. It treats each model as a sealed box that takes a prompt and returns an answer, and it combines the answers. This is the version of combination that actually scales across the whole frontier, and it is where the real gains from “combining all models” are found.

The simplest ensemble is voting. Send the same prompt to several models and take the answer the majority agree on. For tasks with a checkable answer, a math problem, a multiple-choice question, a piece of code that either passes tests or does not, majority voting reliably beats any single model, because the models make different mistakes and the correct answer tends to be the one they converge on. A closely related single-model technique, self-consistency, samples the same model many times and votes among its own answers, and it improves reasoning accuracy for the same reason: independent attempts cancel out random errors.

More sophisticated ensembles fuse rather than vote. LLM-Blender, an influential design, runs a prompt through several models, then uses a trained component to rank the candidate answers pairwise and a second component to fuse the best ones into a single response that can be better than any individual answer. The insight is that different models are strong on different parts of the same problem, so a good fusion can stitch together the best pieces of several answers. This is combination at the level of language, not weights, and it composes models that could never be merged.

Ensembles have a clear cost: you pay to run every model on every query. Sending one prompt to five models costs roughly five times as much as sending it to one, plus the overhead of fusing the results, and it multiplies latency. That expense is why pure ensembling is reserved for high-value queries where accuracy justifies the cost, and why the field moved quickly toward routing, which tries to get ensemble-like quality while paying for only one model most of the time.

There is also a subtlety about when ensembling helps. Combining several strong but similar models yields little, because they agree and their errors correlate. Ensembles pay off when the models are genuinely diverse, making different mistakes on different inputs, which is exactly the situation the varied 2026 frontier provides. A blend of a coding specialist, a reasoning specialist, and a multilingual specialist has uncorrelated weaknesses, so combining their outputs covers more ground than any one covers alone. Research on mixing different models found that diversity, not raw individual strength, is what makes an ensemble beat its best member, and that naively mixing models can even hurt when they are too similar or when weak members drag down the fusion.

Output combination is the honest realization of the combination dream. It delivers what people actually want, answers that draw on the strengths of many models, without pretending that incompatible networks can be melted into one. It works today, across vendors, across architectures, across tokenizers, because it operates on text, the one interface every model shares. The trade-off is cost and latency, and the entire subsequent evolution of the field, mixture of agents, learned routing, and the economics of when to use which, is about managing that trade-off intelligently.

Mixture of agents and layered collaboration

A richer form of output combination arranges models not side by side but in layers, letting them build on one another’s work the way a team of people drafts and revises a document. This is the mixture-of-agents approach, and it treats models as collaborators in a pipeline rather than as competitors whose answers get merged.

The structure is straightforward. In the first layer, several models each produce an independent answer to the prompt. In the next layer, another set of models receives the original prompt plus all the first-layer answers, and each produces a refined response informed by what the others wrote. Stack several such layers and the answers progressively improve, each layer using the previous layer’s outputs as reference material. A striking early result showed that stacking open models in these layers could outperform a single leading proprietary model, suggesting that collaboration among lesser models can rival raw individual strength.

The mechanism resembles human group work. A first draft is rarely the best possible answer. Seeing several independent drafts, then writing an improved version that incorporates their best ideas and avoids their mistakes, tends to produce something better than any single draft. Models exhibit the same pattern: given peer answers as context, they can identify strong points, correct errors, and synthesize. The layered structure formalizes this into a repeatable architecture.

Mixture of agents also exposes the limits of naive layering, and later research pushed back on the early enthusiasm. Mixing genuinely different models does not always help; sometimes a mixture of several samples from a single strong model matches or beats a mixture of different models, because a weak participant can contaminate the collaboration. The quality of the participants and the way they are combined matters more than the mere fact of combining, which echoes the ensemble lesson: diversity helps only when the diverse members are individually competent enough to contribute rather than mislead.

The cost profile is even steeper than a flat ensemble. A multi-layer mixture of agents runs many models across many layers for a single query, multiplying both expense and latency. Newer designs try to cut this by routing within the mixture, deciding on the fly which agents to invoke rather than running all of them at every layer, so the system spends effort only where it improves the answer. This blends the mixture-of-agents idea with the routing idea, and it points toward the direction the whole field has taken.

For the combination question, mixture of agents is important because it shows that combining models can produce emergent quality, answers better than any single participant, but only through structured collaboration on outputs, never through fusion of weights. The gain comes from models reading and improving each other’s language, which requires them to stay separate and intact. Collaboration, not fusion, is what lets combination exceed the best individual model. And collaboration is expensive, which is why the economically decisive innovation was not combining every model on every query, but learning to route each query to the right model, the subject of the next section.

Learned routing and the rise of the router

The most consequential form of model combination in production today does not merge models or run all of them at once. It picks the right model for each request. Routing has quietly become core infrastructure, and it is the answer the industry actually reached for when faced with a world of many strong, distinct, differently priced models.

The premise is simple. Once you have more than one model available, the hard part is no longer calling a model; it is deciding which model to call for each request. A trivial factual question does not need the most expensive frontier system; a hard reasoning problem does. A router is a lightweight component that looks at each query and sends it to the model best suited to handle it, balancing quality, cost, latency, privacy, and modality in a single decision made before the expensive model ever runs.

The field matured fast. RouteLLM established an academic baseline by learning routing decisions from preference data, showing that a router could send easy queries to a cheap model and hard ones to an expensive model while preserving most of the quality of always using the expensive one. A wave of follow-up methods learned routing through contrastive learning, contextual bandits, graph-based link prediction, and reinforcement learning. Some treat the router itself as a small model; others keep it as a fast classifier to avoid adding a full model’s worth of latency to every decision. Benchmarks now exist specifically to compare routers, a sign the subfield has its own identity.

Commercial tooling followed the research. Proxy layers like OpenRouter and LiteLLM let developers reach many models through one interface. Cloud platforms shipped intelligent prompt routing as a standard feature. Semantic routers moved the decision below the application layer, into the inference stack itself. By 2026 the practical advice for building a production system was concrete: start with a proxy router, add a learned classifier, instrument everything, and you have a routing layer in days rather than months. Routing went from a cost-cutting curiosity to production infrastructure, and the reason was money.

The variety of routing strategies reflects a genuine spread of goals. Some routers aim purely at cost, sending as much as possible to cheap models without dropping below a quality threshold. Some are difficulty-aware, estimating how hard a query is and matching it to a model of appropriate strength. Some are budget-aware across a whole session, spending the expensive model’s budget where it matters most. Some route among specialized agents for multi-step tasks, choosing not just a model but a collaboration mode and a role assignment. The common thread is that the router preserves each model’s specialization and exploits it per query, which is precisely the advantage merging throws away.

Routing also reshapes what “combining all models” means in practice. A well-built router with access to the whole frontier through a unified gateway is, functionally, a combined system that draws on every model, sending each request to whichever handles it best. It is combination without fusion, and it is the closest thing to a real “one system that uses all models” that exists. The difference from the fantasy is that it never produces a single blended brain; it produces an orchestra with a conductor, where each instrument stays itself and plays when its part comes up.

The strategic implication for anyone building on these models is to stay model-agnostic and route by task, because the leaderboard changes monthly and a system built to swap models cheaply turns every new flagship into an upgrade rather than a costly migration. That posture, treat models as interchangeable backends behind a router, is the mature industry answer to the whole combination question. It captures the upside of many models while sidestepping both the impossibility of fusion and the expense of always ensembling. And it pays for itself, as the economics make plain.

The economics that make routing win

The decisive argument for routing over both single-model and always-ensemble approaches is financial, and the numbers are large enough to have reshaped how serious systems are built. Combination through routing is not primarily a quality play; it is a cost play that also happens to preserve or improve quality.

The core economic fact is that model costs vary by orders of magnitude. Frontier proprietary models can cost tens of dollars per million output tokens, while capable open-weight and mid-tier models cost cents. A router that sends the easy majority of queries to cheap models and reserves the expensive frontier for the genuinely hard minority can cut costs dramatically while keeping quality nearly intact. Reported figures put routing-driven savings in the range of 40 to 85 percent, with one prominent method demonstrating an 85 percent cost reduction while retaining most of the quality of always using the top model, and a major cloud platform reporting around 60 percent savings from its routing feature.

The reason the savings do not wreck quality is that most queries are easy. In a typical mixed workload, a large share of requests are simple, factual, or routine, well within the reach of a cheap model. Only a minority genuinely need frontier capability. Paying frontier prices for the easy majority is pure waste, and a router eliminates it. The expensive model still handles the hard cases, so the quality on those cases is preserved, while the bill collapses because the easy cases no longer touch the expensive model.

This economic logic explains why routing beat ensembling as the default. Ensembling runs every model on every query, so it multiplies cost across the board, including on the easy queries where the extra models add nothing. Routing runs one model per query most of the time, escalating to more models or a stronger model only when the query warrants it. Routing captures most of the benefit of having many models while paying for many models only when it pays off. The most advanced systems blend the two, using routing to decide when a single cheap model suffices and when to escalate to an ensemble or a stronger model, so the expensive combination happens only on the queries that reward it.

Self-hosting economics add another layer. Running an open-weight model on your own hardware becomes cheaper than paying per-token API prices once your volume crosses a threshold, and that threshold varies with model size. Below it, hosted APIs win once you count engineering time; above it, self-hosting wins. A sophisticated routing layer can span both, sending some traffic to self-hosted open models and some to hosted frontier APIs, tuning the whole cost surface. The falling price of capable open-weight models, driven substantially by the efficiency of mixture-of-experts architectures, keeps pushing more traffic toward the cheap end.

The broader market has voted with its usage. On unified gateways that let developers reach many providers, usage has shifted rapidly toward whichever models offer the best capability per dollar, and cheaper models have captured large and growing shares of total token volume. The combination that won in the market is not a merged super-model; it is a cost-tuned orchestra of many models behind a router, and its success is measured in bills reduced rather than benchmarks topped. That outcome is worth holding against the fantasy: the practical value of combining all models turned out to be efficiency, not omniscience.

Distillation as a softer form of combining

There is a way to move capability from one model into another without merging a single weight and without running both at inference time. Knowledge distillation trains a new model to imitate an existing one, and it is a form of combination through imitation that has become both a standard efficiency technique and a geopolitical flashpoint.

The mechanism is teaching. A capable “teacher” model produces outputs, and a smaller “student” model is trained to reproduce them, learning not just the final answers but often the teacher’s full probability distribution over possible next tokens, which carries far more information than the single chosen word. The student absorbs much of the teacher’s behaviour into a smaller, cheaper package. Distillation is how a compact model can inherit most of a large model’s skill, and it is why small distilled variants of large systems can run on modest hardware while retaining a surprising amount of capability.

Distillation matters for the combination question in a subtle way. You can distill from multiple teachers, training one student to imitate several models at once, so the student combines capabilities from all of them. Because distillation operates on outputs rather than weights, it is not blocked by tokenizer or architecture mismatch; the student can have its own architecture and simply learns to behave like its teachers. This makes multi-teacher distillation one of the few routes to a single model that genuinely draws on several incompatible sources. The catch is that the student is bounded by its own capacity and by how well imitation captures the teachers’ reasoning, so it tends to approximate rather than exceed its teachers, and it inherits their blind spots along with their strengths.

The technique became contentious because it lets capability cross boundaries that companies and governments try to control. If a lab exposes a powerful model through an API, a competitor can query it at scale and distill its behaviour into a rival model, capturing much of the capability without the training cost. This is combination as extraction, and it undercuts the advantage of the lab that spent the money to build the frontier system. Reports through 2026 described deliberate campaigns to distill leading systems, using large numbers of accounts and adversarial prompts to harvest outputs, and framed it as a drain on hard-won capability.

Governments treated it as a strategic problem. Through 2026, United States policy explicitly targeted distillation as a channel through which frontier capability could leak to foreign competitors, with official memos instructing agencies and cloud platforms to share intelligence about extraction campaigns. Analysts noted that distillation weakens traditional export controls, because you do not need to move a model’s weights across a border if you can reconstruct much of its behaviour from its outputs. The concern is not hypothetical: usage data showed how quickly demand shifts toward cheaper models that match frontier capability, and distillation is one of the mechanisms that lets those cheaper models catch up.

The defensive response has been partly technical, detecting and blocking suspected distillation traffic, and partly collaborative, with labs coordinating through shared forums, though that coordination runs into antitrust uncertainty of its own. Whether the countermeasures work remains unsettled, and the tension between protecting frontier investment and the ease of learning from exposed outputs is one of the defining strategic problems of the current moment.

Distillation rounds out the technical picture of combination. Weight merging fuses compatible models. Mixture of experts builds combination into one architecture. Ensembles and routing combine outputs. Distillation combines through imitation. None of them produces the mythical merged super-mind; each produces a specific, bounded kind of combination with its own costs and limits. And distillation, by letting capability leak across every boundary, sets up the larger anxiety that the next sections address: not what a combined model would be, but what a combined industry would be.

Homogenization and the cost of everything converging

Even without any deliberate merging, the models are drifting toward each other, and that quiet convergence carries a cost the combination question rarely considers. If every model trains on overlapping data, distills from the same teachers, and tunes for the same benchmarks, the field starts to homogenize, and a homogeneous field has hidden failure modes that a diverse one does not.

The convergence is visible in several ways. Architectures have converged on sparse mixture of experts almost universally. Benchmark scores have compressed, with the top models clustering within a few points of each other on the tests that matter. Training data overlaps heavily, because everyone scrapes similar web-scale corpora. Distillation actively copies behaviour from leaders to followers. The result is that models increasingly resemble one another, not because anyone merged them, but because they are converging on the same solutions to the same problems. A field where every model is trained similarly on similar data will produce models that fail similarly, and correlated failure is exactly what makes ensembles and routing less useful, since the whole benefit of combining models depends on their making different mistakes.

Empirical work on how models behave hints at the shape of the problem. A study of how frontier models recommend brands found that their recommendations cluster in a much narrower band than the real market’s diversity, regardless of the actual spread of options. Several different models, asked the same kind of question, converged on a similar narrow set of answers. If models homogenize, their collective output narrows, and combining many homogeneous models does not broaden the answer; it reinforces the same narrow consensus. The combination that was supposed to give you the best of every perspective instead gives you the same perspective, amplified.

This is the deeper irony of the combination dream. People imagine that combining all models would capture the full diversity of approaches. But the models are becoming less diverse over time, so a combination of them increasingly reflects one dominant way of thinking rather than many. The value of having many models comes from their differences, and those differences are eroding under competitive and economic pressure to converge on whatever works best on shared benchmarks.

Homogenization also concentrates risk. If most of the world’s AI systems share architectural assumptions, training data, and behavioural tendencies, then a flaw common to all of them, a shared bias, a shared vulnerability, a shared blind spot, becomes a systemic flaw with no diverse alternatives to fall back on. A monoculture is productive and fragile at the same time. In agriculture, planting one high-yield crop everywhere maximizes output and maximizes the damage when one disease arrives. The same logic applies to models: convergence raises average capability while removing the diversity that would contain a shared failure.

The counter-pressure is the open-weight ecosystem, which introduces genuine architectural and training diversity, and the ongoing competition among labs pursuing different approaches. Whether that diversity survives the gravitational pull toward convergence is an open question, and it connects directly to the organizational version of the combination question. The technical trend toward homogenized models and the market trend toward consolidated companies are the same phenomenon viewed from two angles, and the second angle, what happens if the companies themselves combine, is where the largest stakes lie.

Sector by sector the picture changes

The abstract question of combining models becomes concrete when you look at how it plays out in specific industries, because the right form of combination depends heavily on what a sector needs, what it can spend, and what it is allowed to do with its data. The same underlying techniques, merging, mixture of experts, routing, ensembling, distillation, land very differently across software, healthcare, finance, legal work, and customer operations.

In software development, combination is already the norm, though not through weight merging. Coding tools route between models depending on the task: a fast cheap model for autocomplete, a stronger model for writing a function, a frontier model for resolving a real bug in a large repository. The tight clustering of top coders on software-engineering benchmarks means the choice of scaffold, how the model is prompted, given tools, and iterated, often matters more than which model sits underneath. Teams increasingly build model-agnostic pipelines so they can swap in whichever coder leads this month, and some run ensembles that generate several candidate patches and select the one that passes tests. For coding, the winning combination is a routed pipeline with test-based selection, not a merged model, because correctness is checkable and diversity of attempts directly raises the hit rate.

In healthcare, the binding constraints are privacy, reliability, and regulation, which reshape the combination calculus entirely. Sensitive patient data often cannot leave controlled environments, which pushes toward self-hosted open-weight models rather than external APIs, and toward routing that keeps regulated data on compliant infrastructure while sending only non-sensitive queries to external systems. Distillation is attractive here because a smaller distilled model can run on-premises under tight control while retaining much of a larger model’s clinical knowledge. Ensembling has real appeal for high-stakes decisions, where cross-checking several models’ outputs can catch errors, but every added model expands the surface that must be validated and documented for regulators.

In finance, the drivers are latency, auditability, and cost at scale. Real-time applications need fast models, so routing sends time-critical queries to low-latency systems and reserves slower, stronger models for analysis that can tolerate delay. Auditability pushes against opaque combinations: a routed system that logs which model handled which query is easier to explain to a regulator than a merged model whose behaviour is an inscrutable average. Because financial workloads run at enormous volume, the cost savings from routing translate into large absolute sums, making the economic case for combination especially strong.

In legal work, accuracy and traceability dominate. The cost of a confidently wrong answer is high, so ensembling and cross-checking earn their keep, and the ability to cite which model produced which claim matters for professional accountability. Long-context handling is valuable for working through lengthy documents, which favours models with strong long-context performance and routing that directs document-heavy tasks to them. Here too, self-hosting appears wherever client confidentiality forbids sending material to external services.

In customer operations, the practical lesson from deployment is that the system around the model matters more than the model itself. Well-designed agents that route queries, pull from a knowledge base, and escalate to humans at the right moment reach high automation rates regardless of which underlying model powers them. Across sectors, the pattern repeats: combination succeeds as orchestration tuned to the sector’s constraints, and fails as an attempt to build one model that does everything. The differences between sectors are differences in constraints, privacy, latency, auditability, cost, and the correct combination architecture falls out of those constraints rather than from any universal recipe. This is also why the practical guidance for teams, addressed next, is less about which technique is best in the abstract and more about matching technique to situation.

Practical guidance for teams tempted to combine models

For an organization actually deciding how to use many models, the abstract analysis resolves into a set of concrete choices. The mistake to avoid is reaching for the most complex form of combination when a simpler one would do, or attempting weight merging across incompatible models because the idea sounds appealing. A practical decision process runs roughly as follows.

Start by asking whether you need combination at all. For many workloads, a single well-chosen model with good prompting outperforms a poorly designed combination, and the gap between top models is now small enough that prompt and workflow quality often matter more than model choice. If one strong model meets your needs, the cheapest reliable combination is no combination. Complexity should be added only when a single model demonstrably falls short.

If you do need to draw on several models, the next question is whether you can afford to run more than one per query. If accuracy is worth the cost, on high-stakes decisions, on checkable tasks like code, ensembling or a small mixture of agents can beat any single model, especially when the models are genuinely diverse. If cost matters, which is almost always, routing is the default: send each query to the cheapest model that can handle it, escalating only when needed. The reported savings of 40 to 85 percent from routing are large enough that most production systems should have a routing layer before they consider anything fancier.

The five mechanisms of combination differ sharply on the two questions that matter most in practice: whether they work across unrelated models, and what they cost.

Combination mechanisms compared on cross-family reach and main cost

The table makes the practical hierarchy clear: routing is the low-cost default that works across the whole frontier, ensembling buys accuracy at multiplied cost, merging is cheap but boxed inside a family, and distillation is the route to a single deployable model that draws on incompatible sources. Only routing, ensembling, and distillation cross the family boundary at all.

Weight merging is worth considering only in a narrow case: you have several models from the same family, fine-tuned on complementary tasks, and you want one model that handles all of them without the cost of running several. Then tools like MergeKit and methods like DARE-TIES can produce a strong merged generalist cheaply. Do not attempt to merge models from different families; the tokenizer and architecture mismatch make it incoherent, and the effort is wasted. If the models you want to combine are unrelated, combine their outputs through routing or ensembling instead.

Distillation is the right tool when you need one deployable model that captures capability from larger or multiple sources, especially under constraints, on-premises deployment, tight latency, limited hardware, that rule out calling the originals at inference time. Expect the distilled model to approximate rather than exceed its teachers, and be aware of the legal and policy sensitivities around distilling from models you access through an API.

Whatever you build, build it to swap. The single most durable piece of advice in a field where the leaderboard changes monthly is to stay model-agnostic behind a routing or gateway layer, so that adopting a new model is a configuration change rather than a rewrite. A system designed to swap models cheaply turns every new release into a free upgrade, while a system hardwired to one model or one merged blend turns every release into a migration project.

Finally, instrument everything. Combination systems fail in quiet ways, a router sending too much to an expensive model, an ensemble whose fusion is dragged down by a weak member, a merged model that lost a capability nobody tested. Logging which model handled which query, measuring quality and cost per route, and testing capabilities after any change are what keep a combined system honest. The teams that succeed with combination treat it as an operations discipline, not a one-time architecture decision, and that discipline is what separates a combination that saves money and improves answers from one that quietly degrades both.

The corporate version of the same question

Strip away the technical layer and the combination question has a blunt business form: what if the companies training these models stopped competing and combined into one? This reading is less fanciful than it sounds, because a partial version is already underway, structured carefully to avoid the word “merger.”

The economics of frontier development push hard toward consolidation. Training a frontier model costs enormous sums, requires scarce compute, and depends on a small pool of specialized talent. These are the conditions under which markets concentrate. Analysts describe the frontier as shaped by winner-take-all forces, where a few dominant players expect to capture most of the profit and influence, and where the resources needed to compete keep rising. The developers of the most capable generalist models expect to concentrate trillions of dollars of value and enormous deal-making power in a small number of firms, and they invest accordingly.

The consolidation is happening through mechanisms designed to slip past merger review. Through 2026, frontier labs repeatedly acquired specific capabilities not by buying companies outright but by hiring entire teams and licensing their technology, structures crafted to avoid classification as a merger. In one documented stretch, several labs made several such deals within days, each targeting a different capability gap, none announced as related events. One lab licensed an entire team from another company through a structure explicitly designed to avoid antitrust classification. The frontier reached a scale where buying a specific technical capability through a talent-and-licensing deal is faster and cheaper than building it, and structuring it as a talent deal sidesteps the scrutiny a formal acquisition would attract. Meanwhile, the largest labs raised capital at valuations in the hundreds of billions, giving them the balance sheets to absorb smaller players as rounding errors.

This is combination of a different and more consequential kind. It does not fuse models; it fuses the organizations that make them. And unlike weight merging, it faces no tokenizer or architecture wall. Companies combine capabilities through people and contracts effortlessly, which is exactly why the organizational version of the question is the one with real stakes.

A full corporate merger of the frontier, every major lab under one roof, would not produce a smarter model for the technical reasons already covered; the merged company would still train models one at a time, and those models would face the same interference and averaging limits. What it would produce is a single entity controlling the compute, the data pipelines, the talent, and the most capable systems simultaneously. The danger of combining the industry is not a super-intelligence; it is a super-concentration, a single point of control over a technology the economy increasingly runs on.

The infrastructure layer already shows how concentrated the surrounding market is. A handful of cloud providers control the compute. One company holds a near-monopoly on the most advanced AI chips, and another on the lithography machines that make them. The foundation-model layer sits on top of these choke points, and only a few countries even host companies capable of training models at the largest scale. Consolidation at the model layer would stack another concentration on top of an already concentrated stack.

The organizational combination question therefore inverts the technical one. Technically, combining all models is hard and produces mediocrity. Organizationally, combining all the model-makers is easy and produces concentration. The first is a mathematical curiosity; the second is a live policy problem that regulators, investors, and safety researchers are actively wrestling with, and it splits into two arguments that pull in opposite directions.

Concentration of power and the safety argument on both sides

The prospect of a consolidated AI industry produces a genuine dilemma, because the same concentration that worries competition regulators is sometimes argued to help with safety, and the two concerns point toward opposite policies. Taking both seriously is necessary to understand why the question has no easy answer.

The competition case against concentration is straightforward. Information industries have a long history of beginning in a burst of innovation and then consolidating around monopolies that can suppress competition and slow the very innovation that built them. A frontier controlled by one or a few firms concentrates not just money but the power to decide how a foundational technology develops, who can access it, and on what terms. When AI shifts from a discretionary tool to core infrastructure, control over it becomes control over a dependency the whole economy shares, and the firms that hold it gain power over everyone downstream. Emerging competitors face barriers built from data, compute, and capital advantages that are self-reinforcing, since the leaders’ scale lets them stay ahead and absorb challengers.

The safety argument for some coordination runs the other way. Frontier models carry risks, and a field of many labs racing to ship the most capable system as fast as possible creates pressure to cut corners on testing and safeguards. Some argue that coordination among labs, sharing safety research, agreeing on testing standards, jointly defending against misuse, reduces the race pressures that make everyone less careful. Labs have formed shared bodies for exactly this purpose. The tension is that the same coordination that could improve safety can also look like collusion, and antitrust law does not cleanly distinguish a safety agreement from a market-sharing one. Reports noted that collaborative defense efforts among labs were hampered precisely by uncertainty over whether the coordination would run afoul of competition rules.

This puts policymakers in a bind. Push hard on competition, and you may fragment the coordination that safety arguably needs, and you may hand advantage to jurisdictions that impose no such constraints, since restricting one country’s labs can simply shift usage and development to another’s. Push toward coordination, and you risk entrenching a cartel that concentrates power and dampens the competition that drives improvement and keeps prices down. There is no setting of the dial that satisfies both concerns at once, which is why the debate is unresolved and why reasonable people land in different places on it.

A further complication is the geopolitics. Frontier capability is increasingly treated as a matter of national strategy, with export controls, distillation crackdowns, and concerns about capability leaking to rival states. In that frame, some argue for consolidating national champions strong enough to compete internationally, which cuts against domestic competition policy. Others warn that concentrating capability, whether in a company or a state, creates a single point of failure and a target for capture. Power concentrating into a small number of entities, whether corporate or governmental, is the shared worry underneath every version of this debate, and different observers weigh the corporate and the governmental versions differently.

The honest summary is that combining the industry is neither clearly good nor clearly bad; it trades competition and resilience against coordination and scale, and which trade is worth making depends on judgments about how dangerous the technology is, how well coordination actually improves safety, and how much one trusts concentrated power to act well. Those judgments are not settled, and the policy tools meant to manage them are the subject of the next section.

Regulatory and compliance angles that shape the outcome

Regulation shapes both versions of the combination question, the technical and the corporate, and the rules in force by 2026 pull in several directions at once. Anyone thinking about combining models, whether as an engineering choice or a business one, runs into a policy environment that is active, fragmented, and still forming.

On the competition side, regulators in the United States and the European Union tightened oversight of the AI market through 2026, with national AI action plans and digital-market rules imposing compliance obligations that fall hardest on large firms. The intent is to keep the market open, but a recurring criticism is that heavy compliance costs can favour incumbents who can afford them over the startups the rules are meant to protect. Competition policy aimed at preventing concentration can, through compliance burden, entrench the very incumbents it targets, an irony that complicates every attempt to regulate consolidation.

The talent-and-licensing deal structures that labs used to absorb capabilities were specifically designed around merger-review thresholds, which tells you that the labs anticipate friction on traditional acquisitions and are adapting faster than the rules. This creates a cat-and-mouse pattern where regulators write rules for mergers and labs achieve the effect of mergers through arrangements the rules do not cleanly cover. Whether authorities extend merger scrutiny to reach acqui-hires and licensing deals is an open regulatory question with large consequences for how much the industry can consolidate.

On the capability-control side, distillation and export controls became central. United States policy through 2026 explicitly treated model distillation as a leakage channel and directed agencies and cloud platforms to coordinate against extraction campaigns, framing it as a national-security concern. Distillation strains export controls because reconstructing a model’s behaviour from its outputs does not require moving its weights across any border, which means traditional controls on shipping technology are poorly matched to the way capability actually spreads. The effectiveness of technical countermeasures against distillation was, by mid-2026, still unproven.

There is also a jurisdictional fragmentation that directly affects combination strategies. Data-sovereignty and privacy rules in various regions constrain which models can process which data and where. This pushes regulated industries toward self-hosted open-weight models and toward routing architectures that keep sensitive data on compliant infrastructure. A combination system spanning multiple jurisdictions has to route not just for cost and quality but for legal compliance, sending each query to a model whose deployment satisfies the rules that apply to that data. Compliance has become a routing criterion in its own right, alongside cost, latency, and quality.

The licensing terms of the models themselves add another layer. Open-weight models come under a range of licenses, some permissive, some with commercial restrictions, and combining models means combining their license obligations. A merged or ensembled system inherits the constraints of every model it includes, and jurisdictions differ in which licenses they favour for compliance. Western-jurisdiction deployments often prefer models under permissive licenses from providers aligned with local regulatory expectations, which shapes which models are practical to combine for a given organization.

The regulatory picture, viewed as a whole, does not point to a single outcome. It makes literal corporate combination legally fraught, makes distillation-based capability transfer a policy target, and turns compliance into a factor that combination architectures must actively route around. For the technical builder, the practical upshot is that the choice of which models to combine is constrained as much by license and jurisdiction as by capability, and that a sound combination strategy has to treat compliance as a first-class design input rather than an afterthought. Privacy, closely related, deserves its own treatment.

Privacy and data handling inside multi-model systems

Combining models multiplies the paths that data can take, and each path is a place where sensitive information can leak, be logged, or cross a boundary it should not. Any serious combination system has to treat data handling as a core design problem, because the more models a query touches, the more surfaces its contents are exposed to.

Consider what happens to a single query in different combination architectures. In a routed system, the query goes to one model, but the router itself has to inspect the query to decide where to send it, so the routing layer sees everything. In an ensemble, the query is copied to every model in the ensemble, so its contents are exposed to each provider. In a mixture of agents, the query and the intermediate answers pass through multiple models across multiple layers, creating a chain of exposure. Every additional model in a combination is another party that processes the data, and in a multi-vendor system, another company whose data-handling practices you inherit.

For regulated data, this is decisive. Sending sensitive information to an external API means trusting that provider’s retention, logging, and security practices, and in many regulated contexts that trust is not permitted at all. This is a major reason privacy-sensitive sectors gravitate toward self-hosted open-weight models: keeping the model on controlled infrastructure keeps the data there too. A combination system for such a sector often cannot freely route to external frontier APIs; it must keep sensitive queries on internal models and reserve external systems for non-sensitive traffic, if it uses them at all.

Distillation offers a privacy-relevant path here as well. A distilled model small enough to run on-premises lets an organization capture much of a large model’s capability while keeping all data internal, avoiding the exposure that calling an external API would create. This is one of the practical attractions of distillation for privacy-constrained deployments: it converts a capability that would otherwise require sending data outside into one that runs entirely within the organization’s control.

The routing layer’s visibility deserves special caution. Because the router inspects every query to make its decision, it is a natural chokepoint for both control and risk. Done well, it can enforce privacy policy centrally, stripping or masking sensitive fields before forwarding, keeping regulated queries on compliant models, and logging access for audit. Done carelessly, it becomes a single component that sees all traffic and could leak or mishandle it. The router is simultaneously the best place to enforce privacy and the worst place to get privacy wrong, and designing it as a deliberate policy-enforcement point rather than a dumb dispatcher is what separates a compliant combination system from a leaky one.

There is also the matter of what combination systems remember. Systems that cache results, store intermediate outputs, or log for quality monitoring accumulate data that must be governed, and the more models and layers involved, the more places this data collects. A combination architecture that saves money on inference but quietly builds an ungoverned store of sensitive queries across several logging points has traded one cost for a larger liability. Treating data minimization and clear retention rules as part of the combination design, rather than as a compliance patch applied afterward, is what keeps the efficiency gains from turning into a privacy exposure. These handling concerns feed directly into the broader risk picture, which is worth laying out plainly.

Risks, limits, and failure modes worth naming

Every form of combination carries characteristic ways of failing, and naming them plainly is more useful than any general reassurance. The risks differ by method, and a builder who knows the specific failure mode of the approach they chose is far better placed than one who trusts that combination simply works.

Weight merging fails through capability loss that hides until tested. A merge can look fine on the benchmarks you checked while having quietly lost a capability you did not. Because merging averages parameters, it can erode a skill that lived in the extremes without any obvious signal, and the loss surfaces only when a real query hits the missing capability. The mitigation is systematic testing of every capability the source models had, before and after the merge, but the risk is that you cannot test for a capability you did not know to check.

Mixture of experts fails through routing pathologies and load imbalance. If the router collapses onto a few experts, the model wastes capacity; if a load-bearing expert is disturbed, the whole model can degrade catastrophically, as the pruning experiments showed. These failures are internal and can be hard to diagnose from outside, and they are why sparse models demand careful monitoring of routing behaviour, not just output quality.

Ensembles and mixtures of agents fail through correlated errors and contamination. The benefit of combining outputs depends on the models making independent mistakes, but as the field homogenizes, their errors correlate, and the ensemble’s advantage shrinks. Worse, a weak participant can drag down the fused result, so adding a model can lower quality rather than raise it. The mitigation is to combine genuinely diverse, individually competent models and to measure whether the combination actually beats its best member, rather than assuming it does.

Routing fails through misrouting and drift. A router that sends a hard query to a weak model produces a bad answer at a good price, which is a false economy. Routers trained on one distribution of queries can drift as real traffic changes, quietly degrading over time. And a router aiming purely at cost can starve quality without anyone noticing until users complain. The mitigation is continuous evaluation of routing decisions against outcomes, not just against cost.

Distillation fails through inheriting blind spots and losing safeguards. A student model absorbs its teachers’ weaknesses along with their strengths, and safety behaviours in particular can vanish in the copying, since imitation of outputs does not reliably transfer the guardrails that shaped them. A distilled model can therefore be less safe than its teacher while appearing similarly capable, which is one reason distillation raises misuse concerns.

Across all methods, three risks recur. Homogenization removes the diversity that combination depends on, so the better everything gets at converging, the less combination helps. Concentration at the organizational level creates single points of failure and control. And evaluation debt builds up whenever a combination is trusted without the testing that would reveal its quiet degradations. The last is the most insidious, because combination systems are complex enough that their failures are easy to miss and easy to rationalize.

None of these risks argues against combination as such. Routing and ensembling deliver real value, and mixture of experts powers the whole frontier. The argument is for combining with eyes open, knowing that each method trades one thing for another and fails in a specific way, and that the failures are usually quiet rather than loud. The systems that succeed are the ones whose builders treat combination as a set of trade-offs to be measured, not a free lunch to be assumed. That posture, combined with realistic expectations about what combination can and cannot deliver, is what the plausible futures reward.

Realistic scenarios for the next few years

Projecting where combination goes from here means separating the trend lines that are already firm from the outcomes that remain genuinely uncertain. Several directions look firm enough to state with reasonable confidence, framed as analysis rather than prediction.

The clearest trend is that orchestration will keep beating fusion as the practical way to combine models. Routing is already production infrastructure, its economics are decisive, and the tooling keeps maturing. The likely path is toward smarter routers that decide not just which model but how much effort a query deserves, when to escalate to an ensemble, and when a cheap model suffices, with the routing decision moving deeper into the inference stack. The mature architecture is a model-agnostic layer that treats every model as a swappable backend, and that pattern will spread because it turns each new release into an upgrade rather than a migration.

Within families, automated merging will keep improving as a way to recycle and consolidate fine-tunes. Evolutionary and differentiable methods will make it easier to blend related models into strong generalists cheaply, and this will remain valuable for teams that want one model instead of many closely related ones. It will not extend to cross-family fusion, because the tokenizer and architecture walls are structural, not engineering gaps waiting to be closed.

Mixture of experts will remain the frontier architecture, and the interesting question shifts from whether to use it to how to configure the expert topology, and how to make routing and load balancing more resilient against the collapse failures recent work exposed. Expect continued work on making sparse models cheaper to serve, since memory, not compute, is their binding cost.

The organizational trajectory points toward continued quiet consolidation through talent and licensing deals, unless regulators extend scrutiny to reach them. The economics favour concentration, and the labs are structuring around merger review faster than the rules adapt. The counterweight is the open-weight ecosystem, which keeps genuine diversity alive and keeps prices falling, and whose continued competitiveness is the single biggest factor determining whether the frontier consolidates or stays plural.

The genuinely uncertain outcomes cluster around a few questions. Whether open weights stay competitive with closed systems will decide how concentrated the field becomes. Whether distillation countermeasures work will decide how fast capability leaks and how durable any lab’s lead is. Whether coordination among labs is treated as safety cooperation or as collusion will shape both safety and competition. And whether homogenization continues or diversity reasserts itself will determine how much value combination can even deliver, since combination feeds on difference.

One scenario is worth naming because it captures the plausible near future: an expert-orchestration world, where no single model dominates, where systems routinely draw on many models through routers and gateways, where families of related models are merged into strong generalists, where mixture of experts remains the internal architecture, and where the “combined intelligence” people imagine is realized as an orchestra of specialists rather than a single fused mind. This world already exists in partial form, and the trends point toward its deepening rather than toward the fantasy of one merged super-model. The future of combination is a system of many models coordinated well, not one model that swallowed the rest. What that leaves genuinely open is examined last.

Open questions the evidence cannot yet settle

Several parts of the combination question sit beyond what current evidence can resolve, and honesty requires marking them as open rather than forcing a conclusion. These are the places where confident answers would outrun the facts.

The first is whether cross-architecture fusion will ever become practical. Today it is ill-defined because models lack a shared coordinate system, but research on aligning models through permutation symmetries, on re-embedding vocabularies, and on merging in shared latent spaces is active. It is possible that future methods will make some form of principled cross-family combination feasible, though the tokenizer and representation gaps are deep enough that a full solution is not visibly close. Whether the compatibility wall is a permanent structural fact or a hard engineering problem awaiting a breakthrough is genuinely unsettled.

The second is how far the pull toward the mean actually binds. The evidence that averaging erodes peaks is strong within current merging methods, but methods that extrapolate beyond source models in weight space, or that learn coordination rather than average, hint that the ceiling might be higher than naive averaging suggests. It remains unclear how much of the mediocrity result is fundamental to combining models and how much is an artifact of the specific techniques used so far.

The third is whether homogenization continues or reverses. If models keep converging, combination loses its value, but competitive pressure, open-weight diversity, and deliberate attempts to build differentiated systems could sustain the difference that makes combination worthwhile. The trajectory of diversity in the model ecosystem is the hinge on which the long-term value of combination turns, and it is not predictable from current data.

The fourth is the organizational one. Whether the frontier consolidates into a few firms or stays plural depends on regulatory choices, the durability of open weights, the effectiveness of distillation countermeasures, and geopolitical forces that are all in flux. The economics favour concentration, but the counter-pressures are real, and the outcome is contingent on decisions not yet made.

The fifth is the safety-versus-competition trade-off, which is a values question as much as an empirical one. How dangerous frontier models actually are, how much coordination genuinely improves safety, and how much to trust concentrated power are judgments on which informed people disagree, and no amount of technical analysis resolves them. These are questions the evidence informs but cannot decide, and pretending otherwise would misrepresent the state of knowledge.

What can be said firmly is the shape of the answer to the original question. Combining all available models does not produce a smarter super-model, because fusion is mostly impossible across the real frontier and produces averages where it is possible. It produces real value through orchestration, routing, ensembling, and mixture of experts, which combine capability while preserving specialization. And its largest stakes are organizational, where combining the makers of models raises concentration risks that dwarf the technical curiosity of combining the models themselves. The uncertainties above sit around that firm core, and they are the parts worth watching as the field develops.

A closer look at what a naive full merge would actually produce

It helps to trace, step by step, what would happen if someone ignored every warning and tried to average the weights of a handful of frontier models directly, because the concrete failure sequence makes the abstract impossibility tangible.

The attempt would fail at the first step, loading the models into a common format for averaging. The models have different numbers of parameters, so their weight tensors have different shapes. A tool asked to average a matrix of one dimension with a matrix of another dimension would simply error out; the operation is undefined. To proceed at all, you would have to force the models into a common shape, discarding or interpolating parameters to make the dimensions match, and that forcing already destroys the correspondence that averaging assumes. Before any semantic problem arises, the arithmetic itself refuses to run on models of different shapes.

Suppose you cheated and selected only models that happened to share dimensions. The next failure is the embedding table. Each model’s embeddings are indexed by its own tokenizer’s vocabulary, and those vocabularies differ. Row 5,000 encodes different tokens in different models. Averaging the embedding tables blends the representations of unrelated tokens, so the very first layer of the merged model, the one that turns input into vectors, now maps tokens to meaningless averaged vectors. The model’s input representation is corrupted at the source, and everything downstream inherits the corruption.

Suppose you cheated further and used only models with an identical tokenizer, which across the true frontier is nearly impossible but grant it. Now the internal representations collide. Two models trained independently place their learned features in different neurons; the concept a given neuron encodes in one model is encoded by a different neuron in the other. Averaging the weights combines neuron 12,000 of one model with neuron 12,000 of the other, but those neurons learned different things, so the average encodes neither. The merged model’s internal features become incoherent superpositions of unrelated learned concepts, and its behaviour degrades toward noise rather than toward a blend of capabilities.

Even in the one case where this works, models fine-tuned from a shared base with a shared tokenizer and matching shapes, the output is the averaged generalist discussed earlier: competent, unremarkable, and weaker at each specialty than the specialists it came from. That is the best case, and it is precisely the case the merging community actually operates in, which is why it never attempts the naive full merge across families in the first place.

The walkthrough clarifies why the impossibility is not a matter of insufficient computing power or cleverness. A naive full merge fails at the level of definition, then at the level of input encoding, then at the level of internal representation, and even when it succeeds it produces mediocrity. Each layer of failure is independent, so fixing one does not rescue the attempt. This is the concrete reality behind the paint-mixing metaphor, and it is why serious practitioners treat the fantasy of merging everything as a category mistake rather than an ambitious goal.

The path from merging to orchestration

The current state of combination did not appear fully formed; it emerged from a decade of scaling pressure that pushed the field first toward bigger dense models, then toward sparsity, then toward orchestration. Understanding that path explains why orchestration, not fusion, became the answer.

The early scaling era chased size directly. Bigger dense models trained on more data reliably performed better, and for several years the recipe was simply to build larger networks. But dense scaling ran into a wall: every parameter added to a dense model must be used on every token, so the compute cost of both training and serving grew in lockstep with capability. The industry hit the point where making a dense model smarter made it proportionally more expensive to run, which was economically unsustainable at the frontier.

Mixture of experts was the escape. By activating only a fraction of parameters per token, sparse models broke the link between total capacity and per-token cost, letting models grow enormous in knowledge while staying affordable to run. The idea had existed since 1991, but it took the scaling wall to make it the obvious answer, and by the mid-2020s it swept the frontier. This was the first form of combination to win at scale: many experts inside one model, coordinated by a learned router.

Merging emerged in parallel from a different pressure, the waste of experimentation. Fine-tuning a model many ways to find the best variant produced many models and used only one, discarding the rest. Merging offered a way to recycle that effort by blending the variants, turning discarded experiments into contributions to a stronger generalist. Toolkits democratized it, and merged models climbed the open leaderboards, establishing merging as a mainstream technique, within its family constraint, in about two years.

Routing and ensembling emerged from yet another pressure, the proliferation of models. Once many strong models existed at wildly different prices, the question of which to use for each query became unavoidable, and the answer, a router, turned the abundance of models from a confusion into an asset. The economics did the rest: routing’s cost savings were too large to ignore, and it became infrastructure.

The through-line is that each form of combination arose to solve a specific pressure, scaling cost, experimentation waste, or model proliferation, and none of them arose to build a merged super-mind, because no pressure ever pointed that way. The fantasy of fusing everything into one smarter model was never the goal the field was actually pursuing. The real goals were efficiency, recycling, and coordination, and the techniques that won, mixture of experts and routing, are the ones that served those goals. Combination, properly understood, is the industry’s answer to cost and abundance, not its path to superintelligence. Seen through this history, the question “what if we combined all models” turns out to have already been answered, repeatedly, in ways more useful and more mundane than the question imagines.

The benchmark trap behind the combined-score fantasy

Part of what makes combining all models sound appealing is a misreading of benchmarks. People see one model leading on coding, another on reasoning, another on math, and imagine that a combined model would simply hold all the leading scores at once. That intuition rests on a mistake about what benchmark scores are and how they relate to a single system’s behaviour.

A benchmark score is a measurement of one model’s behaviour on one distribution of tasks. It is not a transferable component that can be detached from the model and installed elsewhere. A model’s high coding score is a property of that whole model, not a module you can extract and bolt onto another system. The strength that produces the score is distributed across the model’s parameters, entangled with everything else it does, and shaped by the specific training that also determines its weaknesses. There is no coding-score organ to transplant.

This is why a merged model does not inherit the top score of each ingredient. Merging averages the parameters, and the averaged parameters produce averaged behaviour, which lands below each specialist on its specialty. The combined model does not get the best coding score and the best reasoning score; it gets a middling version of both. The benchmark leaderboard is not a menu from which a combination can order the best of each row, because each row’s score is inseparable from the model that earned it.

There is a further trap in the idea of a “combined score” even for orchestration systems. A router that sends coding queries to the best coder and reasoning queries to the best reasoner could, in principle, post the top score on each benchmark by routing each benchmark to the right model. But this measures the router’s ability to pick, not the existence of a single model that is best at everything. The impressive combined result belongs to the orchestration system, not to any model within it, and it is achieved by keeping the models separate, precisely the opposite of merging them into one. The scores stay high because the specialists stay intact and get chosen appropriately.

Benchmarks also mislead because they saturate and compress. As frontier models cluster within a few points of each other on the tests that once separated them, the differences that remain are often within the noise of the evaluation, and the choice of scaffold or prompt can swing the result more than the underlying model. A combination built to chase small benchmark gaps may be chasing measurement artifacts rather than real capability differences. And contamination concerns, where test data leaks into training, further muddy what a high score even means.

The practical lesson is to distrust the combined-score fantasy specifically. No single system will hold every top score, because scores are properties of whole models and averaging models averages their scores. The way to get top performance across categories is orchestration that routes to specialists, and the credit for that performance belongs to the routing system’s design, not to any merged intelligence. Treating benchmarks as detachable capabilities is the error that makes the combination fantasy seem reasonable, and seeing through it dissolves much of the fantasy’s appeal.

A team of human experts as the sharpest analogy

A useful way to sanity-check claims about combining models is to ask how the analogous claim would sound for combining people, because the intuitions transfer more cleanly than the technical details, and they point in the same direction the evidence does.

Imagine you have five brilliant specialists: a mathematician, a programmer, a translator, a doctor, and a lawyer. The combination fantasy is the equivalent of proposing to blend these five people into one person who is simultaneously the best at all five fields. Stated about people, the absurdity is obvious. You cannot average five brains into one super-brain. What you can do is put the five experts in a room and coordinate them, which is exactly what orchestration does with models. The team, well-coordinated, outperforms any individual on the full range of problems, not because it fused into one mind but because it routes each problem to the right expert and lets them collaborate.

The analogy illuminates several of the technical findings. The pull toward the mean corresponds to what would happen if you forced the five experts to always answer every question by committee consensus: the mathematician’s sharp answer on a math question gets watered down by four non-mathematicians voting, and the group’s answer on every question is blander than the relevant expert’s would have been. Averaging is committee-by-default, and it wastes expertise exactly as forced consensus wastes a specialist.

Routing corresponds to a good manager who knows each expert’s strength and sends each question to the right person, escalating to a group discussion only for genuinely hard cross-disciplinary problems. This is fast and reliable, and it preserves each expert’s edge. Mixture of agents corresponds to structured collaboration, where experts draft, review each other’s work, and refine, which produces better results than any single first draft. The forms of combination that work for models are the same forms that work for teams of people, and the form that fails, fusion into one, fails for the same reason in both cases: expertise is not additive by blending.

The analogy also clarifies the diversity point. A team of five identical experts is far less useful than a team of five different ones, because the identical team makes the same mistakes and covers the same ground. Homogenization among models is the equivalent of a team drifting toward sameness, losing the complementary differences that made combining them worthwhile. The value of a team, like the value of a model ensemble, comes from genuine diversity of strength.

And the organizational question maps cleanly too. Combining the five experts into one firm that monopolizes all five specialties is the corporate consolidation question, and its risks, concentration of power, loss of competition, single points of failure, are the same whether the specialists are people or the firms are AI labs. The team analogy holds across every version of the combination question, technical and organizational, and in every version it favours coordination over fusion and diversity over sameness. That convergence between the human intuition and the technical evidence is a good sign that the conclusion is sound: the way to get the best of many capabilities is to combine them as a coordinated system of distinct parts, never to melt them into a single average.

The combination question as a mirror for how intelligence gets imagined

Underneath the technical and organizational layers, the question of combining all models rests on an assumption worth examining directly, because the assumption is where the confusion starts. The fantasy treats machine intelligence as a fluid that can be poured from many containers into one, accumulating as it goes. The evidence says intelligence in these systems does not behave like a fluid at all.

A model’s capabilities are not a quantity of some substance called intelligence. They are a structured profile, a specific arrangement of strengths and weaknesses produced by a specific architecture, training data, and set of choices. You cannot add profiles together to get a bigger profile; you can only average them into a flatter one or coordinate them while keeping each intact. This is why merging produces mediocrity and orchestration produces breadth. The fluid intuition predicts that combining should accumulate; the structural reality is that combining either dilutes, through averaging, or coordinates, through routing.

The fluid intuition also underlies the fear and the hope that a combined model would be a step toward some general super-intelligence. If intelligence were additive, combining everything would be a shortcut to more of it. But because it is structural, combining everything just rearranges the same structures without adding a new level of capability. A routed system that draws on every model is genuinely more useful than any single model across a range of tasks, but it is not a new kind of mind; it is the same models, better coordinated. The ceiling on what any single model can do is set by how that model was built, and stapling models together does not raise that ceiling. It widens the range of tasks the system handles well by choosing the right builder for each job.

This reframing matters for expectations. People who imagine that combining all models is a path to a dramatically smarter system are looking in the wrong place. The gains from combination are real but bounded: efficiency, breadth, resilience, and cost control, not a leap in raw capability. The leaps in capability, when they come, come from better individual models, from new architectures and training methods and scale, not from blending the models that already exist. Combination is a way to use what exists better, not a way to transcend it.

There is a modest and useful conclusion in this. The most valuable thing to build is not a merged super-model, which is impossible across the real frontier and mediocre where possible, but a well-designed system that treats the diversity of available models as an asset to coordinate. That system already exists in partial form as the routers, gateways, and mixture-of-experts architectures that run the frontier, and its continued improvement is where the practical value lies. The dream of one model to rule them all dissolves on contact with the mathematics, and what remains is more useful than the dream: an orchestra of distinct instruments, each kept in tune, each playing when its part arrives, coordinated well enough that the whole outperforms any soloist while never pretending to be a single, larger instrument.

That is the honest answer to what would happen if you combined every large language model. Fuse their weights and you get an average, if the arithmetic even runs. Coordinate their outputs and you get an orchestra worth building. Merge the companies that make them and you get a concentration of power worth watching closely. The interesting outcome was never a bigger brain. It was a better system, and a harder set of questions about who controls it.

For anyone building on these models, that reframing turns into a practical discipline. Stop looking for the one model or the one merge that solves everything, because it does not exist, and start treating the field of models as a supply of specialists to coordinate. Route by task, keep the coordination layer model-agnostic so new releases arrive as upgrades, reserve ensembling for the queries where accuracy pays for the extra cost, and merge only within families where the arithmetic is coherent. Measure what the combination actually delivers rather than trusting that it delivers anything, because combination systems fail quietly. And watch the organizational layer as closely as the technical one, since the concentration of the model-makers will shape access, price, and safety more than any clever blend of their outputs ever could. The question that sounded like a thought experiment about intelligence turns out to be a working manual for how to use many models well, and a warning about what happens if too few hands come to hold them.

Common questions about combining large language models

Author:
Jan Bielik
CEO & Founder of Webiano Digital & Marketing Agency

This article is an original analysis supported by the sources cited below

DeepSeek-V3 Technical Report The technical report for a 671-billion-parameter mixture-of-experts model that activates 37 billion parameters per token across 256 routed experts with top-8 selection, and the source for the load-balancing and inference-cost claims used throughout the piece.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity The foundational paper showing that a sparsely activated model can hold enormous total capacity at constant per-token compute, and the origin of the trillion-parameter sparse-versus-dense comparison referenced in the MoE sections.

Qwen3 Technical Report Documentation for one of the strong open-weight model families cited in the 2026 frontier survey, useful for the architecture and active-parameter figures used in the compatibility discussion.

Gemma 3 Technical Report The technical report for a smaller open model family, cited to illustrate the architectural spread that makes cross-family weight merging impossible.

Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts The paper behind the bias-based load-balancing approach used in recent frontier MoE models, supporting the discussion of how routers avoid expert collapse without an auxiliary loss.

Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy Without Increasing Inference Time The origin of the model-soup idea and the evidence that fine-tuned variants of one base model lie in a shared low-loss basin, which underpins the averaging-baseline section.

Editing Models with Task Arithmetic The paper that introduced task vectors and showed capabilities can be added and subtracted through weight-space arithmetic, the basis for the task-arithmetic section and the shared-base requirement.

TIES-Merging: Resolving Interference When Merging Models The source for the trim, elect-sign, and merge procedure and the analysis of redundant-parameter and sign-conflict interference that the interference section relies on.

Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch The paper introducing DARE, which drops and rescales delta parameters, and the source for the redundancy-of-fine-tuning claim behind extreme sparsification.

Arcee’s MergeKit: A Toolkit for Merging Large Language Models The reference implementation paper covering the practical merging methods, the shared-tokenizer and shared-architecture constraints, and the community-merge activity discussed in the tooling passages.

MergeKit open-source repository The maintained codebase documenting supported merge methods including linear averaging, SLERP, task-arithmetic, TIES, and DARE variants, used to ground the description of how these methods are applied in practice.

Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities A broad survey of the model-merging field that supports the taxonomy of methods and the framing of merging as one of several combination strategies.

Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM The paper describing how separately trained expert models can be combined into a single MoE and then finetuned so a router learns to coordinate them, central to the sparse-upcycling and branch-train-mix section.

LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion The source for the ranking-and-fusion ensembling approach and the observation that the best model varies example by example, used in the ensembles section.

Mixture-of-Agents Enhances Large Language Model Capabilities The paper introducing the layered mixture-of-agents pattern where models draft and refine each other’s outputs, and the source for the collaboration-beats-single-draft result.

Together Mixture-of-Agents repository The open implementation of the mixture-of-agents method, cited to show how a layered multi-model orchestration is built in practice.

RouteLLM: Learning to Route LLMs with Preference Data The learned-routing paper behind the cost-versus-quality trade-off and the large reported savings from sending easy queries to cheaper models, used in the routing and economics sections.

DeepSeek-V3 open-source repository The public model checkpoints and code for a leading open-weight MoE system, cited as evidence of the open-weight competition that acts as a counterweight to consolidation.

ST-MoE: Designing Stable and Transferable Sparse Expert Models A study of stability and transfer in sparse expert models that supports the discussion of routing fragility and why experts are difficult to add or remove freely.

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models The precursor method that trains domain experts in parallel and combines them at inference, cited to contrast merge-only combination with router-based coordination.

OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models An open study of MoE training and routing behaviour that supports the memory-cost and expert-utilization points in the mixture-of-experts sections.

Mistral 7B The report for a widely used open model, cited as part of the open-weight field whose architectural diversity the homogenization and consolidation sections depend on.