Early fusion

Also known as: feature-level fusion

On this page 7

Early fusion is a strategy in multimodal AI that combines inputs from several modalities at the start of processing, so that one shared model learns from the joint representation rather than from separate streams. It is also called feature-level fusion, because the combination happens on raw signals or low-level features such as pixels, audio frames, sensor readings or token embeddings.

The defining property is that cross-modal interactions can be learned from the first layer onward. This gives early fusion its main strength, the ability to capture fine-grained correlations between modalities, and its main weakness, a strong dependence on aligned, complete and comparably scaled inputs. Early fusion sits at one end of the multimodal fusion spectrum, opposite late fusion, with intermediate or hybrid designs in between.

How early fusion combines modalities

Early fusion works by mapping every modality into a common input format and merging the results before the main model processes them. In classical machine learning, the merge is usually feature concatenation: a vector of audio features and a vector of visual features are joined into one longer vector that a classifier receives. In modern deep learning, the merge is more often token-level interleaving, where image patches, text tokens and audio frames are projected into the same embedding space and fed to one transformer as a single sequence.

The typical pipeline has four stages. Each one introduces its own design decisions, and errors at an early stage propagate through the whole model because there is no modality-specific path to fall back on.

StageWhat happensTypical technique
SynchronizationInputs are aligned in time or spaceResampling, timestamp matching, frame windows
NormalizationValue ranges are made comparableStandardization, per-modality scaling
ProjectionEach modality enters a shared dimensionLinear layers, patch embedding, tokenizers
Joint modelingOne network learns from the merged inputTransformer, convolutional or recurrent model
Stages of an early fusion pipeline

In token-based systems, the joint transformer attends across all modalities in every layer. A word token can attend directly to an image patch in the first block, which is exactly the kind of low-level interaction that later fusion strategies cannot express until much deeper in the network.

Variants of early fusion

Early fusion is not a single architecture but a family of input-level merging methods. They differ in what is merged and how much structure each modality keeps before the merge.

VariantMerged representationTypical use
Raw signal stackingChannels of aligned sensorsRGB-D images, multispectral imagery
Feature concatenationHandcrafted or shallow featuresClassical emotion and sentiment models
Token interleavingDiscrete or continuous tokens in one sequenceNatively multimodal transformers
Discrete unified vocabularyImage and audio codes in the text vocabularyModels that generate several modalities
Common forms of early fusion

Raw signal stacking is the oldest form: a depth map is added as a fourth channel to an RGB image, and a convolutional network treats the result as one tensor. Token interleaving is the dominant form in current research, because an image tokenizer or vision encoder can turn pictures into visual tokens that sit next to text tokens. When the image tokens come from a discrete codebook, the model can predict them just like words, which is the basis of several any-to-any designs.

Early fusion compared with late and intermediate fusion

Early fusion differs from other strategies mainly in where the modalities meet. Late fusion trains or runs separate models per modality and combines their outputs, while intermediate fusion merges learned representations somewhere in the middle, often through cross-attention.

PropertyEarly fusionIntermediate fusionLate fusion
Integration pointInput or first layersHidden layersPredictions or scores
Cross-modal interactionFine-grained, from layer oneModerate, at chosen layersCoarse, decision level only
Alignment requirementStrictModerateLoose
Robustness to missing modalityLow without special trainingMediumHigh
Reuse of pretrained unimodal modelsLimitedGoodExcellent
Training costHigh, one large joint modelMediumLow per model
Fusion strategies by point of integration

The comparison explains a recurring pattern in multimodal AI. Many vision-language models use intermediate fusion through a multimodal projector because it reuses a strong pretrained language model cheaply. Natively multimodal models that train on interleaved data from scratch move toward true early fusion, accepting higher pretraining cost in exchange for deeper joint understanding.

When early fusion is the right choice

Early fusion is the right choice when the useful signal lies in low-level correlations between modalities. Lip movements and speech sounds, depth and color edges, or the position of a word on a scanned page are all examples where one modality changes the meaning of the other at a fine scale.

SituationRecommended approachReason
Tightly synchronized sensorsEarly fusionFine correlations are preserved
Frequently missing modalitiesLate fusionEach model works alone
Strong pretrained unimodal models availableIntermediate fusionReuse lowers training cost
Large interleaved pretraining corpusToken-level early fusionJoint training scales well
Small labeled datasetLate or intermediate fusionFewer parameters to fit jointly
Choosing a fusion point for common situations

Practitioners also consider compute and sequence length. Interleaving hundreds of visual tokens per image with text makes the joint sequence long, and attention cost grows roughly quadratically with sequence length. Token compression, patch merging and lower image resolution are common countermeasures.

Limitations and common mistakes

The most serious limitation of early fusion is brittleness when one modality is missing, noisy or misaligned. Because the model never learned to rely on one modality alone, a dropped camera feed or a desynchronized audio track can degrade results sharply.

  • Ignoring scale differences: unnormalized features let one modality dominate the gradients.
  • Dimensional imbalance: a high-dimensional modality can drown out a compact one in simple concatenation.
  • Modality laziness: the model may learn to rely on the easiest modality and ignore the others, a form of shortcut learning.
  • Skipping modality dropout: without randomly masking modalities during training, robustness to missing inputs stays poor.
  • Assuming more fusion is always better: on small datasets, early fusion often overfits compared with late fusion.

A frequent misconception is that early fusion always produces better multimodal reasoning. Empirical results depend heavily on data volume, alignment quality and the task; with weak alignment or limited data, simpler fusion strategies frequently match or outperform it.

Stripped to the essentials

Early fusion means mixing different kinds of information together right at the start, before the computer tries to understand any of it. Instead of having one model look at a picture and another read the text and then comparing notes, a single model looks at both at once.

Think of tasting a soup. You can taste each ingredient separately and then guess the final flavor, or you can cook everything together and taste the result. Early fusion is the second approach: the ingredients influence each other from the beginning, which reveals combinations you would otherwise miss. The catch is that if one ingredient is missing or spoiled, the whole pot suffers.

Where it is applied

A robotics team building a warehouse picking arm uses a camera that delivers aligned color and depth images at 30 frames per second. Instead of running one network on color and another on depth, the engineers stack the depth map as a fourth channel next to the red, green and blue channels and train a single convolutional network on the combined tensor.

Because edges in the depth map coincide pixel by pixel with color edges, the network learns to separate transparent plastic packaging from the shelf behind it, a case where color alone is ambiguous. The team also applies modality dropout during training, randomly zeroing the depth channel in a share of batches, so the arm keeps working when the depth sensor briefly fails on reflective surfaces. The result is more reliable grasp detection than either single-modality model achieved on the same test set.