Transfer learning

On this page 7

Transfer learning is a machine learning approach in which a model reuses knowledge learned on a source task to perform a different but related target task, instead of being trained from random initialization. In artificial intelligence, it is the reason a single pretrained model can be adapted to medical imaging, legal search or customer support with a fraction of the data and compute that training from scratch would require.

Within model training, transfer learning describes the general principle, while techniques such as fine-tuning, LoRA and distillation are specific ways of carrying it out. The idea rests on a simple observation: features learned from large, broad data (edges and textures in images, syntax and semantics in text) are useful far beyond the task that produced them.

How transfer learning moves knowledge between tasks

Transfer learning works because deep networks learn hierarchical representations. Early layers capture general, reusable patterns, and later layers capture patterns specific to the training objective. When the source and target tasks share structure, the general layers can be kept and only the task-specific part needs new learning.

The process normally has two stages. First, a model is trained on a large source dataset, often through pretraining with a self-supervised objective such as next-token prediction or masked image modeling. Second, the learned weights become the starting point for the target task, where a new output head is attached and some or all parameters are updated on the smaller labeled target dataset.

The table below shows what typically transfers at different depths of a network.

Network depthTypical learned contentTransferability
Early layersEdges, colors, character and subword patternsHigh across most domains
Middle layersShapes, phrases, syntactic structureModerate to high
Late layersTask-specific concepts and class boundariesLow unless tasks are close
Output headMapping to source labels or vocabularyUsually replaced
What different layers contribute to transfer

Main strategies for applying transfer learning

Practitioners choose among several strategies depending on the amount of target data, the similarity between domains and the compute budget. The choice determines how many parameters change and how much of the source knowledge is preserved.

StrategyWhat is updatedBest suited to
Feature extractionOnly a new classifier on frozen featuresVery small datasets, similar domains
Partial fine-tuningUpper layers plus the new headModerate data, moderate domain gap
Full fine-tuningAll model weightsLarge target data or large domain gap
Parameter-efficient tuningSmall adapters such as LoRA matricesLarge models, limited memory
Continued pretrainingAll weights on unlabeled domain textSpecialized vocabulary or style
Common transfer learning strategies

Feature extraction is the cheapest option and the least prone to overfitting, but it cannot adapt the representation itself. Full fine-tuning offers the most flexibility, at the cost of memory, training time and a higher risk of catastrophic forgetting. Parameter-efficient methods sit between the two and have become the default for large language models because they update well under 1 percent of parameters in typical configurations.

Transfer learning is an umbrella concept, and several neighboring terms describe either one way of implementing it or a different learning setup altogether. Confusing them leads to imprecise project plans and mismatched expectations.

ConceptRelationship to transfer learningKey difference
PretrainingProduces the source knowledgeBuilds general representations, not task skills
Fine-tuningMost common transfer methodA specific procedure of updating weights
Multi-task learningShares knowledge across tasksTasks are trained jointly, not in sequence
Continual learningSequential transfer over many tasksMust retain all earlier tasks
DistillationTransfers knowledge between modelsMoves behavior from teacher to student
How transfer learning relates to neighboring terms

A useful distinction is between inductive transfer, where the target task differs from the source task, and transductive transfer, often called domain adaptation, where the task stays the same but the data distribution changes, for example a sentiment model moved from product reviews to clinical notes.

Choosing a source model and configuration

Successful transfer depends mainly on how closely the source domain matches the target. A model pretrained on natural photographs transfers well to retail product images but less well to radiographs or satellite spectra, where texture statistics and channel meanings differ.

  • Prefer a source model whose pretraining data overlaps the target domain in modality, language and style.
  • Use a lower learning rate than in pretraining, commonly 10 to 100 times smaller, so useful weights are not overwritten.
  • Freeze lower layers first and unfreeze gradually when validation results show room for improvement.
  • Hold out a validation set from the target domain and apply early stopping to detect overfitting quickly.
  • Check the license and provenance of the pretrained weights before commercial use.

The following decision table summarizes typical starting points.

SituationRecommended approachReason
Few hundred examples, similar domainFeature extractionLimited data cannot support many updates
Thousands of examples, similar domainPartial or parameter-efficient tuningAdapts features while limiting overfitting
Large data, different domainFull fine-tuningRepresentation itself must change
Unlabeled domain corpus availableContinued pretraining, then tuningAbsorbs domain vocabulary first
Decision guide by data size and domain similarity

Limitations and failure modes

Transfer learning is not automatically beneficial. Negative transfer occurs when source knowledge actively hurts the target task, which happens when the domains are too dissimilar or when spurious correlations from the source data carry over. In such cases a model trained from scratch can outperform the transferred one.

SymptomLikely causeRemedy
Worse accuracy than baselineNegative transfer from distant domainChoose a closer source model or train more layers
Source skills disappearCatastrophic forgettingLower learning rate, adapters or data replay
Validation loss rises earlyOverfitting on small target setFreeze more layers, add data augmentation
Unfair or skewed outputsBias inherited from source dataAudit outputs and rebalance target data
Typical problems, causes and remedies

Inherited bias deserves particular attention: whatever imbalances exist in the pretraining corpus travel with the weights. A further limitation is opacity, since it is rarely clear which source features drive a prediction, so evaluation on the target distribution remains essential.

Stripped to the essentials

Transfer learning means not starting from zero. A computer model that has already learned a great deal from a huge collection of data is taken and taught a new, related job using only a small amount of extra examples.

It resembles a professional pianist learning the organ. The pianist does not relearn how to read music or move the fingers, and only needs to practice what is new about the instrument. In the same way, the model keeps its general knowledge and learns only the specific details, which saves time, data and money.

Where it is applied

A regional hospital wants a system that flags possible pneumonia on chest X-rays, but its radiologists have labeled only about 4,000 images, far too few to train a large vision model from scratch. The data team starts from a convolutional network pretrained on millions of general photographs, then continues pretraining it on a public set of unlabeled chest radiographs to narrow the domain gap.

The team replaces the output layer, freezes the earliest blocks and fine-tunes the rest with a learning rate about 50 times lower than the original. After a few epochs with early stopping, the model reaches clinically useful sensitivity on a held-out test set, a result that the same network trained from random weights failed to approach. The outcome is a working triage tool built in weeks rather than months, with every flagged case still reviewed by a radiologist.