Foundation model

On this page 7

A foundation model is a large machine learning model pretrained on broad, mostly unlabeled data at scale and designed to be adapted to a wide range of downstream tasks rather than built for a single purpose. In artificial intelligence, the term describes the shared base layer beneath modern chatbots, coding assistants, image generators and speech systems, which are usually specialized versions of one general model.

Among AI fundamentals, the foundation model marks a shift from training one model per task to training one model for many tasks. Its value comes from two properties that emerge with scale: transfer, the reuse of learned representations in new settings, and homogenization, the convergence of many applications on a small number of shared models, which also concentrates their strengths and their defects.

How a foundation model is built

A foundation model is built in two broad stages: expensive general pretraining followed by cheaper task-specific adaptation. Pretraining uses self-supervised learning, in which the training signal is derived from the data itself, such as predicting the next token in a sentence or reconstructing a masked image patch. No human labels are needed, which is what makes training on trillions of tokens feasible.

Most current foundation models use the transformer architecture, whose attention mechanism scales efficiently on parallel hardware. Training runs on clusters of thousands of accelerators for weeks or months, and the result is a set of weights that encodes general-purpose representations of language, images, audio or code. Those weights are then shaped for use through instruction tuning, preference alignment and safety training.

The following table summarizes the main stages of the lifecycle and what each one contributes.

StageTypical dataPurpose
PretrainingWeb text, books, code, images, audioLearn broad representations and world knowledge
Instruction tuningCurated prompt and response pairsTeach the model to follow requests
Preference alignmentHuman or AI rankings of outputsMake answers helpful, honest and safe
Task adaptationDomain documents or labeled examplesSpecialize behavior for one use case
Deployment and inferenceUser prompts and retrieved contextServe predictions in real applications
Lifecycle stages of a foundation model

Key properties: scale, emergence and transfer

The defining property of a foundation model is that capability improves predictably with scale. Empirical scaling laws show that loss falls smoothly as parameter count, dataset size and training compute grow together, and compute-optimal training balances the three rather than maximizing model size alone.

Some abilities appear to arise only beyond a certain scale, such as multi-step arithmetic or in-context learning, where the model performs a new task from a few examples placed in the prompt. Researchers debate whether these emergent abilities are genuine thresholds or partly an artifact of how benchmarks are scored, so the term should be used with care.

The table below lists the parameters that most strongly shape what a foundation model can do and what each one costs.

ParameterTypical rangeWhat it affects
Parameter countAbout 1 billion to over 1 trillionCapacity to store patterns and knowledge
Training tokensHundreds of billions to tens of trillionsBreadth of knowledge and robustness
Training computeRoughly 10^22 to 10^26 operationsOverall capability and cost of the run
Context windowA few thousand to over one million tokensHow much input fits in one request
ModalitiesText only to text, image, audio and videoRange of inputs and outputs supported
Parameters that shape foundation model capability

A foundation model is defined by how it is trained and reused, not by a specific architecture or modality. It overlaps with several neighboring terms, and the distinctions matter when discussing regulation, procurement or system design.

ConceptRelationship to foundation modelsKey difference
Large language modelThe most common type of foundation modelLimited to language, while foundation models may be multimodal
Task-specific modelThe older alternative approachTrained from scratch for one narrow job
Pretrained modelA broader categoryMay be small and transfer only to one domain
Artificial general intelligenceA hypothetical future goalFoundation models are general in scope, not human-level in ability
Narrow AIThe category foundation models still belong toBreadth of tasks does not imply general understanding
How the term relates to neighboring concepts

Policy texts often use the parallel term general-purpose AI model; the EU AI Act, for example, sets transparency and documentation duties for such models and stricter obligations for those with systemic risk. In practice, the two labels describe largely the same class of systems.

Adapting a foundation model to a task

Adaptation methods range from zero-cost prompting to full retraining of all weights, and the right choice depends on data availability, latency, budget and how far the task departs from the pretraining distribution. The decision table below compares the most common options.

MethodWhen it fitsMain trade-off
Prompt engineeringTask is close to general abilityCheap and fast, but sensitive to wording
Retrieval-augmented generationAnswers depend on private or current documentsAdds grounding, but needs a search pipeline
Parameter-efficient fine-tuningConsistent style or format is requiredLow cost, with modest gains on new knowledge
Full fine-tuningLarge labeled dataset and a distinct domainHighest control, high cost and forgetting risk
DistillationLow latency or edge deployment mattersSmaller model with some loss of quality
Choosing an adaptation method

A practical rule is to start with prompting and retrieval, measure quality against a fixed evaluation set, and only fine-tune when those methods plateau. Fine-tuning changes behavior well but is a weak way to add new facts, which retrieval usually handles better.

Limitations and risks of foundation models

Because many products inherit the same base, defects in a foundation model propagate downstream to every application built on it. The most important limitations are well documented in practice:

  • Hallucination: fluent but false statements produced with apparent confidence.
  • Bias inherited from training data, which can surface in hiring, lending or content decisions.
  • Knowledge cutoff: the model knows nothing after its training data ends unless given retrieval.
  • Opacity: internal reasoning is difficult to inspect, which complicates audits and accountability.
  • High resource cost for training and serving, which concentrates development among a few organizations.
  • Data provenance questions about copyright, consent and personal data in the training corpus.

A common misconception is that a larger foundation model is always the better choice. For narrow, high-volume tasks, a smaller adapted model is often cheaper, faster and equally accurate, and evaluation on the real task should decide rather than general benchmark rankings.

In plain words

A foundation model is like a person who has read an enormous library before starting any job. That reading does not make the person an expert in tax law or medicine. It does give them a strong general base. Learning a specific job then goes much faster.

Companies take this one well-read model and give it short extra training or clear instructions. One firm uses it to answer customer emails. Another uses it to summarize contracts. The key idea is simple: build the general model once, then reuse it for many tasks. Its mistakes are reused too.

A real-world example

A regional insurance company wants to speed up claims handling, where adjusters spend about 40 minutes per file reading reports, photos and policy terms. Instead of training separate models for document reading, image review and letter drafting, the team adopts one multimodal foundation model and adapts it in stages.

First, the team connects the model to the policy library through retrieval-augmented generation, so each summary cites the clauses that apply. Then it applies parameter-efficient fine-tuning on 3,000 past claim summaries written by senior adjusters to match the required format. An evaluation set of 500 closed claims is used to check accuracy before launch, and every output still goes to a human adjuster for approval.

After three months, average reading time falls to about 15 minutes per file, and the team reports fewer missed policy exclusions. The same base model later supports a customer email assistant with only new prompts and retrieval sources, which shows the reuse advantage of a foundation model in practice.