Model architecture

On this page 7

Model architecture is the structural design of a machine learning model: the types of layers or components it contains, how they are connected, how data flows through them and which parameters the model can learn. Artificial intelligence practitioners treat the architecture as the blueprint that exists before training, while the learned model weights are the values that training writes into that blueprint.

Among the AI fundamentals, the distinction matters because two models with the same architecture can behave very differently after training on different data, and two models with different architectures can reach similar accuracy at very different costs. The architecture sets what the model can represent, how efficiently it learns and how expensive it is to run.

What a model architecture specifies

A model architecture specifies the computational graph of a model: which operations are applied to the input, in what order and with which shapes. It does not specify the numeric values of the weights, which come from training, and it is separate from the training procedure, such as the optimizer, learning rate schedule and loss function.

In practice, an architecture is defined by a small number of design decisions, each of which can be written down in a configuration file. The table below lists the main elements and what each one controls.

ElementWhat it definesTypical example
Layer typesThe basic operations applied to dataConvolution, attention, recurrence, dense
DepthNumber of stacked layers or blocks12 to over 100 blocks
WidthSize of hidden representationsHidden dimension of 768 to 16,384
ConnectivityHow layers pass informationResidual connections, skip paths
Input and output headsHow data enters and predictions leaveEmbedding layer, softmax classifier
Normalization and activationHow signals are scaled and made nonlinearLayer normalization, ReLU, GELU
Core elements of a model architecture

Together these choices fix the parameter count, the memory footprint and the number of operations per input. A configuration that lists depth, width, attention heads and vocabulary size is often enough to rebuild an untrained copy of a published model.

Major families of model architecture

Model architectures fall into recognizable families, each built around an inductive bias, an assumption about the data that the structure encodes. A convolutional network assumes that nearby pixels are related and that patterns can appear anywhere, while a transformer assumes that any element of a sequence may depend on any other.

FamilyCore mechanismBuilt-in assumptionTypical use
Multilayer perceptronStacked dense layersLittle structure in the inputTabular data, small heads
Convolutional networkShared local filtersLocality and translation invarianceImages, audio spectrograms
Recurrent networkState carried across stepsSequential order mattersStreaming signals, older language systems
TransformerSelf-attention over tokensAny position can relate to any otherLanguage, vision, multimodal models
State space modelLinear recurrence with learned dynamicsLong sequences with structured memoryLong documents, genomics
Graph neural networkMessage passing between nodesRelations defined by a graphMolecules, social and road networks
Common architecture families and their built-in assumptions

Real systems frequently combine families. A hybrid architecture may use convolutional stems to process image patches before attention layers, or interleave attention blocks with state space layers to reduce the cost of long contexts. Mixture-of-experts designs add a routing layer that activates only a subset of parameters for each token, which separates the total parameter count from the compute spent per prediction.

Architecture, weights and the trained model

An architecture becomes a usable AI model only after training fills its parameters with learned values. Confusing these layers of description is a frequent source of misunderstanding, for example when people say that two products use “the same model” when they share only an architecture.

ConceptWhat it isChanges when
Model architectureThe structural design and computation graphEngineers redesign layers or connections
HyperparametersNumeric settings of training and sizeTeams tune learning rate, batch size or depth
Model weightsLearned numeric parametersTraining or fine-tuning runs
Trained modelArchitecture plus weightsEither component changes
AI systemModel plus data pipelines, tools and interfacesSurrounding software is updated
How architecture relates to neighboring concepts

Some quantities sit on the boundary. Depth and width are often called architectural hyperparameters because they define the structure yet are tuned like training settings. Neural architecture search automates this tuning by letting an algorithm explore candidate structures, although its high compute cost limits it to well-funded projects or small search spaces.

How architecture shapes cost and capability

The architecture determines the scaling behavior of a model, meaning how accuracy, memory and latency change as data and parameters grow. Standard self-attention, for example, has compute that grows quadratically with sequence length, which is why long-context models adopt sparse, sliding window or linear attention variants.

Hardware also influences which designs succeed. Architectures that map onto dense matrix multiplication run efficiently on GPUs and accelerators, whereas designs with irregular memory access may be elegant on paper yet slow in practice. This observation is closely tied to the bitter lesson: general methods that scale with compute have historically outperformed hand-engineered structure.

Choosing an architecture is therefore a trade-off among accuracy, training budget, inference latency and deployment constraints. The decision table below summarizes typical choices.

SituationReasonable starting pointReason
Tabular business dataGradient-boosted trees or a small MLPLittle spatial or sequential structure to exploit
Image classification on limited dataPretrained convolutional networkStrong locality bias needs fewer examples
Text generation or reasoningDecoder-only transformerProven scaling and broad tooling
Very long sequencesState space or hybrid modelAvoids quadratic attention cost
On-device inferenceCompact or distilled architectureTight memory and power budgets
Choosing an architecture for common situations

Common misconceptions about model architecture

Several recurring misconceptions distort discussions of architecture, especially in product comparisons and public debate. Experienced practitioners tend to correct the following points.

  • A newer architecture is not automatically better. With equal data and compute, well-tuned older designs often close much of the gap.
  • Parameter count is not a full description. Mixture-of-experts models may hold many parameters yet activate only a fraction per token.
  • Architecture does not fix behavior. Training data, objectives and fine-tuning largely decide what a trained model actually does.
  • Copying an architecture does not copy a model. Without the weights and data, a published design yields an untrained network.

A further limitation concerns reproducibility. Small details, such as the placement of normalization layers or the initialization scheme, can change training stability substantially, so architecture descriptions that omit them are often difficult to reproduce faithfully.

In everyday language

A model architecture is like the floor plan of a building. It shows how many rooms there are, how the rooms connect and where people can walk, but the building is still empty. Training is the process of furnishing the rooms so that the building becomes useful.

Two buildings with the same floor plan can feel completely different once furnished, and a clever floor plan can make a building cheaper to heat or easier to move through. In the same way, the architecture decides what a model can learn and how much it costs to run, while training decides what it actually knows.

Seen in the real world

A hospital imaging team wants to flag possible fractures in wrist X-rays and has about 20,000 labeled images. The engineers compare two candidate designs: a vision transformer and a pretrained convolutional network. In trials, the transformer needs more data to stabilize, while the convolutional network, with its built-in locality bias, reaches strong accuracy sooner on the limited dataset.

The team also checks deployment constraints. The scanners connect to a modest on-site server, so the engineers select a compact convolutional architecture that processes an image in well under a second. After training and clinical validation, the model is used as a second reader that prioritizes urgent cases, and radiologists still make every final diagnosis.