Model weights

Also known as: model parameters

On this page 7

Model weights are the learned numerical parameters of a trained machine learning model: the values that scale and combine inputs inside each layer and therefore determine every output the model produces. Artificial intelligence systems based on neural networks store nearly all of their acquired knowledge in these numbers, which is why the terms model weights and model parameters are often used interchangeably.

A model consists of two separable parts: the architecture, which defines the computation, and the weights, which fill that computation with learned values. The same architecture with different weights behaves like a completely different model, so the weights are the core asset that training produces and that deployment, licensing and security efforts protect.

How model weights are learned during training

Model weights are learned by iterative optimization that adjusts each value to reduce a loss function measured on training data. Training starts from random initialization, computes a prediction, measures the error, and then uses backpropagation to determine how much each individual parameter contributed to that error.

Next, an update rule (stochastic gradient descent, Adam or a similar optimizer) moves each weight a small step against its gradient. The size of that step is governed by the learning rate, typically between 1e-5 and 1e-3 for large networks. Repeating this cycle over billions of examples gradually shapes the weights into a compact representation of statistical patterns in the data.

The following table separates the values a model learns from the settings that engineers choose, a distinction that is frequently blurred in casual discussion.

QuantitySet byChanges during trainingExample
WeightsOptimizerYesMatrix entries in a dense layer
BiasesOptimizerYesOffset added to each neuron output
Normalization scalesOptimizerYesGain values in layer normalization
HyperparametersEngineerNoLearning rate, batch size, layer count
Optimizer stateOptimizerYes, but discarded laterAdam moment estimates
Learned values compared with configured settings

What model weights look like inside a network

Model weights are organized as tensors, multidimensional arrays attached to specific layers. A dense layer mapping 4,096 inputs to 4,096 outputs holds a weight matrix of about 16.8 million values plus a bias vector. Convolutional layers hold small kernels, and transformer layers hold separate projection matrices for queries, keys, values and feed-forward blocks.

Individual weights rarely have an interpretable meaning. Knowledge is distributed across many weights, and a single concept usually corresponds to a pattern of activity rather than to one number. This property, rooted in connectionism, is the main reason why interpretability research remains difficult.

The parameter count of a model is simply the total number of weights and biases, and it grows with layer width and depth as the table shows.

Model classApproximate parametersTypical use
Small image classifier1 to 25 millionMobile vision tasks
BERT-base encoderAbout 110 millionText classification and search
Mid-size language model7 to 13 billionLocal assistants and fine-tuning
Large foundation model70 billion to over 1 trillionGeneral-purpose generation
Typical parameter counts by model class

Storage formats, precision and memory footprint

The memory needed to hold model weights equals the parameter count multiplied by bytes per value. A 7-billion-parameter model occupies about 28 GB in 32-bit floating point, about 14 GB in 16-bit precision and roughly 3.5 to 4 GB at 4-bit quantization. Numerical precision therefore decides whether a model fits on a given accelerator.

Quantization maps weights to fewer bits after or during training. It reduces memory and bandwidth costs considerably, but aggressive quantization can cause measurable accuracy loss, especially on reasoning and long-context tasks. The comparison below summarizes common choices.

FormatBytes per weightTypical roleMain trade-off
FP324Master copy during trainingHighest memory cost
BF16 / FP162Standard training and inferenceSmall rounding error
INT8 / FP81Efficient servingNeeds calibration
4-bit formatsAbout 0.5Consumer hardware inferenceQuality loss on hard tasks
Weight precision formats and their trade-offs

Weights are distributed in files such as safetensors, PyTorch checkpoints, ONNX graphs or the GGUF format used by local inference tools. The safetensors format is preferred for sharing because, unlike pickle-based checkpoints, loading it cannot execute arbitrary code.

Model weights are only one component of a deployable AI model, and several neighboring terms describe different layers of the same system. Confusing them leads to errors in licensing decisions and in claims about openness.

ConceptWhat it isRelationship to weights
Model architectureThe computation graph and layer designDefines the shape of the weights
Training dataExamples used for optimizationSource of what the weights encode
CheckpointSaved snapshot during trainingContains weights plus optimizer state
EmbeddingsVectors produced for inputsComputed using an embedding weight matrix
AI inferenceRunning a trained modelUses frozen weights without updating them
How weights relate to neighboring concepts

A release labeled open weights publishes the parameter files but not necessarily the training data, training code or full documentation. Open-source AI in the strict sense requires more, which is why open weights and open source are not synonyms.

Adapting, protecting and governing weights

Trained weights can be adapted rather than rebuilt. Full fine-tuning updates all parameters on new data, while parameter-efficient methods such as LoRA freeze the base weights and train small low-rank matrices that often amount to well under 1 percent of the original count. Weights from several fine-tuned variants can also be combined through model merging.

Because weights embody the investment of training, they are a high-value security target. Leaked frontier weights cannot be recalled, and anyone holding them can remove safety tuning. Governance frameworks therefore treat weight security as a core control, alongside access logging, encrypted storage and restricted export.

  • Weights can memorize training data, so extraction attacks may reveal private text or code.
  • Malicious checkpoint files can carry executable payloads when loaded with unsafe deserializers.
  • Backdoored weights can behave normally until a hidden trigger appears in the input.
  • License terms attached to weights may restrict commercial use even when files are freely downloadable.

A non-technical view

Think of a large mixing desk with billions of tiny volume knobs. Training is the long process of turning each knob until the desk produces the right sound for every song it hears. The model weights are the final knob positions.

Nobody sets those positions by hand, and no single knob means much by itself. Together, however, they hold everything the system has learned. Copy the knob positions, and you have copied the model. That is why companies guard them carefully and why releasing them publicly is a significant decision.

A concrete example

A hospital research team wants a language model that summarizes discharge notes without sending patient data to an external service. The team downloads an open-weights model with 8 billion parameters in BF16 format, about 16 GB, and verifies that the license permits internal clinical research.

Instead of retraining everything, the engineers apply LoRA to 3,000 anonymized summaries, training adapters of roughly 20 million parameters on a single 48 GB GPU in a few hours. They then quantize the merged weights to 4 bits, reducing the file to about 5 GB so it runs on a workstation inside the hospital network.

Clinicians rate the summaries as accurate in most reviewed cases, while the team keeps the weight files on encrypted storage because the adapters may have memorized fragments of patient records. The weights, not the code, became the asset that required protection.