Base model

Also known as: pretrained base model

On this page 7

A base model is a pretrained language model that has learned only to predict the next token in large volumes of text and has not yet received instruction tuning, preference optimization or safety training. In the field of language models and artificial intelligence, it is the raw foundation from which chat assistants, code models and domain-specific systems are later derived.

Because a base model was trained to continue documents rather than answer requests, it behaves like an extremely well-read autocomplete engine. Given a question, it may answer it, generate more questions, or continue as if the text were a forum thread. This makes a base model powerful but unpredictable, and it explains why practitioners also call it a pretrained base model or simply a pretrained checkpoint.

How a base model is created during pretraining

A base model is created through self-supervised pretraining, in which a neural network, almost always a decoder-only transformer, learns to assign probabilities to the next token given all previous tokens. No human labels are required: the text itself supplies the target, so the corpus can scale to trillions of tokens drawn from web pages, books, code repositories and scientific papers.

The training objective is cross-entropy loss on next-token prediction. Minimizing this loss forces the network to internalize grammar, facts, reasoning patterns and stylistic conventions, because all of them help predict what comes next. The resulting weights store a large amount of parametric knowledge, bounded by the knowledge cutoff of the training data.

The table below summarizes the main pretraining ingredients and what each one determines in the finished base model.

IngredientTypical scale or choiceWhat it determines
Training corpusHundreds of billions to tens of trillions of tokensBreadth of knowledge, languages and domains
TokenizerByte-pair encoding, 32,000 to 260,000 vocabulary entriesEfficiency per language and handling of code
Parameter countUnder 1 billion to several hundred billionCapacity to store patterns and facts
Context length2,048 to over 100,000 tokensHow much text the model can condition on
Data mixtureRatios of web, code, math and booksRelative strengths such as coding or reasoning
Pretraining ingredients of a base model

Base model versus instruction-tuned and chat models

A base model differs from an instruction-tuned model in what it has been optimized to do: the base model imitates the distribution of its training text, while an instruction-tuned model has been further trained to follow user instructions in a specific conversational format. Most commercial assistants pass through several post-training stages after the base checkpoint is frozen.

Typical post-training includes supervised fine-tuning on curated prompt-response pairs, followed by preference optimization such as reinforcement learning from human feedback or direct preference optimization. These stages add a chat template, refusal behavior and a consistent assistant persona, none of which exist in the base model.

StageTraining signalResulting behavior
Base modelNext-token prediction on raw textContinues any document, no fixed persona
Instruction-tuned modelSupervised prompt and response pairsAnswers requests in a predictable format
Preference-optimized modelHuman or AI preference comparisonsHelpful, cautious and stylistically consistent
Reasoning modelReinforcement learning on verifiable tasksProduces extended reasoning before answering
How the model changes across training stages

A frequent misconception is that a base model is simply a weaker assistant. In fact, post-training adds little new knowledge; it mainly shapes how existing capabilities are expressed. Many benchmark scores on knowledge-heavy tasks are already present in the base model when it is prompted carefully.

How to prompt a base model effectively

A base model is prompted effectively by writing the beginning of a document whose natural continuation is the desired output. Instead of issuing a command, the user frames the task as a text the model has seen many times, such as a question and answer page, a table, a code file or a set of worked examples.

Few-shot prompting is the most reliable technique, because demonstrations establish both the task and the output format through in-context learning. Clear delimiters and stop sequences are essential, since a base model has no built-in notion of when an answer is finished and will otherwise keep generating.

GoalPrompt framingWhy it works
ClassificationSeveral labeled examples, then an unlabeled oneThe pattern makes the label the likeliest next token
SummarizationArticle text followed by the line TL;DR:Forum conventions associate this marker with summaries
Code completionFunction signature and docstringCode repositories pair docstrings with implementations
Data extractionPartially filled table or JSON objectThe model continues the established structure
Prompt patterns that work with a base model

Sampling settings matter more than with chat models. A moderate temperature between 0.7 and 1.0 preserves diversity for creative continuation, while greedy decoding or low temperature suits extraction. A repetition penalty can prevent the loops that base models fall into more readily than tuned models.

When teams choose a base model

Teams choose a base model when they need full control over downstream behavior rather than the defaults imposed by a vendor’s post-training. Base checkpoints are most common in the open-weight model ecosystem, where developers download the pretrained weights and run their own fine-tuning.

  • Custom fine-tuning: starting from a base model avoids inheriting refusal patterns or a chat persona that conflict with a specialized application.
  • Domain adaptation: continued pretraining on medical, legal or financial text works best on a checkpoint that has not been narrowed by instruction tuning.
  • Research and evaluation: base models reveal what was learned in pretraining, separate from post-training effects, and give better-calibrated logprobs for scoring tasks.
  • Synthetic data and creative text: base models produce more varied, less formulaic output, because preference tuning tends to reduce output diversity.

Limitations and risks of base models

The main limitation of a base model is that it lacks alignment and safety training, so it reproduces whatever patterns appear in its corpus, including toxic language, misinformation and private data that leaked into the training set. It should never be exposed directly to end users without additional safeguards.

SymptomCauseMitigation
Endless generationNo learned end of answerSet stop sequences and max tokens
Ignores the requestInstruction framing is rare in raw textRewrite the prompt as a document continuation
Fluent but false claimsImitation of plausible text, not truthGround with retrieval and verify outputs
Harmful or biased contentUnfiltered patterns from web dataAdd moderation and fine-tune before release
Common problems when deploying a base model

Hallucination is not worse in principle in a base model, yet it is harder to detect, because the model gives no signal of uncertainty and does not decline questions. Base models are also sensitive to small prompt changes, so results should be evaluated across several phrasings before conclusions are drawn.

In simple terms

A base model is like a person who has read an enormous library but has never been taught how to hold a conversation. If you start a sentence, it can finish it brilliantly. If you ask it a question, it might answer, or it might write ten more questions, because it is only guessing what text usually comes next.

Companies take this raw reader and give it extra training so it behaves like a helpful assistant. The base model holds the knowledge; later training teaches the manners. That is why developers who want to build something very specific often prefer to start from the base model itself.

A practical example

A hospital research group wants a model that drafts structured radiology report summaries. A general chat assistant keeps adding polite introductions and disclaimers that break the strict report format, so the team downloads an 8-billion-parameter open-weight base model instead.

They first run continued pretraining on roughly 2 billion tokens of de-identified clinical text, then fine-tune on 15,000 report and summary pairs using a fixed template with an explicit end marker. Because the base model carries no assistant persona, the tuned model learns the house format quickly and stops cleanly at the marker.

In internal review, radiologists accept most drafts after minor edits, and the team keeps a human sign-off step for every report. The outcome is a specialized model that follows the clinical format exactly, something that would have been harder to achieve by prompting a general chat model.