A base model is a pretrained language model that has learned only to predict the next token in large volumes of text and has not yet received instruction tuning, preference optimization or safety training. In the field of language models and artificial intelligence, it is the raw foundation from which chat assistants, code models and domain-specific systems are later derived.
Because a base model was trained to continue documents rather than answer requests, it behaves like an extremely well-read autocomplete engine. Given a question, it may answer it, generate more questions, or continue as if the text were a forum thread. This makes a base model powerful but unpredictable, and it explains why practitioners also call it a pretrained base model or simply a pretrained checkpoint.
How a base model is created during pretraining
A base model is created through self-supervised pretraining, in which a neural network, almost always a decoder-only transformer, learns to assign probabilities to the next token given all previous tokens. No human labels are required: the text itself supplies the target, so the corpus can scale to trillions of tokens drawn from web pages, books, code repositories and scientific papers.
The training objective is cross-entropy loss on next-token prediction. Minimizing this loss forces the network to internalize grammar, facts, reasoning patterns and stylistic conventions, because all of them help predict what comes next. The resulting weights store a large amount of parametric knowledge, bounded by the knowledge cutoff of the training data.
The table below summarizes the main pretraining ingredients and what each one determines in the finished base model.
| Ingredient | Typical scale or choice | What it determines |
|---|---|---|
| Training corpus | Hundreds of billions to tens of trillions of tokens | Breadth of knowledge, languages and domains |
| Tokenizer | Byte-pair encoding, 32,000 to 260,000 vocabulary entries | Efficiency per language and handling of code |
| Parameter count | Under 1 billion to several hundred billion | Capacity to store patterns and facts |
| Context length | 2,048 to over 100,000 tokens | How much text the model can condition on |
| Data mixture | Ratios of web, code, math and books | Relative strengths such as coding or reasoning |
Base model versus instruction-tuned and chat models
A base model differs from an instruction-tuned model in what it has been optimized to do: the base model imitates the distribution of its training text, while an instruction-tuned model has been further trained to follow user instructions in a specific conversational format. Most commercial assistants pass through several post-training stages after the base checkpoint is frozen.
Typical post-training includes supervised fine-tuning on curated prompt-response pairs, followed by preference optimization such as reinforcement learning from human feedback or direct preference optimization. These stages add a chat template, refusal behavior and a consistent assistant persona, none of which exist in the base model.
| Stage | Training signal | Resulting behavior |
|---|---|---|
| Base model | Next-token prediction on raw text | Continues any document, no fixed persona |
| Instruction-tuned model | Supervised prompt and response pairs | Answers requests in a predictable format |
| Preference-optimized model | Human or AI preference comparisons | Helpful, cautious and stylistically consistent |
| Reasoning model | Reinforcement learning on verifiable tasks | Produces extended reasoning before answering |
A frequent misconception is that a base model is simply a weaker assistant. In fact, post-training adds little new knowledge; it mainly shapes how existing capabilities are expressed. Many benchmark scores on knowledge-heavy tasks are already present in the base model when it is prompted carefully.
How to prompt a base model effectively
A base model is prompted effectively by writing the beginning of a document whose natural continuation is the desired output. Instead of issuing a command, the user frames the task as a text the model has seen many times, such as a question and answer page, a table, a code file or a set of worked examples.
Few-shot prompting is the most reliable technique, because demonstrations establish both the task and the output format through in-context learning. Clear delimiters and stop sequences are essential, since a base model has no built-in notion of when an answer is finished and will otherwise keep generating.
| Goal | Prompt framing | Why it works |
|---|---|---|
| Classification | Several labeled examples, then an unlabeled one | The pattern makes the label the likeliest next token |
| Summarization | Article text followed by the line TL;DR: | Forum conventions associate this marker with summaries |
| Code completion | Function signature and docstring | Code repositories pair docstrings with implementations |
| Data extraction | Partially filled table or JSON object | The model continues the established structure |
Sampling settings matter more than with chat models. A moderate temperature between 0.7 and 1.0 preserves diversity for creative continuation, while greedy decoding or low temperature suits extraction. A repetition penalty can prevent the loops that base models fall into more readily than tuned models.
When teams choose a base model
Teams choose a base model when they need full control over downstream behavior rather than the defaults imposed by a vendor’s post-training. Base checkpoints are most common in the open-weight model ecosystem, where developers download the pretrained weights and run their own fine-tuning.
- Custom fine-tuning: starting from a base model avoids inheriting refusal patterns or a chat persona that conflict with a specialized application.
- Domain adaptation: continued pretraining on medical, legal or financial text works best on a checkpoint that has not been narrowed by instruction tuning.
- Research and evaluation: base models reveal what was learned in pretraining, separate from post-training effects, and give better-calibrated logprobs for scoring tasks.
- Synthetic data and creative text: base models produce more varied, less formulaic output, because preference tuning tends to reduce output diversity.
Limitations and risks of base models
The main limitation of a base model is that it lacks alignment and safety training, so it reproduces whatever patterns appear in its corpus, including toxic language, misinformation and private data that leaked into the training set. It should never be exposed directly to end users without additional safeguards.
| Symptom | Cause | Mitigation |
|---|---|---|
| Endless generation | No learned end of answer | Set stop sequences and max tokens |
| Ignores the request | Instruction framing is rare in raw text | Rewrite the prompt as a document continuation |
| Fluent but false claims | Imitation of plausible text, not truth | Ground with retrieval and verify outputs |
| Harmful or biased content | Unfiltered patterns from web data | Add moderation and fine-tune before release |
Hallucination is not worse in principle in a base model, yet it is harder to detect, because the model gives no signal of uncertainty and does not decline questions. Base models are also sensitive to small prompt changes, so results should be evaluated across several phrasings before conclusions are drawn.
In simple terms
A base model is like a person who has read an enormous library but has never been taught how to hold a conversation. If you start a sentence, it can finish it brilliantly. If you ask it a question, it might answer, or it might write ten more questions, because it is only guessing what text usually comes next.
Companies take this raw reader and give it extra training so it behaves like a helpful assistant. The base model holds the knowledge; later training teaches the manners. That is why developers who want to build something very specific often prefer to start from the base model itself.
A practical example
A hospital research group wants a model that drafts structured radiology report summaries. A general chat assistant keeps adding polite introductions and disclaimers that break the strict report format, so the team downloads an 8-billion-parameter open-weight base model instead.
They first run continued pretraining on roughly 2 billion tokens of de-identified clinical text, then fine-tune on 15,000 report and summary pairs using a fixed template with an explicit end marker. Because the base model carries no assistant persona, the tuned model learns the house format quickly and stops cleanly at the marker.
In internal review, radiologists accept most drafts after minor edits, and the team keeps a human sign-off step for every report. The outcome is a specialized model that follows the clinical format exactly, something that would have been harder to achieve by prompting a general chat model.