GPT, short for generative pre-trained transformer, is an architecture and family of language models that generate text by predicting the next token in a sequence, using a decoder-only transformer first pre-trained on very large text corpora and then adapted to specific behaviors. The name describes the recipe: the model is generative, it is pre-trained without task labels, and it is built on the transformer.
The term originated with a series of models introduced from 2018 onward, and it is now also used generically for any large decoder-only language model trained in this way. Not to be confused with the GUID Partition Table used in disk partitioning. GPT-style models underpin most modern chat assistants, coding tools and text generation systems.
How a GPT model generates text
A GPT model reads text as a sequence of tokens, subword units produced by a tokenizer such as byte-pair encoding. Each token is mapped to an embedding vector, combined with positional information and passed through a stack of identical transformer blocks. The final layer outputs a probability distribution over the whole vocabulary for the next token.
Each block contains masked multi-head self-attention and a feed-forward network, wrapped in residual connections and layer normalization. The causal mask ensures that each position attends only to earlier positions, which is what makes the model autoregressive: it generates one token, appends it to the input and repeats.
The parameters that most strongly determine a GPT model’s capability and cost are summarized below.
| Parameter | Typical range | What it affects |
|---|---|---|
| Number of layers | 12 in small models to over 100 in large ones | Depth of representation |
| Model dimension | 768 to more than 12,000 | Capacity per token |
| Attention heads | 12 to over 100 | Parallel attention patterns |
| Context window | A few thousand to over a million tokens | How much text is visible at once |
| Vocabulary size | About 30,000 to over 200,000 tokens | Tokenization efficiency across languages |
| Parameter count | Hundreds of millions to hundreds of billions | Knowledge capacity and inference cost |
Generation is fast in practice because of the key-value cache: attention keys and values for tokens already processed are stored and reused, so each new token requires computation only for itself. The cache grows with context length, which is why long contexts consume large amounts of accelerator memory during inference.
Pre-training, fine-tuning and alignment
GPT models are trained in stages. Pre-training optimizes the next-token prediction objective, a cross-entropy loss, over trillions of tokens of web text, books, code and other sources. This self-supervised stage needs no labels and gives the model broad linguistic and factual patterns.
A pre-trained base model continues text but does not reliably follow instructions. Supervised fine-tuning on curated demonstrations teaches it to respond as an assistant, and preference optimization, such as reinforcement learning from human feedback or direct preference optimization, shapes helpfulness, tone and refusal behavior.
| Stage | Data | Result |
|---|---|---|
| Pre-training | Very large unlabeled corpora | Base model with general knowledge |
| Supervised fine-tuning | Instruction and response examples | Instruction-following behavior |
| Preference optimization | Ranked or compared responses | Responses aligned with preferences |
| Reasoning training | Problems with verifiable answers | Longer, more reliable step-by-step work |
Research on scaling laws showed that pre-training loss falls predictably as parameters, data and compute grow together, which motivated the steady increase in model size and training data.
Controlling generation
At inference time, the model’s probability distribution is turned into text by a decoding strategy. The same model can be precise or creative depending on these settings.
| Setting | Typical values | Effect |
|---|---|---|
| Temperature | 0 to about 1.5 | Lower values make output more deterministic |
| Top-p sampling | 0.8 to 1.0 | Samples only from the most probable mass |
| Top-k sampling | 20 to 100 | Restricts choice to the k likeliest tokens |
| Maximum output tokens | Task-dependent limit | Caps response length and cost |
| Stop sequences | Custom strings | Ends generation at defined markers |
Behavior is further steered by the system prompt and the user prompt. Techniques such as few-shot prompting and chain-of-thought prompting exploit in-context learning, the ability of large GPT models to adapt to a task from examples in the prompt without changing their weights.
GPT compared with other language model designs
Transformer language models come in three main structural variants, and GPT represents the decoder-only branch. Each variant fits different tasks.
| Design | Training objective | Typical use |
|---|---|---|
| Decoder-only, GPT-style | Next-token prediction | Open-ended generation, chat, code |
| Encoder-only, BERT-style | Masked token prediction | Classification, embeddings, search |
| Encoder-decoder, T5-style | Span or sequence reconstruction | Translation, summarization |
The decoder-only design became dominant for general assistants because a single objective scales well and every task can be framed as text continuation. The term LLM is broader than GPT: it includes encoder models, mixture-of-experts variants and open-weight models from many developers.
Limitations and risks of GPT models
Because a GPT model optimizes for plausible continuation rather than verified truth, it can produce fluent but false statements, known as hallucinations. Its knowledge is also frozen at a training cutoff unless it is connected to retrieval or tools.
- Ground answers in sources through retrieval when factual accuracy matters.
- Validate structured output such as JSON or code before using it automatically.
- Guard against prompt injection when the model reads untrusted documents or web content.
- Watch the context window, since quality can degrade for information buried in very long inputs.
- Account for training-data bias, which can surface in tone, assumptions and omissions.
Cost and latency grow with model size and token count, so many applications route simple requests to smaller models and reserve the largest ones for complex tasks.
The idea in brief
A GPT model is a very advanced autocomplete for text. After reading enormous amounts of writing, it has learned which words tend to follow others in almost any situation.
When you ask it something, it writes the answer one small piece at a time, each time choosing a likely next piece. This is why it can write emails, explain ideas or draft code. It is also why it can sound confident when it is wrong: it predicts likely words, not checked facts.
A practical scenario
A software company’s support team receives about 3,000 tickets a week. It deploys a GPT-based assistant that drafts replies. The system prompt defines tone and rules, a retrieval step attaches the three most relevant help articles to each ticket, and the temperature is set to 0.2 so that answers stay consistent and close to the documentation.
Agents review every draft before sending. After two months, the team finds that most drafts need only light edits, while drafts for billing disputes often need rewriting, so those tickets are routed directly to people. The result is shorter response times on routine questions, with human review as the safeguard against confident but incorrect answers.