Emergent abilities are capabilities that are absent or near chance in smaller language models but appear, often abruptly, once a model crosses a certain scale of parameters, training data or compute. In artificial intelligence research, the term describes skills such as multi-step arithmetic, following novel instructions or answering questions that require chained reasoning, which cannot be predicted by simply extrapolating the performance curve of smaller models.
The concept matters because it shapes how laboratories plan training runs, how evaluators design benchmarks and how policymakers reason about unpredictable capability jumps. It is also contested: a significant body of work argues that many apparent jumps are partly an artifact of the evaluation metric rather than a genuine discontinuity inside the model. The term is sometimes used interchangeably with emergent capabilities.
How emergence is defined and measured
An ability is called emergent when a plot of task performance against model scale shows a flat region near random accuracy followed by a sharp rise above some threshold. Scale is usually expressed as training compute in FLOPs or as parameter count, and performance is measured on a fixed benchmark with a fixed prompt format. The defining feature is not that the ability is large, but that it is not predictable from smaller models.
This contrasts with the smooth behavior described by neural scaling laws, where pretraining loss falls as a power law of compute. A model can therefore improve steadily in next-token prediction loss while a downstream task metric stays flat and then jumps. The gap between a smooth internal quantity and a jumpy external metric is the core of both the phenomenon and the debate around it.
The table below summarizes the quantities that researchers track when they claim or test emergence.
| Quantity | What it captures | Typical behavior with scale |
|---|---|---|
| Pretraining loss | Average next-token cross-entropy | Smooth power-law decline |
| Exact-match accuracy | All tokens of an answer correct | Often flat, then sudden rise |
| Multiple-choice accuracy | Highest-scored option is correct | Near chance, then jump |
| Token-level log-likelihood | Probability of the correct answer | Usually gradual improvement |
| Brier score or calibrated metrics | Quality of probability estimates | Mostly continuous |
Commonly cited examples of emergent abilities
Most reported examples come from few-shot evaluations of large decoder-only models, where the model sees a handful of demonstrations in the prompt and must generalize. Several of these abilities are closely tied to prompting techniques such as chain-of-thought prompting, which itself only helps once a model is large enough; in small models it can even reduce accuracy.
The scale thresholds in the following table are approximate and model-family dependent. They shift with data quality, tokenization, instruction tuning and prompt design, so they should be read as orders of magnitude rather than fixed constants.
| Ability | Example task | Rough onset scale |
|---|---|---|
| Multi-digit arithmetic | Three-digit addition without tools | Tens of billions of parameters |
| Chain-of-thought benefit | Grade-school math word problems | Roughly 10^23 training FLOPs |
| In-context learning | Learning a new mapping from examples | Gradual onset, sharper for hard tasks |
| Instruction following | Executing unseen task descriptions | Strongly aided by instruction tuning |
| Word unscrambling | Recovering a word from shuffled letters | Large models, metric dependent |
Some behaviors described as emergent are better explained by post-training than by pretraining scale. Instruction following and refusal behavior, for example, depend heavily on supervised fine-tuning and preference optimization, which means scale alone is not the whole story.
The mirage argument and metric choice
The strongest critique holds that many emergent abilities are a measurement mirage. If a task is scored with a nonlinear or discontinuous metric such as exact match, a model whose per-token accuracy rises smoothly will still score zero until nearly every token is right, and then appear to jump. Re-scoring the same outputs with a continuous metric, such as token edit distance or log-probability of the correct answer, often reveals steady progress.
A second contributor is coarse sampling of model sizes. When only four or five model sizes are evaluated, a smooth sigmoid can look like a step. Small evaluation sets add noise that hides early gains. Neither point proves that all emergence is illusory, but together they mean that a single sharp curve is weak evidence of a true discontinuity.
The comparison below contrasts the two main interpretations.
| Aspect | Genuine emergence view | Metric artifact view |
|---|---|---|
| Source of the jump | Qualitative change in learned circuits | Thresholded scoring of smooth gains |
| Continuous metrics | Still show a kink | Show steady improvement |
| Predictability | Hard to forecast from small models | Forecastable with the right metric |
| Implication for safety | Surprise capabilities are plausible | Surprises are mostly avoidable |
Current evidence supports a middle position: many headline jumps soften under continuous metrics, yet some tasks that require composing several sub-skills still show sharp transitions, because success requires every component to work at once.
Mechanisms proposed to explain emergence
Researchers have proposed several mechanisms for real capability transitions. The compositional explanation states that a task needing k independent sub-skills succeeds with probability roughly equal to the product of each sub-skill accuracy, so overall success stays low until every component is reliable. This produces a sigmoid-shaped curve even when each sub-skill improves smoothly.
Interpretability work points to circuit formation. In small transformers, induction heads that copy patterns from earlier context form during a short phase of training, coinciding with a visible bump in loss and a sudden gain in in-context learning. Related findings on grokking, where generalization appears long after memorization, show that phase transitions in training dynamics are real, although their link to large-scale emergence is still being studied.
Practical consequences for building and evaluating models
For practitioners, emergent abilities change how capability forecasting and model selection are done. A small or distilled model that fails a task outright is not proof that a larger sibling will also fail, and a small language model may lack abilities that its bigger counterpart displays reliably. The decision table below translates the research into working rules.
| Situation | Recommended practice | Reason |
|---|---|---|
| Forecasting a larger run | Track continuous metrics on small models | Reveals progress hidden by exact match |
| Choosing a deployment model | Test the exact task at target size | Abilities may not transfer downward |
| Designing a benchmark | Report several metrics and many sizes | Avoids false steps and false plateaus |
| Safety evaluation | Probe dangerous skills before release | Jumps can surprise developers |
| Using chain-of-thought | Verify benefit on the chosen model | Small models may lose accuracy |
Frontier safety frameworks explicitly reference unexpected capability gains and require dangerous-capability evaluations at regular compute intervals. Their reasoning is simple: if a new ability can appear between two checkpoints, testing only the final model leaves too little time to respond.
Common misconceptions about emergent abilities
- Emergence is not consciousness or agency; it refers only to task performance curves.
- Emergence is not magic: the underlying training objective is still next-token prediction.
- A jump on one benchmark does not imply a general jump in intelligence.
- Thresholds are not universal; better data or tuning can move them to smaller models.
- Not every surprising output is emergent; many are hallucinations or memorized patterns.
Understanding it simply
Emergent abilities are skills that a language model seems to gain suddenly when it becomes big enough. A smaller model fails the task almost completely, and a larger one suddenly handles it well. The jump surprises people because the earlier, smaller models gave no clear hint that it was coming.
A useful comparison is a team spelling a long word aloud, one letter per person. If each person is only slightly reliable, the whole word almost always comes out wrong. As each person gets a bit better, the word suddenly starts coming out right. Each member improved gradually, but the result looked sudden. Scientists still debate how much of emergence is this kind of effect and how much is a real change inside the model.
Applied in practice
An evaluation team at a software company is choosing a model to extract totals from scanned invoices and check whether line items add up. They test a family of models at roughly 1, 8 and 70 billion parameters. Scored by exact match on the final total, the two smaller models reach only about 5 percent and 12 percent, while the largest reaches about 85 percent, which looks like a classic emergent jump.
Before deciding, the team re-scores the outputs with a continuous metric: the share of line items extracted correctly and the absolute error of each sum. The 8-billion model turns out to read most line items correctly but makes one arithmetic slip in most invoices. Adding a simple calculator tool to its pipeline lifts exact-match accuracy to about 80 percent. The team deploys the mid-sized model with tool support, cutting inference cost substantially while keeping accuracy close to the largest model.