Emergent abilities

Also known as: emergent capabilities

On this page 8

Emergent abilities are capabilities that are absent or near chance in smaller language models but appear, often abruptly, once a model crosses a certain scale of parameters, training data or compute. In artificial intelligence research, the term describes skills such as multi-step arithmetic, following novel instructions or answering questions that require chained reasoning, which cannot be predicted by simply extrapolating the performance curve of smaller models.

The concept matters because it shapes how laboratories plan training runs, how evaluators design benchmarks and how policymakers reason about unpredictable capability jumps. It is also contested: a significant body of work argues that many apparent jumps are partly an artifact of the evaluation metric rather than a genuine discontinuity inside the model. The term is sometimes used interchangeably with emergent capabilities.

How emergence is defined and measured

An ability is called emergent when a plot of task performance against model scale shows a flat region near random accuracy followed by a sharp rise above some threshold. Scale is usually expressed as training compute in FLOPs or as parameter count, and performance is measured on a fixed benchmark with a fixed prompt format. The defining feature is not that the ability is large, but that it is not predictable from smaller models.

This contrasts with the smooth behavior described by neural scaling laws, where pretraining loss falls as a power law of compute. A model can therefore improve steadily in next-token prediction loss while a downstream task metric stays flat and then jumps. The gap between a smooth internal quantity and a jumpy external metric is the core of both the phenomenon and the debate around it.

The table below summarizes the quantities that researchers track when they claim or test emergence.

QuantityWhat it capturesTypical behavior with scale
Pretraining lossAverage next-token cross-entropySmooth power-law decline
Exact-match accuracyAll tokens of an answer correctOften flat, then sudden rise
Multiple-choice accuracyHighest-scored option is correctNear chance, then jump
Token-level log-likelihoodProbability of the correct answerUsually gradual improvement
Brier score or calibrated metricsQuality of probability estimatesMostly continuous
Quantities used to study emergent abilities

Commonly cited examples of emergent abilities

Most reported examples come from few-shot evaluations of large decoder-only models, where the model sees a handful of demonstrations in the prompt and must generalize. Several of these abilities are closely tied to prompting techniques such as chain-of-thought prompting, which itself only helps once a model is large enough; in small models it can even reduce accuracy.

The scale thresholds in the following table are approximate and model-family dependent. They shift with data quality, tokenization, instruction tuning and prompt design, so they should be read as orders of magnitude rather than fixed constants.

AbilityExample taskRough onset scale
Multi-digit arithmeticThree-digit addition without toolsTens of billions of parameters
Chain-of-thought benefitGrade-school math word problemsRoughly 10^23 training FLOPs
In-context learningLearning a new mapping from examplesGradual onset, sharper for hard tasks
Instruction followingExecuting unseen task descriptionsStrongly aided by instruction tuning
Word unscramblingRecovering a word from shuffled lettersLarge models, metric dependent
Abilities often described as emergent

Some behaviors described as emergent are better explained by post-training than by pretraining scale. Instruction following and refusal behavior, for example, depend heavily on supervised fine-tuning and preference optimization, which means scale alone is not the whole story.

The mirage argument and metric choice

The strongest critique holds that many emergent abilities are a measurement mirage. If a task is scored with a nonlinear or discontinuous metric such as exact match, a model whose per-token accuracy rises smoothly will still score zero until nearly every token is right, and then appear to jump. Re-scoring the same outputs with a continuous metric, such as token edit distance or log-probability of the correct answer, often reveals steady progress.

A second contributor is coarse sampling of model sizes. When only four or five model sizes are evaluated, a smooth sigmoid can look like a step. Small evaluation sets add noise that hides early gains. Neither point proves that all emergence is illusory, but together they mean that a single sharp curve is weak evidence of a true discontinuity.

The comparison below contrasts the two main interpretations.

AspectGenuine emergence viewMetric artifact view
Source of the jumpQualitative change in learned circuitsThresholded scoring of smooth gains
Continuous metricsStill show a kinkShow steady improvement
PredictabilityHard to forecast from small modelsForecastable with the right metric
Implication for safetySurprise capabilities are plausibleSurprises are mostly avoidable
Competing interpretations of capability jumps

Current evidence supports a middle position: many headline jumps soften under continuous metrics, yet some tasks that require composing several sub-skills still show sharp transitions, because success requires every component to work at once.

Mechanisms proposed to explain emergence

Researchers have proposed several mechanisms for real capability transitions. The compositional explanation states that a task needing k independent sub-skills succeeds with probability roughly equal to the product of each sub-skill accuracy, so overall success stays low until every component is reliable. This produces a sigmoid-shaped curve even when each sub-skill improves smoothly.

Interpretability work points to circuit formation. In small transformers, induction heads that copy patterns from earlier context form during a short phase of training, coinciding with a visible bump in loss and a sudden gain in in-context learning. Related findings on grokking, where generalization appears long after memorization, show that phase transitions in training dynamics are real, although their link to large-scale emergence is still being studied.

Practical consequences for building and evaluating models

For practitioners, emergent abilities change how capability forecasting and model selection are done. A small or distilled model that fails a task outright is not proof that a larger sibling will also fail, and a small language model may lack abilities that its bigger counterpart displays reliably. The decision table below translates the research into working rules.

SituationRecommended practiceReason
Forecasting a larger runTrack continuous metrics on small modelsReveals progress hidden by exact match
Choosing a deployment modelTest the exact task at target sizeAbilities may not transfer downward
Designing a benchmarkReport several metrics and many sizesAvoids false steps and false plateaus
Safety evaluationProbe dangerous skills before releaseJumps can surprise developers
Using chain-of-thoughtVerify benefit on the chosen modelSmall models may lose accuracy
Working rules for teams evaluating capability at scale

Frontier safety frameworks explicitly reference unexpected capability gains and require dangerous-capability evaluations at regular compute intervals. Their reasoning is simple: if a new ability can appear between two checkpoints, testing only the final model leaves too little time to respond.

Common misconceptions about emergent abilities

  • Emergence is not consciousness or agency; it refers only to task performance curves.
  • Emergence is not magic: the underlying training objective is still next-token prediction.
  • A jump on one benchmark does not imply a general jump in intelligence.
  • Thresholds are not universal; better data or tuning can move them to smaller models.
  • Not every surprising output is emergent; many are hallucinations or memorized patterns.

Understanding it simply

Emergent abilities are skills that a language model seems to gain suddenly when it becomes big enough. A smaller model fails the task almost completely, and a larger one suddenly handles it well. The jump surprises people because the earlier, smaller models gave no clear hint that it was coming.

A useful comparison is a team spelling a long word aloud, one letter per person. If each person is only slightly reliable, the whole word almost always comes out wrong. As each person gets a bit better, the word suddenly starts coming out right. Each member improved gradually, but the result looked sudden. Scientists still debate how much of emergence is this kind of effect and how much is a real change inside the model.

Applied in practice

An evaluation team at a software company is choosing a model to extract totals from scanned invoices and check whether line items add up. They test a family of models at roughly 1, 8 and 70 billion parameters. Scored by exact match on the final total, the two smaller models reach only about 5 percent and 12 percent, while the largest reaches about 85 percent, which looks like a classic emergent jump.

Before deciding, the team re-scores the outputs with a continuous metric: the share of line items extracted correctly and the absolute error of each sum. The 8-billion model turns out to read most line items correctly but makes one arithmetic slip in most invoices. Adding a simple calculator tool to its pipeline lifts exact-match accuracy to about 80 percent. The team deploys the mid-sized model with tool support, cutting inference cost substantially while keeping accuracy close to the largest model.