In-context learning

Also known as: ICL

On this page 7

In-context learning (ICL) is the ability of a pretrained model to perform a new task from examples and instructions placed in its prompt, without any change to its parameters. In language models, the model reads a few demonstrations of an input-output pattern, infers the task, and applies it to a new input within the same forward pass.

The idea was popularized when large autoregressive models showed that a handful of demonstrations could rival task-specific training on some benchmarks. ICL is the mechanism behind few-shot prompting and much of practical prompt engineering, and it explains why one general model can translate, classify and extract data without separate fine-tuning for each job.

How in-context learning works inside the model

In-context learning works because a transformer conditions every predicted token on all earlier tokens in the context window. When a prompt contains several input-output pairs, the model detects the regularity and continues it, so the learning happens in activations, not in weights. Nothing is stored after the request ends; the next request starts from the same frozen parameters.

Research offers several complementary explanations. Induction heads are attention circuits that find an earlier occurrence of the current token and copy what followed it, which supports pattern completion. Other work shows that attention layers can approximate implicit gradient descent on the demonstrations, and a Bayesian view treats ICL as inferring a latent task that pretraining already made familiar. None of these accounts is complete, and the exact mechanism is still an open research question.

The table below lists the main components of a prompt that relies on ICL and what each contributes.

ComponentRoleTypical form
Task instructionStates the goal and constraintsOne or two sentences
DemonstrationsShow the input-output mapping2 to 32 labeled pairs
Format markersSeparate inputs, outputs and examplesLabels such as Input: and Output:
QueryThe new case the model must solveSame format as the demonstrations
Output slotSignals where the answer beginsAn empty Output: label
Components of an in-context learning prompt

Zero-shot, one-shot and few-shot in-context learning

In-context learning is usually described by the number of demonstrations the prompt contains. Zero-shot prompting gives only an instruction, while few-shot prompting adds examples; with long context windows, many-shot ICL with hundreds of examples has become practical and can approach fine-tuned accuracy on some classification tasks.

VariantDemonstrationsBest suited forMain weakness
Zero-shotNoneCommon, clearly described tasksFormat and labels drift
One-shotOneFixing an output formatModel may copy the single example
Few-shotAbout 2 to 32Classification, extraction, styleSensitive to example choice and order
Many-shotHundreds or moreSpecialized labels, rare formatsHigh token cost and latency
Variants of in-context learning by number of demonstrations

The gains are not linear. Accuracy often rises quickly over the first few examples and then flattens, and for strong instruction-tuned models a precise instruction sometimes matches a few-shot prompt. The right number depends on task difficulty, label ambiguity and the model itself.

In-context learning compared with fine-tuning and retrieval

In-context learning differs from fine-tuning because it adapts behavior per request without training. Fine-tuning changes weights permanently through gradient updates, while retrieval-augmented generation supplies facts rather than task demonstrations. Teams often combine all three, so the comparison is about trade-offs, not exclusive choices.

MethodChanges weightsSetup effortPer-request costTypical use
In-context learningNoMinutesHigher, prompt tokens repeatPrototypes, varied small tasks
Fine-tuningYesDays, needs labeled dataLower per callStable, high-volume tasks
Retrieval augmentationNoIndex and pipelineModerateFresh or private knowledge
Instruction tuningYes, by the model providerNot done by usersNone extraGeneral instruction following
How ICL compares with other adaptation methods

A key distinction concerns parametric knowledge. ICL can teach a format, a labeling scheme or a reasoning style, but it cannot reliably add large bodies of new facts, because everything must fit inside the context window and be paid for on every call.

Designing demonstrations that work

Effective in-context learning depends more on demonstration quality and format than on sheer quantity. Studies have found that models rely heavily on the label space, input distribution and layout of the examples; in some settings even randomized labels preserve much of the benefit, which shows that the examples often signal the task rather than teach it.

  • Choose examples that cover the label space and the realistic range of inputs, including hard borderline cases.
  • Keep a strictly consistent format for every demonstration and for the query.
  • Select examples semantically similar to the query when a retrieval step is available.
  • Shuffle or balance the order, because the last examples and the most frequent label tend to bias the answer.
  • Place the instruction before the examples and restate key constraints near the query in long prompts.

For reasoning tasks, demonstrations can include worked steps, which turns ICL into few-shot chain-of-thought prompting. This helps larger models most and can hurt small ones that imitate the form without the logic.

Limitations and common failure modes

In-context learning is powerful but brittle. The same examples in a different order can shift accuracy noticeably, and the model may follow surface patterns rather than the intended rule. The table below links frequent symptoms to causes and remedies.

SymptomLikely causeRemedy
Answers favor one labelMajority or recency biasBalance and shuffle examples, calibrate outputs
Output copies an exampleToo few or too similar demonstrationsAdd diverse examples
Results vary across runsOrder sensitivity and samplingFix order, lower temperature
Middle examples ignoredLost in the middle effectShorten prompt, place key cases at ends
Costs grow quicklyLong demonstrations on every callUse prompt caching or fine-tune
Typical ICL problems and how to address them

ICL also raises security concerns. Because the model treats anything in its context as potential instruction, untrusted text inside a prompt can redirect behavior, a weakness known as prompt injection. Treat demonstrations and retrieved content as data, and evaluate every ICL prompt on a held-out test set before relying on it in production.

A plain-language summary

In-context learning means a language model picks up a task from the examples you show it, right there in the message. Imagine handing a new colleague three filled-in forms and asking them to fill in a fourth. They have not been retrained; they simply notice the pattern and repeat it.

The model works the same way. It does not remember the lesson afterward, and the next conversation starts fresh. The quality of the examples matters a great deal, because the model copies what it sees, including mistakes and odd habits.

An example from practice

A customer support team at a mid-sized software company needs to sort incoming tickets into seven categories, such as billing, login problems and feature requests. Instead of training a classifier, an engineer writes a prompt with a one-sentence instruction and fourteen labeled tickets, two for each category, in a fixed Input and Category format.

On a held-out set of 500 past tickets, the zero-shot version of the prompt reaches about 78 percent agreement with human agents, mostly confusing billing with account questions. After the engineer adds the fourteen demonstrations and balances their order, agreement rises to roughly 91 percent. The team ships the prompt within a day and plans to fine-tune a smaller model later, once ticket volume makes the repeated prompt tokens more expensive than training.