In-context learning (ICL) is the ability of a pretrained model to perform a new task from examples and instructions placed in its prompt, without any change to its parameters. In language models, the model reads a few demonstrations of an input-output pattern, infers the task, and applies it to a new input within the same forward pass.
The idea was popularized when large autoregressive models showed that a handful of demonstrations could rival task-specific training on some benchmarks. ICL is the mechanism behind few-shot prompting and much of practical prompt engineering, and it explains why one general model can translate, classify and extract data without separate fine-tuning for each job.
How in-context learning works inside the model
In-context learning works because a transformer conditions every predicted token on all earlier tokens in the context window. When a prompt contains several input-output pairs, the model detects the regularity and continues it, so the learning happens in activations, not in weights. Nothing is stored after the request ends; the next request starts from the same frozen parameters.
Research offers several complementary explanations. Induction heads are attention circuits that find an earlier occurrence of the current token and copy what followed it, which supports pattern completion. Other work shows that attention layers can approximate implicit gradient descent on the demonstrations, and a Bayesian view treats ICL as inferring a latent task that pretraining already made familiar. None of these accounts is complete, and the exact mechanism is still an open research question.
The table below lists the main components of a prompt that relies on ICL and what each contributes.
| Component | Role | Typical form |
|---|---|---|
| Task instruction | States the goal and constraints | One or two sentences |
| Demonstrations | Show the input-output mapping | 2 to 32 labeled pairs |
| Format markers | Separate inputs, outputs and examples | Labels such as Input: and Output: |
| Query | The new case the model must solve | Same format as the demonstrations |
| Output slot | Signals where the answer begins | An empty Output: label |
Zero-shot, one-shot and few-shot in-context learning
In-context learning is usually described by the number of demonstrations the prompt contains. Zero-shot prompting gives only an instruction, while few-shot prompting adds examples; with long context windows, many-shot ICL with hundreds of examples has become practical and can approach fine-tuned accuracy on some classification tasks.
| Variant | Demonstrations | Best suited for | Main weakness |
|---|---|---|---|
| Zero-shot | None | Common, clearly described tasks | Format and labels drift |
| One-shot | One | Fixing an output format | Model may copy the single example |
| Few-shot | About 2 to 32 | Classification, extraction, style | Sensitive to example choice and order |
| Many-shot | Hundreds or more | Specialized labels, rare formats | High token cost and latency |
The gains are not linear. Accuracy often rises quickly over the first few examples and then flattens, and for strong instruction-tuned models a precise instruction sometimes matches a few-shot prompt. The right number depends on task difficulty, label ambiguity and the model itself.
In-context learning compared with fine-tuning and retrieval
In-context learning differs from fine-tuning because it adapts behavior per request without training. Fine-tuning changes weights permanently through gradient updates, while retrieval-augmented generation supplies facts rather than task demonstrations. Teams often combine all three, so the comparison is about trade-offs, not exclusive choices.
| Method | Changes weights | Setup effort | Per-request cost | Typical use |
|---|---|---|---|---|
| In-context learning | No | Minutes | Higher, prompt tokens repeat | Prototypes, varied small tasks |
| Fine-tuning | Yes | Days, needs labeled data | Lower per call | Stable, high-volume tasks |
| Retrieval augmentation | No | Index and pipeline | Moderate | Fresh or private knowledge |
| Instruction tuning | Yes, by the model provider | Not done by users | None extra | General instruction following |
A key distinction concerns parametric knowledge. ICL can teach a format, a labeling scheme or a reasoning style, but it cannot reliably add large bodies of new facts, because everything must fit inside the context window and be paid for on every call.
Designing demonstrations that work
Effective in-context learning depends more on demonstration quality and format than on sheer quantity. Studies have found that models rely heavily on the label space, input distribution and layout of the examples; in some settings even randomized labels preserve much of the benefit, which shows that the examples often signal the task rather than teach it.
- Choose examples that cover the label space and the realistic range of inputs, including hard borderline cases.
- Keep a strictly consistent format for every demonstration and for the query.
- Select examples semantically similar to the query when a retrieval step is available.
- Shuffle or balance the order, because the last examples and the most frequent label tend to bias the answer.
- Place the instruction before the examples and restate key constraints near the query in long prompts.
For reasoning tasks, demonstrations can include worked steps, which turns ICL into few-shot chain-of-thought prompting. This helps larger models most and can hurt small ones that imitate the form without the logic.
Limitations and common failure modes
In-context learning is powerful but brittle. The same examples in a different order can shift accuracy noticeably, and the model may follow surface patterns rather than the intended rule. The table below links frequent symptoms to causes and remedies.
| Symptom | Likely cause | Remedy |
|---|---|---|
| Answers favor one label | Majority or recency bias | Balance and shuffle examples, calibrate outputs |
| Output copies an example | Too few or too similar demonstrations | Add diverse examples |
| Results vary across runs | Order sensitivity and sampling | Fix order, lower temperature |
| Middle examples ignored | Lost in the middle effect | Shorten prompt, place key cases at ends |
| Costs grow quickly | Long demonstrations on every call | Use prompt caching or fine-tune |
ICL also raises security concerns. Because the model treats anything in its context as potential instruction, untrusted text inside a prompt can redirect behavior, a weakness known as prompt injection. Treat demonstrations and retrieved content as data, and evaluate every ICL prompt on a held-out test set before relying on it in production.
A plain-language summary
In-context learning means a language model picks up a task from the examples you show it, right there in the message. Imagine handing a new colleague three filled-in forms and asking them to fill in a fourth. They have not been retrained; they simply notice the pattern and repeat it.
The model works the same way. It does not remember the lesson afterward, and the next conversation starts fresh. The quality of the examples matters a great deal, because the model copies what it sees, including mistakes and odd habits.
An example from practice
A customer support team at a mid-sized software company needs to sort incoming tickets into seven categories, such as billing, login problems and feature requests. Instead of training a classifier, an engineer writes a prompt with a one-sentence instruction and fourteen labeled tickets, two for each category, in a fixed Input and Category format.
On a held-out set of 500 past tickets, the zero-shot version of the prompt reaches about 78 percent agreement with human agents, mostly confusing billing with account questions. After the engineer adds the fourteen demonstrations and balances their order, agreement rises to roughly 91 percent. The team ships the prompt within a day and plans to fine-tune a smaller model later, once ticket volume makes the repeated prompt tokens more expensive than training.