Many-shot prompting

Also known as: many-shot in-context learning

On this page 7

Many-shot prompting is a prompting technique in which a language model receives hundreds or even thousands of input-output examples inside a single prompt, so that it infers the task from demonstrations rather than from instructions alone. Also called many-shot in-context learning, it extends classic few-shot prompting to the scale made possible by long context windows of hundreds of thousands to millions of tokens.

Within artificial intelligence, the technique matters because it occupies the space between few-shot prompting and fine-tuning: it can adapt a general model to a specialized task with substantial accuracy gains, yet it requires no change to model weights and no training infrastructure.

How many-shot prompting works

The mechanism is in-context learning: the model conditions its next-token predictions on every token in the prompt, including the demonstrations. With a handful of examples, the model mostly recognizes which known task is being requested. With hundreds of examples, it receives enough evidence to pick up finer decision boundaries, label conventions and output formats that a short prompt cannot express.

Research published in 2024 on long-context models showed that accuracy on tasks such as translation into low-resource languages, summarization, classification and mathematical reasoning often keeps improving as the number of shots grows into the hundreds. The same work found that very large demonstration sets can override biases learned in pretraining, for example when labels are deliberately flipped, which few-shot prompts rarely achieve.

The gains are not uniform. Some tasks plateau after a few dozen examples, and some show a decline at the largest counts, so the effective number of shots must be measured per task rather than assumed. The table below places many-shot prompting on the spectrum of example counts.

RegimeNumber of examplesMain effect on the model
Zero-shot promptingNoneRelies on instructions and pretraining
One-shot promptingOneShows format and expected style
Few-shot promptingAbout 2 to 30Identifies the task and output pattern
Many-shot promptingAbout 50 to several thousandLearns task-specific nuances and labels
Example counts across prompting regimes

Variants of many-shot prompting

The main obstacle is the supply of high-quality labeled examples, especially for reasoning tasks that need written solutions. Two variants address this. Reinforced in-context learning replaces human-written rationales with solutions generated by the model and filtered for correct final answers. Unsupervised in-context learning supplies only the problems, without solutions, and relies on the model to activate relevant knowledge.

VariantContent of each exampleBest suited to
Standard many-shotHuman-labeled input and outputClassification, extraction, translation
Reinforced ICLInput plus verified model-generated rationaleMath and multi-step reasoning
Unsupervised ICLInput only, without answersDomains the model partly knows
Retrieved many-shotExamples selected by similarity to the queryLarge, diverse example pools
Main variants and their data requirements

The retrieved form overlaps with dynamic few-shot prompting, which also selects examples per query. The difference is scale: many-shot designs keep a large fixed or semi-fixed block of demonstrations, which also makes the prompt prefix suitable for caching.

How it compares with few-shot prompting and fine-tuning

Many-shot prompting is a trade between flexibility and cost per request. Fine-tuning pays the cost of adaptation once and then serves short prompts, while many-shot prompting pays for a long prompt on every call. Few-shot prompting is cheap but captures less of the task.

PropertyFew-shot promptingMany-shot promptingFine-tuning
Examples neededA handfulHundreds to thousandsHundreds to many thousands
Weight changesNoneNoneModel weights updated
Cost per requestLowHigh without cachingLow after training
Time to updateMinutesMinutesHours to days
Typical accuracy gainModestOften substantialOften highest on narrow tasks
Many-shot prompting compared with alternatives

A practical rule follows from this comparison: many-shot prompting suits rapid iteration, closed models without tuning access and tasks that change often, whereas fine-tuning wins when the task is stable and request volume is very high.

Designing an effective many-shot prompt

Good many-shot prompts are curated datasets, not dumps of available records. Examples should cover the full range of labels and edge cases in realistic proportions, use one consistent format and contain no contradictions, because the model copies inconsistencies as faithfully as it copies good patterns.

  • Balance the label distribution so the model does not learn a majority-class shortcut.
  • Use clear delimiters between examples and a fixed field order.
  • Place the instruction and the query after the examples, close to the generation point.
  • Shuffle and test example order, because order still affects results.
  • Cache the static prefix where the provider supports prompt caching to cut cost and latency.

Evaluation should vary the shot count, for example 25, 100, 250 and 500 examples, on a held-out test set. The resulting curve shows where accuracy saturates and prevents paying for tokens that add nothing.

Limitations and security risks

The most visible limits are cost, latency and context capacity. A prompt with 500 examples can easily reach 100,000 tokens or more, which raises the price of each call and the time to first token. Long contexts also suffer from uneven attention, so examples in the middle of the prompt may influence the model less than those near the ends.

The same mechanism creates a known attack surface. Many-shot jailbreaking fills a long prompt with fabricated dialogues in which an assistant complies with harmful requests, and the probability of a harmful answer rises with the number of such dialogues. Applications that let users insert long text into prompts need input filtering and output monitoring.

SymptomLikely causeFix
Accuracy drops at high shot countsNoisy or redundant examplesDeduplicate and clean the example set
One label dominates outputsImbalanced demonstrationsRebalance labels across examples
Results change between runsOrder sensitivityFix and test a stable ordering
Costs exceed budgetLong prompt on every callPrompt caching or fine-tuning
Common problems and fixes

A non-technical view

Imagine training a new employee to sort customer letters. You could explain the rules in a few sentences, or you could hand over a binder with five hundred letters that are already sorted and say: do it like this. The binder usually teaches more, because it shows all the tricky cases.

Many-shot prompting does exactly that for an AI model. Instead of a short instruction, the model reads a very large set of solved examples right before the new question. Nothing about the model changes permanently; it simply works from the binder each time.

A concrete example

An insurance company routes incoming claim descriptions to one of 24 internal categories. A zero-shot prompt with category definitions reaches about 71 percent accuracy on a test set of 1,000 claims, mostly because several categories overlap in subtle ways that the definitions do not capture.

The analytics team assembles 400 labeled historical claims, balanced so that every category has at least 12 examples, and places them before each new claim. The prompt grows to roughly 90,000 tokens, and the team uses prompt caching so that only the new claim is processed at full price.

Accuracy rises to about 88 percent, close to a fine-tuned model the team had considered, while category definitions can still be changed by editing examples. Testing at 100 and 200 examples showed most of the gain by 200 shots, so the team later trimmed the prompt to reduce latency.