Dynamic few-shot prompting

Also known as: dynamic example selection

On this page 7

Dynamic few-shot prompting, also called dynamic example selection, is a prompting technique in which the demonstrations placed in a prompt are chosen per request from a larger pool, based on how relevant they are to the current input. Instead of a fixed set of examples written once into a prompt template, the system retrieves the examples most similar to the incoming query and inserts them before the task, so a language model sees guidance that fits the case in front of it.

The technique builds on in-context learning, the ability of large language models to infer a task from examples without any change to their weights. It sits between static few-shot prompting and fine-tuning: it adapts behavior to each input, yet the model itself stays untouched.

How dynamic few-shot prompting selects examples

Dynamic few-shot prompting works as a small retrieval pipeline in front of the model. A curated example pool stores input and output pairs, often hundreds or thousands of them. Each example input is converted into a vector embedding and kept in an index. When a new request arrives, it is embedded the same way, and the system picks the top k nearest examples, typically 3 to 8, which are then formatted into the prompt.

The selection step can use semantic similarity, keyword overlap such as BM25, metadata filters (language, product, customer tier) or a combination of these. Many implementations also apply diversity rules, for example maximal marginal relevance, so the chosen examples are not near-duplicates of each other. The ordering matters as well: placing the most similar example closest to the query often improves results, although the effect varies by model.

The table below lists the components of a typical dynamic few-shot pipeline and the role of each.

ComponentFunctionTypical choice
Example poolStores verified input and output pairsHundreds to thousands of labeled cases
Embedding modelTurns text into comparable vectorsGeneral-purpose sentence embedding model
IndexSupports fast nearest-neighbor searchVector database or in-memory index
SelectorChooses which examples to includeTop k by similarity with a diversity rule
Prompt templateFormats instructions, examples and queryFixed instruction block plus example slots
Components of a dynamic few-shot pipeline

Selection strategies and key parameters

The quality of dynamic few-shot prompting depends mainly on which examples are selected and how many. Too few examples give weak guidance; too many consume the context window, raise cost and latency, and can push the model toward copying surface patterns. The best number is found empirically on an evaluation set rather than assumed.

The following table compares common selection strategies and their trade-offs.

StrategyHow it picks examplesStrengthWeakness
Semantic similarityNearest embeddings to the queryStrong topical relevanceMay return redundant examples
Lexical matchingShared keywords, for example BM25Precise for codes and namesMisses paraphrases
Maximal marginal relevanceBalances similarity and noveltyDiverse, informative setExtra tuning parameter
Metadata filteringRestricts pool by category firstRespects hard constraintsNeeds well-labeled examples
Label-balanced selectionCovers several output classesReduces label biasLess similar examples overall
Example selection strategies compared

Other parameters include the similarity threshold below which no example is used, the token budget reserved for examples, and the format consistency between examples and the expected output. If examples use one output format and the instruction asks for another, models usually follow the examples.

Dynamic few-shot prompting differs from its neighbors in where the guidance comes from and when it is chosen. One-shot prompting supplies a single fixed example, while many-shot prompting places dozens or hundreds of examples into a long context. Retrieval-augmented generation retrieves facts or documents, whereas dynamic few-shot prompting retrieves demonstrations of the desired behavior.

ApproachExamples usedAdapts per inputMain cost
Zero-shot promptingNoneNoLower accuracy on unusual formats
Static few-shot promptingFixed handfulNoExamples may not fit the query
Dynamic few-shot promptingRetrieved handfulYesRetrieval infrastructure and latency
Many-shot promptingDozens to hundredsRarelyLarge token usage per call
Fine-tuningThousands in trainingLearned into weightsTraining effort and retraining for changes
Dynamic few-shot prompting and neighboring approaches

Related techniques such as Auto-CoT and active prompting also focus on choosing demonstrations, but they typically build a reusable example set in advance. Dynamic few-shot prompting makes that choice at inference time for every single request.

When to use dynamic example selection

Dynamic few-shot prompting pays off when a task has many distinct input types that each need a different demonstration, and when a library of verified examples already exists or can be collected. Typical cases are text-to-SQL generation, structured data extraction from varied documents, classification with many labels and customer support replies that follow case-specific policies.

SituationRecommended choiceReason
Uniform inputs, one output formatStatic few-shot promptRetrieval adds cost without benefit
Varied inputs with a large example libraryDynamic few-shot promptingRelevant guidance for each case
Frequent changes to examples or rulesDynamic few-shot promptingUpdate the pool without retraining
Stable task with very high volumeFine-tuningShorter prompts lower cost per call
Model lacks required factsRetrieval-augmented generationKnowledge gaps need documents, not demos
Deciding whether to use dynamic few-shot prompting

A useful practice is to log failed outputs, correct them and add them to the pool, which turns production errors into future demonstrations. This feedback loop is one of the main practical advantages of the technique over a fixed prompt.

Limitations and common mistakes

The technique is only as good as its example pool. Incorrect, outdated or inconsistent examples are copied faithfully, and because selection changes from request to request, behavior can vary in ways that are hard to reproduce. Logging the selected example identifiers with each response is therefore essential for debugging and evaluation.

  • Similarity is not usefulness: an example that looks similar may demonstrate the wrong reasoning or label.
  • Data leakage in evaluation: test cases must be excluded from the pool, or measured accuracy will be inflated.
  • Sensitive data exposure: examples drawn from real customer records can reach other users through the prompt.
  • Prompt caching suffers: changing examples per request reduces cache hits, so a fixed prefix should come first.
  • Retrieval latency adds up: embedding and search add tens of milliseconds or more to each call.

The short version

Language models learn quickly from examples placed in the prompt. Dynamic few-shot prompting simply picks the best examples for each question instead of always showing the same ones.

Imagine a new employee with a thick binder of solved cases. For every new request, a colleague pulls out the three cases most like it and puts them on the desk. The employee works faster and makes fewer mistakes, because the guidance always fits the task at hand. The main point: better-matched examples lead to better answers, without retraining the model.

Where it makes a difference

A retail analytics team lets business users ask questions in English, which a language model turns into SQL queries against a database with about 120 tables. A static prompt with five examples works for sales questions but often fails for inventory and returns, because the examples never show the joins those questions need.

The team builds a pool of about 900 verified question and SQL pairs, embeds every question, and retrieves the five most similar pairs for each new request, with a filter that keeps only examples from the same business area. Every wrong query reported by users is corrected and added to the pool each week.

On a held-out set of 300 questions, the share of queries that run and return correct results rises from roughly 62 percent to roughly 84 percent. Average prompt length stays nearly the same, and the added retrieval step costs about 40 ms per request.