Active prompting

On this page 7

Active prompting is a technique for building few-shot chain-of-thought exemplars in which a language model first answers a pool of unlabeled questions several times, the questions with the highest answer uncertainty are selected, and humans write reasoning chains only for those questions. The annotated items then serve as the demonstrations in the final prompt.

Prompting research introduced the method in 2023 by borrowing the idea of uncertainty-based sampling from active learning. Instead of choosing exemplars at random or by hand, active prompting spends scarce human annotation effort where the model is most confused, which typically yields more useful demonstrations for complex reasoning tasks in arithmetic, commonsense and symbolic domains.

How active prompting selects exemplars

Active prompting follows a four-stage pipeline that converts model uncertainty into a ranked annotation queue. The process needs only a pool of task questions without answers, a model that can be sampled several times, and a small annotation budget.

  1. Uncertainty estimation: each question is answered k times (often 5 to 10) with sampling enabled and a few existing or zero-shot chain-of-thought prompts.
  2. Selection: the questions are ranked by an uncertainty metric, and the top n (commonly 4 to 8) are chosen.
  3. Annotation: human experts write a correct step-by-step rationale and final answer for each selected question.
  4. Inference: the annotated questions become the few-shot demonstrations for every new test question.

The logic is that a question on which the model already agrees with itself teaches it little, while a question that produces many conflicting answers marks a gap in its reasoning. A carefully written rationale for such a question gives the model a pattern it could not reliably infer on its own.

Uncertainty metrics used in active prompting

The choice of uncertainty metric decides which questions reach the annotators, and each metric suits a different answer format. The table below lists the metrics described in the original work and their typical behavior.

MetricHow it is computedBest suited for
DisagreementNumber of distinct answers divided by kShort discrete answers such as numbers
EntropyEntropy over the frequency of each answerTasks where answer distribution matters
VarianceSpread of numeric answers after normalizationArithmetic problems with numeric outputs
Self-confidenceModel rates its own answer confidenceCases where sampling is too costly
Uncertainty metrics for ranking candidate questions

In reported experiments, disagreement and entropy performed similarly and both outperformed random selection, while self-confidence was the weakest signal because language models tend to be overconfident when asked to rate themselves. Practitioners therefore usually start with disagreement, which is cheap and easy to interpret.

Active prompting changes which examples enter the prompt, not how the model reasons at inference time. That focus separates it from several neighboring techniques that it can be combined with.

TechniqueHow exemplars are obtainedHuman effort
Active promptingMost uncertain questions annotated by humansSmall, targeted
Manual few-shot chain-of-thoughtHand-picked questions with written rationalesSmall, untargeted
Auto-CoTClustered questions with model-generated rationalesNone
Dynamic few-shot promptingSimilar examples retrieved per queryRequires a labeled pool
Self-consistencyUses existing exemplars and votes over samplesNone beyond exemplars
Active prompting compared with neighboring techniques

Self-consistency is closely related because both sample several reasoning paths, but it uses the samples to vote on a final answer, while active prompting uses them before deployment to locate weak spots. The two combine well: active prompting picks the exemplars and self-consistency aggregates answers at inference.

Applying active prompting in practice

Active prompting pays off when a task requires multi-step reasoning, unlabeled questions are plentiful, and expert annotation is expensive. The decision table below summarizes when it is the right choice.

SituationRecommended approachReason
Large unlabeled pool, scarce expertsActive promptingAnnotation goes to the most informative items
Simple classification taskPlain few-shot promptingUncertainty selection adds little value
No budget for human annotationAuto-CoT or zero-shot chain-of-thoughtRationales can be generated automatically
Highly varied incoming queriesDynamic few-shot retrievalPer-query similarity beats one fixed set
Strict latency or cost limitsSmall fixed exemplar setSelection cost is paid once, but prompts stay long
When to use active prompting

The selection step is a one-time cost: answering 1,000 questions with k = 5 means 5,000 model calls, after which the exemplar set is reused for every request. Teams should keep a held-out evaluation set to confirm that the selected exemplars actually improve accuracy over a random baseline.

Limitations and common mistakes

Active prompting is not a guarantee of better accuracy, and its benefit varies by model and task. Gains reported in research were typically a few percentage points over standard chain-of-thought prompting, and stronger recent models leave less room for improvement.

  • Uncertain is not always informative: some questions are uncertain because they are ambiguous or mislabeled, and annotating them adds noise.
  • Annotation quality dominates: a sloppy or inconsistent rationale can harm results more than a random exemplar.
  • Model dependence: exemplars selected with one model may not transfer equally well to another.
  • Disagreement fails on open-ended outputs: free-text answers need normalization or semantic clustering before counting.
  • Lack of diversity: the top uncertain questions may all share one pattern, so checking topical coverage is advisable.

A plain-language summary

Active prompting works like a good tutor preparing worked examples. Instead of picking practice problems at random, the tutor first watches where the student keeps giving different answers to the same question. Those confusing problems get carefully worked solutions, and the student studies them before the test.

For an AI model, the process is the same. The model answers many questions several times, people explain the ones it struggles with most, and those explanations go into its prompt. The key idea: spend human effort only where the model is most confused.

An example from practice

An insurance company uses a language model to calculate claim reimbursement amounts from policy rules and invoice data. The team has 2,000 past claims without written reasoning and only two senior adjusters available for a single afternoon of annotation.

An engineer runs each claim five times through the model with sampling enabled and ranks the claims by answer disagreement. Most claims return one consistent figure, but about 150 produce three or more different amounts, usually where deductibles, coverage caps and partial exclusions interact. The adjusters write detailed reasoning for the eight most uncertain cases, covering different rule combinations.

With those eight exemplars in the prompt, accuracy on a held-out set of 300 claims rises noticeably compared with eight randomly chosen examples. The outcome is a measurable gain in accuracy from one afternoon of expert work, and the same procedure is repeated whenever policy rules change.