Active prompting is a technique for building few-shot chain-of-thought exemplars in which a language model first answers a pool of unlabeled questions several times, the questions with the highest answer uncertainty are selected, and humans write reasoning chains only for those questions. The annotated items then serve as the demonstrations in the final prompt.
Prompting research introduced the method in 2023 by borrowing the idea of uncertainty-based sampling from active learning. Instead of choosing exemplars at random or by hand, active prompting spends scarce human annotation effort where the model is most confused, which typically yields more useful demonstrations for complex reasoning tasks in arithmetic, commonsense and symbolic domains.
How active prompting selects exemplars
Active prompting follows a four-stage pipeline that converts model uncertainty into a ranked annotation queue. The process needs only a pool of task questions without answers, a model that can be sampled several times, and a small annotation budget.
- Uncertainty estimation: each question is answered k times (often 5 to 10) with sampling enabled and a few existing or zero-shot chain-of-thought prompts.
- Selection: the questions are ranked by an uncertainty metric, and the top n (commonly 4 to 8) are chosen.
- Annotation: human experts write a correct step-by-step rationale and final answer for each selected question.
- Inference: the annotated questions become the few-shot demonstrations for every new test question.
The logic is that a question on which the model already agrees with itself teaches it little, while a question that produces many conflicting answers marks a gap in its reasoning. A carefully written rationale for such a question gives the model a pattern it could not reliably infer on its own.
Uncertainty metrics used in active prompting
The choice of uncertainty metric decides which questions reach the annotators, and each metric suits a different answer format. The table below lists the metrics described in the original work and their typical behavior.
| Metric | How it is computed | Best suited for |
|---|---|---|
| Disagreement | Number of distinct answers divided by k | Short discrete answers such as numbers |
| Entropy | Entropy over the frequency of each answer | Tasks where answer distribution matters |
| Variance | Spread of numeric answers after normalization | Arithmetic problems with numeric outputs |
| Self-confidence | Model rates its own answer confidence | Cases where sampling is too costly |
In reported experiments, disagreement and entropy performed similarly and both outperformed random selection, while self-confidence was the weakest signal because language models tend to be overconfident when asked to rate themselves. Practitioners therefore usually start with disagreement, which is cheap and easy to interpret.
How it differs from related prompting techniques
Active prompting changes which examples enter the prompt, not how the model reasons at inference time. That focus separates it from several neighboring techniques that it can be combined with.
| Technique | How exemplars are obtained | Human effort |
|---|---|---|
| Active prompting | Most uncertain questions annotated by humans | Small, targeted |
| Manual few-shot chain-of-thought | Hand-picked questions with written rationales | Small, untargeted |
| Auto-CoT | Clustered questions with model-generated rationales | None |
| Dynamic few-shot prompting | Similar examples retrieved per query | Requires a labeled pool |
| Self-consistency | Uses existing exemplars and votes over samples | None beyond exemplars |
Self-consistency is closely related because both sample several reasoning paths, but it uses the samples to vote on a final answer, while active prompting uses them before deployment to locate weak spots. The two combine well: active prompting picks the exemplars and self-consistency aggregates answers at inference.
Applying active prompting in practice
Active prompting pays off when a task requires multi-step reasoning, unlabeled questions are plentiful, and expert annotation is expensive. The decision table below summarizes when it is the right choice.
| Situation | Recommended approach | Reason |
|---|---|---|
| Large unlabeled pool, scarce experts | Active prompting | Annotation goes to the most informative items |
| Simple classification task | Plain few-shot prompting | Uncertainty selection adds little value |
| No budget for human annotation | Auto-CoT or zero-shot chain-of-thought | Rationales can be generated automatically |
| Highly varied incoming queries | Dynamic few-shot retrieval | Per-query similarity beats one fixed set |
| Strict latency or cost limits | Small fixed exemplar set | Selection cost is paid once, but prompts stay long |
The selection step is a one-time cost: answering 1,000 questions with k = 5 means 5,000 model calls, after which the exemplar set is reused for every request. Teams should keep a held-out evaluation set to confirm that the selected exemplars actually improve accuracy over a random baseline.
Limitations and common mistakes
Active prompting is not a guarantee of better accuracy, and its benefit varies by model and task. Gains reported in research were typically a few percentage points over standard chain-of-thought prompting, and stronger recent models leave less room for improvement.
- Uncertain is not always informative: some questions are uncertain because they are ambiguous or mislabeled, and annotating them adds noise.
- Annotation quality dominates: a sloppy or inconsistent rationale can harm results more than a random exemplar.
- Model dependence: exemplars selected with one model may not transfer equally well to another.
- Disagreement fails on open-ended outputs: free-text answers need normalization or semantic clustering before counting.
- Lack of diversity: the top uncertain questions may all share one pattern, so checking topical coverage is advisable.
A plain-language summary
Active prompting works like a good tutor preparing worked examples. Instead of picking practice problems at random, the tutor first watches where the student keeps giving different answers to the same question. Those confusing problems get carefully worked solutions, and the student studies them before the test.
For an AI model, the process is the same. The model answers many questions several times, people explain the ones it struggles with most, and those explanations go into its prompt. The key idea: spend human effort only where the model is most confused.
An example from practice
An insurance company uses a language model to calculate claim reimbursement amounts from policy rules and invoice data. The team has 2,000 past claims without written reasoning and only two senior adjusters available for a single afternoon of annotation.
An engineer runs each claim five times through the model with sampling enabled and ranks the claims by answer disagreement. Most claims return one consistent figure, but about 150 produce three or more different amounts, usually where deductibles, coverage caps and partial exclusions interact. The adjusters write detailed reasoning for the eight most uncertain cases, covering different rule combinations.
With those eight exemplars in the prompt, accuracy on a held-out set of 300 claims rises noticeably compared with eight randomly chosen examples. The outcome is a measurable gain in accuracy from one afternoon of expert work, and the same procedure is repeated whenever policy rules change.