Auto-CoT

Also known as: Automatic chain-of-thought

On this page 7

Auto-CoT (automatic chain-of-thought) is a prompting technique that builds chain-of-thought demonstrations automatically instead of relying on examples written by hand. In prompting for large language models, it selects a diverse set of questions, lets the model generate a step-by-step rationale for each one with zero-shot chain-of-thought, and then uses those question-rationale pairs as few-shot examples for new questions.

The method was introduced in 2022 by researchers who observed that manually written reasoning examples are costly and task-specific, while zero-shot reasoning alone is less reliable. Auto-CoT closes that gap by combining both approaches: the model writes its own demonstrations, and a diversity-based selection step limits the damage done by the mistakes those demonstrations inevitably contain.

How Auto-CoT constructs demonstrations

Auto-CoT works in two stages: question clustering and demonstration sampling. Both stages run once per task, before any user question is answered, and produce a fixed prompt prefix that is reused for every query.

  1. Embed the question pool. Each unlabeled question from the task is converted into a vector with a sentence encoder; the original work used Sentence-BERT.
  2. Cluster with k-means. The vectors are grouped into k clusters, where k equals the number of demonstrations the prompt will contain, typically around eight.
  3. Pick a representative per cluster. Questions in each cluster are sorted by distance to the centroid, and the closest acceptable question is chosen.
  4. Generate a rationale. The model answers that question with a zero-shot chain-of-thought trigger, an instruction to reason step by step, producing a rationale and an answer.
  5. Filter with simple heuristics. Candidates are rejected if the question is too long or the rationale has too many steps, and the next question in the cluster is tried.

The heuristics in the final step matter because short questions with short rationales are more likely to be correct. The original paper used limits of roughly 60 tokens per question and five reasoning steps per rationale, values that act as a cheap proxy for quality when no labels exist.

The following table shows the components and the purpose of each.

ComponentInputOutputPurpose
Sentence encoderRaw questionsEmbedding vectorsMeasure semantic similarity
K-means clusteringEmbedding vectorsk question groupsEnsure topical diversity
Zero-shot CoT callOne question per groupRationale and answerCreate reasoning chains
Selection heuristicsCandidate demonstrationsAccepted demonstrationsReduce likely errors
Components of the Auto-CoT pipeline

Why diversity beats similarity

The central insight of Auto-CoT is that diverse demonstrations tolerate errors better than similar ones. An obvious alternative is to retrieve the questions most similar to the test question and use their generated rationales. That approach performs worse, because zero-shot rationales contain mistakes, and similar questions tend to share the same failure mode.

The authors described this effect as misleading by similarity: when a model wrongly solves several near-identical questions, showing those wrong chains as examples teaches it to repeat the error. Clustering spreads the demonstrations across different question types, so a single wrong rationale affects only a fraction of the prompt, and the remaining correct chains still convey the reasoning format.

The table compares the selection strategies that were studied.

StrategyHow examples are chosenMain weakness
Random samplingQuestions drawn at random from the poolMay cluster around one error type
Similarity retrievalNearest neighbors of the test questionPropagates shared mistakes
Diversity clusteringOne representative per k-means clusterDepends on embedding quality
Demonstration selection strategies

Auto-CoT belongs to a family of reasoning techniques and is best understood by what it automates and what it leaves unchanged. It changes only how few-shot examples are obtained; the inference step is ordinary few-shot chain-of-thought prompting.

TechniqueSource of demonstrationsHuman effortTypical reliability
Manual chain-of-thoughtHand-written rationalesHighStrong but task-specific
Zero-shot chain-of-thoughtNone, only a trigger phraseNoneLower on complex tasks
Auto-CoTModel-generated, diversity-selectedLowComparable to manual in original tests
Active promptingUncertain questions annotated by humansMediumTargets hardest cases
Dynamic few-shot promptingRetrieved per query from a labeled setMediumGood with verified labels
Auto-CoT and neighboring techniques

Auto-CoT is also compatible with self-consistency: several reasoning paths can be sampled with the automatically built prompt, and the most frequent answer is selected. In the original experiments on arithmetic, commonsense and symbolic reasoning benchmarks, Auto-CoT matched or exceeded manually designed demonstrations with a GPT-3 class model, although results vary with model and task.

Applying Auto-CoT in practice

Auto-CoT is most useful when a team has many unlabeled questions of one task type and no time to write reasoning examples. A pool of a few hundred representative questions usually gives clustering enough material; very small pools reduce the benefit of diversity.

SituationRecommendationReason
Large unlabeled question poolUse Auto-CoTClustering has enough variety
A few verified examples existPrefer manual or dynamic few-shotCorrect chains beat generated ones
Frontier reasoning modelTest zero-shot firstBuilt-in reasoning may suffice
High-stakes domainReview generated rationalesErrors in demonstrations persist
When Auto-CoT is a good fit

Practitioners often add a light human review of the eight or so selected rationales. This hybrid keeps most of the automation while removing the obvious errors, and it costs minutes instead of the hours needed to write chains from scratch.

Limitations and open questions

Auto-CoT does not verify correctness; its heuristics only make errors less likely. The technique also adds a one-time preprocessing cost for embedding, clustering and generation, and the resulting prompt is static, so it may drift out of date if the question distribution changes.

  • Embedding choice shapes clusters, and a weak encoder can group questions by surface wording instead of reasoning type.
  • Benefits shrink with modern reasoning models, which often perform well with zero-shot instructions alone.
  • Heuristic thresholds are task-dependent and were tuned on benchmark data, not on every domain.
  • Long prompts increase cost, since every query carries the full set of demonstrations.

A non-technical view

Language models often answer better when they see a few worked examples first. Writing those examples by hand takes time, so Auto-CoT lets the model write its own worked examples.

The trick is choosing different kinds of questions to practice on. Imagine a teacher preparing a review sheet: picking one problem from each chapter is safer than picking five problems from the same chapter, because a single misunderstanding would then spoil the whole sheet.

A concrete example

A logistics company wants a language model to answer arithmetic questions from warehouse staff, such as how many pallets remain after several shipments. The team has about 800 past questions without solutions and no time to write reasoning examples.

They embed the questions, run k-means with eight clusters, and pick one short question from each: some about subtraction over shipments, some about unit conversion, some about percentages of damaged goods. The model writes a step-by-step solution for each, and an analyst spends ten minutes checking them, correcting one percentage error.

The resulting prompt is used for every new question. On a held-out test set of checked answers, the team finds fewer arithmetic slips than with zero-shot prompting, and the prompt took under an hour to build instead of a day of manual writing.