Chain-of-thought (CoT) is a technique in which a language model produces a sequence of intermediate reasoning steps before stating its final answer, instead of jumping directly from question to conclusion. In the study of language models within artificial intelligence, the term covers both chain-of-thought prompting, where the user asks for or demonstrates step-by-step reasoning, and the trained reasoning traces that modern reasoning models generate on their own.
The method matters because a model computes each token from the tokens before it. Writing out partial results gives the model additional computation and working memory inside its own context window, which measurably improves accuracy on multi-step problems such as arithmetic word problems, logic puzzles, code tracing and planning tasks.
How chain-of-thought reasoning works inside a model
Chain-of-thought works by turning one hard prediction into many easier predictions. A transformer performs a fixed amount of computation per generated token, so a question that requires five dependent operations is difficult to answer in a single token. When the model writes each operation as text, every later step can attend to the earlier results that are already on the page.
This makes the generated text function as an external scratchpad. The model does not store hidden intermediate values between tokens beyond what the attention mechanism can recover from the visible sequence, so the written steps become the carrier of state. The effect is strongest on tasks with serial dependencies, where step three needs the output of step two.
The behavior is largely an emergent capability of scale. Early experiments showed that small models gained little or even lost accuracy with step-by-step prompts, while larger models improved substantially. Today, instruction-tuned and reasoning-tuned models are trained explicitly to produce such reasoning, so the benefit appears across a wider range of model sizes.
The following table summarizes the main components of a chain-of-thought interaction and the role each one plays.
| Component | Role | Typical form |
|---|---|---|
| Trigger or instruction | Asks the model to reason before answering | “Think step by step” or a system prompt rule |
| Worked exemplars | Show the expected reasoning format | Two to eight solved examples in the prompt |
| Reasoning trace | Holds intermediate results and checks | Numbered steps or free-form text |
| Final answer | Delivers the result in a parsable form | A marked line such as “Answer: 42” |
Main variants of chain-of-thought prompting
Chain-of-thought has evolved into a family of related methods that differ in how the reasoning is elicited, sampled and verified. The choice among them depends on cost, latency and how much accuracy the task demands.
| Variant | How it works | Best suited for |
|---|---|---|
| Few-shot CoT | Prompt includes solved examples with reasoning | Tasks with a stable, repeatable format |
| Zero-shot CoT | A short instruction requests step-by-step reasoning | Quick use without curated examples |
| Self-consistency | Samples several reasoning paths and takes a majority vote | Math and logic with a single correct answer |
| Tree of thoughts | Explores and prunes branching partial solutions | Search and planning problems |
| Least-to-most prompting | Decomposes a problem into ordered subproblems | Compositional tasks that exceed exemplar length |
| Trained reasoning | Model is fine-tuned or reinforced to reason internally | Hard reasoning with reasoning models |
Self-consistency deserves special mention because it often yields the largest single gain. Sampling at a nonzero temperature produces diverse reasoning paths, and errors in individual paths tend to scatter while correct paths converge. The trade-off is a linear increase in cost, since five samples cost roughly five times as much as one.
In reasoning models, chain-of-thought is no longer a prompt trick but a trained behavior. These models are optimized with reinforcement learning on verifiable tasks, so they learn to produce long internal traces that include backtracking and self-correction. Many providers hide or summarize these traces, which changes how developers inspect them.
Chain-of-thought compared with related prompting techniques
Chain-of-thought is one technique within prompt engineering, and it is often combined with neighboring methods rather than replacing them. The table below separates it from the techniques it is most often confused with.
| Technique | Main idea | Relation to chain-of-thought |
|---|---|---|
| Zero-shot prompting | Ask directly without examples | Can be combined with a reasoning instruction |
| Few-shot prompting | Provide input and output examples | CoT adds the reasoning between input and output |
| Prompt chaining | Split a task across several model calls | Reasoning is distributed across separate prompts |
| Reasoning model | Model trained to reason by default | CoT is built into training, not the prompt |
| Retrieval augmentation | Supply external documents as context | Supplies facts, while CoT structures their use |
The key distinction is between reasoning within one generation and reasoning across multiple calls. Prompt chaining gives developers explicit control and validation between steps, while chain-of-thought keeps the steps inside a single response, which is simpler but harder to audit programmatically.
Applying chain-of-thought in practice
Chain-of-thought should be used when a task requires several dependent steps and skipped when the answer is a direct lookup or a simple classification. On easy tasks, the extra tokens add latency and cost without accuracy gains, and occasionally the model talks itself out of a correct first impression.
| Situation | Recommended approach | Reason |
|---|---|---|
| Multi-step arithmetic | Few-shot CoT with self-consistency | Reduces calculation slips |
| Simple fact retrieval | Direct answer without CoT | Reasoning adds tokens but no accuracy |
| Structured output for software | Reason first, then a delimited answer | Keeps the final result parsable |
| Using a reasoning model | Short, clear instructions only | Explicit CoT prompts may duplicate or hinder trained reasoning |
| Latency-critical chat | Brief or no visible reasoning | Long traces slow the first useful token |
A reliable pattern is to ask for reasoning followed by a clearly marked final answer line. Applications can then parse only that line and discard or log the trace. When the reasoning will be shown to end users, it should be requested in a concise style, because long traces increase token usage and can expose unhelpful detail.
Limitations and common misconceptions
The most important limitation is that a reasoning trace is not guaranteed to be faithful. Research has shown that models sometimes reach an answer through cues unrelated to the written steps and then produce a plausible justification afterward. The trace is therefore useful evidence, but it is not a reliable explanation of the computation.
- Correct steps do not guarantee a correct answer, and a correct answer can follow flawed steps.
- Errors compound in long chains, because one wrong intermediate value propagates to every later step.
- Chain-of-thought does not add knowledge: if a fact is missing from the model, reasoning can produce a confident hallucination.
- Token cost grows quickly, since traces can be several times longer than the answer itself.
- Visible traces can leak information, such as fragments of a system prompt or sensitive input data.
Another misconception is that the magic phrase “think step by step” works equally well everywhere. Its effect depends on model size, training and task type, and modern models often reason without being asked. Practitioners should evaluate CoT against a direct-answer baseline on their own data rather than assuming a benefit.
In simple terms
Chain-of-thought means letting an AI show its work before it gives an answer, much like a student solving a math problem on paper instead of in their head. Writing each step down helps the model keep track of what it has already figured out.
This usually makes answers to tricky, multi-step questions more accurate. It costs more time and text, and the written steps are not always an honest picture of how the model really decided, so the final answer still needs checking.
A practical example
A logistics team builds an internal assistant that answers questions such as how many pallets fit in a shipment given box dimensions, weight limits and a mixed order of products. With direct prompting, the model gives wrong totals in roughly one answer out of four on a test set of 200 real questions, mostly because it skips a weight or stacking constraint.
The team changes the prompt to ask for numbered steps covering dimensions, weight and stacking, followed by a line that begins with “Answer:”. It also samples five reasoning paths and keeps the most common answer. Errors on the same test set drop to a small fraction of the previous rate, while the average response becomes about three times longer.
To control cost, the team routes simple single-product questions to a direct prompt and reserves chain-of-thought with self-consistency for mixed orders. The application shows staff only the final answer, while the full reasoning is logged for auditing when a shipment plan is disputed.