Prompt chaining

On this page 7

Prompt chaining is a technique for working with language models in which a complex task is decomposed into a sequence of separate model calls, and the output of each call becomes part of the input to the next. Instead of asking one large prompt to research, reason, format and verify at once, the application runs small, focused prompts in a fixed order, often with code between them that parses, validates or transforms intermediate results.

In artificial intelligence engineering, prompt chaining sits between a single prompt and a fully autonomous agent. The control flow is defined by the developer, not chosen by the model, which makes chains predictable, testable and easy to debug. The technique is a core pattern of prompt engineering for production systems built on an LLM.

How a prompt chain passes work between steps

A prompt chain works by turning one goal into a pipeline of narrow subtasks, each with its own instructions, inputs and expected output format. Step one might extract facts from a document, step two might classify them, and step three might draft a summary from the classified facts only. Every step receives only the context it needs, which keeps prompts short and reduces distraction inside the context window.

Between steps, ordinary code acts as glue logic. It parses JSON, checks that required fields exist, filters items, calls external tools or databases, and decides which prompt runs next. This gate between calls is what separates a chain from a long conversation: errors can be caught before they propagate, and a failing step can be retried without repeating the whole task.

The table below lists the components that appear in most production chains and the role each one plays.

ComponentRole in the chainTypical implementation
Step promptPerforms one narrow subtaskTemplate with variables and a fixed output format
Intermediate outputCarries results to the next stepJSON object, list or short text
Validation gateRejects malformed or implausible outputSchema check, regular expression, rule or second model
RouterSelects the next step from the resultConditional code or a classification prompt
State storeKeeps data shared across stepsVariables, cache or database record
Components of a typical prompt chain

Common prompt chaining patterns

Prompt chaining appears in several recurring shapes, and most real systems combine two or three of them. The simplest is a linear chain, where steps always run in the same order. More advanced designs add branching, parallel work or loops that revise a draft until it passes a check.

PatternStructureBest suited for
Linear chainFixed sequence of stepsExtraction, transformation and formatting pipelines
Conditional routingA classifier picks one branchSupport tickets, intent handling, mixed input types
Map and reduceSame prompt on many chunks, then a merge stepLong documents beyond one context window
Generate and critiqueDraft, review, then reviseWriting quality, code review, policy compliance
Retrieval chainQuery rewriting, search, then answerGrounded answers over private documents
Prompt chaining patterns and when each applies

The generate and critique loop deserves care. A reviewing prompt can catch omissions and format errors well, but a model reviewing its own output tends to share the same blind spots, so factual checks work better when the critic receives external evidence or a deterministic test.

Prompt chaining is often confused with chain-of-thought and with agents, although the three differ in who controls the steps and where the reasoning happens. Chain-of-thought keeps all reasoning inside one model response; prompt chaining spreads work across several calls with code in between; an agent lets the model decide its own next action in a loop.

ApproachWho controls the flowNumber of callsMain strength
Single promptNo flow, one responseOneLowest latency and cost
Chain-of-thoughtThe model, within one answerOneBetter multistep reasoning in place
Prompt chainingThe developer, in codeSeveral, fixedPredictability and inspectable steps
AI agentThe model, in a loopVariableFlexibility on open-ended tasks
How prompt chaining differs from neighboring approaches

A practical rule follows from this comparison: use a chain when the steps are known in advance, and consider an agent only when the path genuinely depends on what the model discovers along the way. Many teams find that a well-designed chain handles most business workflows with fewer failures than an agent.

Designing a reliable prompt chain

A reliable chain starts with clear step boundaries: each prompt should have one job that can be described in a single sentence and evaluated on its own. When a step needs two unrelated skills, splitting it usually raises accuracy more than rewording the instructions.

  • Define a strict output format for every step, ideally a JSON schema, and use constrained decoding or structured output where the platform supports it.
  • Set temperature low for extraction and classification steps, and allow higher values only in creative drafting steps.
  • Pass only the fields the next step needs, rather than the full transcript of earlier calls.
  • Log the input and output of every step, so that a wrong final answer can be traced to the step that caused it.
  • Choose a model per step: a small language model often handles routing or extraction, while a larger reasoning model handles the hardest step.

Evaluation is easier with chains than with single prompts because each step can have its own test set and metric. A team can measure extraction precision, routing accuracy and summary quality separately, then improve the weakest link instead of tuning one opaque prompt.

Limitations and failure modes of prompt chaining

The main weakness of prompt chaining is error propagation: a mistake in an early step is passed forward as if it were fact, and later steps rarely question it. Per-step accuracy also multiplies, so five steps that each succeed 95 percent of the time give an end-to-end success rate of only about 77 percent without validation.

SymptomLikely causeFix
Final answer contains invented factsHallucination accepted in an early stepValidate against source text before passing on
Parsing errors between stepsFree-form output instead of a schemaEnforce structured output and retry on failure
High latency for usersToo many sequential callsMerge trivial steps, run independent steps in parallel
Rising cost per requestFull history copied into every stepPass compact summaries or selected fields
Inconsistent tone or termsSteps written without shared conventionsReuse one system prompt and glossary
Typical chain failures, causes and fixes

Chains also add latency and cost, because every step is a separate request with its own input tokens. For a simple task, a single well-written prompt is often faster and just as accurate, so chaining should be justified by measurable gains in quality or control, not used by default.

The idea in brief

Prompt chaining means breaking a big request into small steps and giving each step to the AI model separately. The answer from one step is handed to the next, much like an assembly line where each station does one job and passes the part along.

Because every station is simple, it is easier to spot where something went wrong and fix only that station. The trade-off is that the whole line takes a little longer and costs a little more than one quick request. The key benefit is control: people decide the order of the work, and they can check each piece before it moves on.

A practical scenario

A mid-sized insurance company receives about 2,000 claim emails a day and wants a short, structured summary of each one for its adjusters. A single prompt that reads the email and writes the summary produces uneven results: policy numbers are sometimes missed and damage types are mixed up.

The engineering team builds a four-step chain. The first prompt extracts fields into JSON, such as policy number, date and claimed amount, and code checks the policy number against the internal database. The second prompt classifies the claim type with a small, inexpensive model. The third prompt drafts the summary using only the validated fields and the original email, and the fourth prompt checks the draft against a list of required elements.

After two weeks of testing on 500 labeled emails, the team reports that missing policy numbers dropped to almost zero because invalid values were caught at the gate, and adjusters spend noticeably less time correcting summaries. Total latency rises to about six seconds per email, which is acceptable for a background process.