Decomposed prompting

Also known as: DecomP

On this page 7

Decomposed prompting (DecomP) is a modular prompting technique in which a decomposer prompt breaks a complex task into a sequence of simpler sub-tasks and sends each one to a dedicated sub-task handler: another prompt, a specialized model or a symbolic tool. In prompting for large language models, it treats a solution as a small program of prompt calls rather than a single long reasoning trace.

The approach was introduced by Khot and colleagues in 2022 and presented at ICLR 2023. Its central idea is borrowed from software engineering: separate the control logic from the workers, so that each component can be designed, tested, improved or replaced independently.

How decomposed prompting structures a task

Decomposed prompting works by letting the decomposer emit one sub-task call at a time, executing it, and feeding the result back before the next step is chosen. The decomposer prompt contains few-shot demonstrations written as sequences of question and handler pairs, so the model learns which handler name to invoke and what sub-question to pass.

A controller outside the model parses each call, routes it to the named handler and appends the answer to the running trace. The loop ends when the decomposer emits a termination marker such as an end-of-question token. Because handlers can themselves be decomposed, the method supports hierarchical and recursive decomposition, for example reversing a long list by splitting it into halves.

ComponentWhat it doesTypical implementation
Decomposer promptPlans the next sub-task and its inputFew-shot prompt with handler calls
Sub-task handlerSolves one narrow operationDedicated few-shot prompt
Symbolic handlerPerforms exact or external operationsRetrieval, Python function, calculator
ControllerParses calls and passes resultsShort orchestration code
TraceRecords calls and answers so farText appended to the decomposer context
Roles in a decomposed prompting system

Types of sub-task handlers

The strength of DecomP lies in mixing handler types within one solution. A handler only needs a defined input and output, so the decomposer does not care whether a language model, a search engine or ordinary code answers the call.

The following table shows the main handler categories and when each fits. In the original experiments, replacing a weak prompt-based handler with a symbolic retrieval function produced the largest gains on open-domain multi-hop questions.

Handler typeBest forMain advantageMain risk
Few-shot promptNarrow linguistic sub-tasksFocused examples for one skillErrors still possible
Recursive decomposer callInputs too long for one stepScales to arbitrary lengthDeep call chains add latency
Retrieval functionFact lookup in a corpusGrounds answers in documentsDepends on index quality
Code functionArithmetic, sorting, string operationsExact and deterministic resultsRequires correct argument parsing
Fine-tuned modelHigh-volume repeated sub-taskLower cost per callTraining and maintenance effort
Handler categories and their best uses

Decomposed prompting differs from neighboring techniques mainly in who executes each step. In chain-of-thought prompting, one model writes every step inside a single response; in DecomP, steps are delegated to separate handlers with their own prompts or code.

TechniqueHow steps are producedKey difference from DecomP
Chain-of-thoughtOne model reasons in a single outputNo modular handlers or tools
Least-to-most promptingDecompose first, then solve in orderFixed plan, one shared solver prompt
Self-ask promptingModel asks and answers follow-up questionsUsually one handler, optionally search
Program-aided language modelModel writes code that an interpreter runsWhole solution expressed as one program
Plan-and-solve promptingModel writes a plan, then executes itPlan and execution in the same prompt
How DecomP relates to neighboring prompting methods

DecomP can be seen as a generalization of least-to-most prompting: it adds interleaved planning, specialized handlers and recursion. It also anticipates later agent and tool-calling patterns, where a model chooses functions from a defined catalog.

Designing a decomposed prompting pipeline

A good DecomP design starts by identifying the sub-tasks on which a single prompt fails, then giving each one a handler with a clear contract. Teams that decompose too finely pay for many model calls without a gain in accuracy.

  1. Run the single-prompt baseline and collect its failure cases.
  2. Name the recurring operations behind those failures, such as lookup, extraction or arithmetic.
  3. Define each handler contract with an exact input and output format.
  4. Write decomposer demonstrations that show realistic call sequences.
  5. Evaluate each handler in isolation before testing the full pipeline.

The decision table below helps judge whether the technique is worth its overhead for a given workload. The number of model calls typically grows linearly with the number of sub-steps, so latency and cost rise accordingly.

SituationRecommendationReason
Multi-hop questions over documentsUse DecomP with retrieval handlersEach hop needs grounded evidence
Exact symbolic operationsUse DecomP with code handlersModels are unreliable at exact manipulation
Short single-step questionsUse a single promptOrchestration overhead brings no benefit
Strict latency budgetPrefer chain-of-thoughtSequential calls add delay
When to choose decomposed prompting

Limitations and failure modes

Decomposed prompting introduces new failure points at the boundaries between components. A wrong decomposition sends correct handlers the wrong questions, and errors propagate because later steps trust earlier answers.

  • Planning errors: the decomposer skips a needed step or calls an unsuitable handler.
  • Format mismatch: a handler returns text the controller cannot parse.
  • Error cascades: one wrong intermediate answer corrupts every dependent step.
  • Higher cost and latency: many sequential calls replace one response.
  • Maintenance burden: each handler prompt needs its own tests and versioning.

Mitigations include validating handler outputs against a schema, adding a verification handler for critical steps, and logging the full trace so failures can be attributed to a specific component instead of the whole system.

Put simply

Decomposed prompting is like a project manager who splits a big job into small tasks and hands each one to the right specialist. One prompt plans the work, other prompts or tools do the individual pieces, and the results are put together at the end.

This helps because an AI model that struggles with a long, tangled request often does well on small, clearly defined questions. The price is that the system needs more steps, so it is slower and costs more to run.

In practice

A support team at a software company needs answers to questions such as which product release fixed a bug that a customer reported against a specific version. A single prompt often guesses, because the facts are spread across release notes, bug reports and version tables.

The team builds a DecomP pipeline with three handlers: a retrieval function over the bug tracker, a retrieval function over release notes, and a small code function that compares version numbers. The decomposer first finds the bug identifier, then looks up its fix, then checks which release contains that fix. On a test set of 200 historical questions, the decomposed version answers clearly more questions correctly than the single prompt, and each wrong answer can be traced to the failing step. The trade-off is an average of four model calls per question instead of one.