DSPy

On this page 8

DSPy is an open-source Python framework for building language model applications as declarative programs whose prompts and few-shot examples are optimized automatically against a metric instead of being written by hand. Among AI frameworks & tools in artificial intelligence, it occupies a distinct position: developers describe what each step must do, and DSPy decides how to phrase the instructions for a given model.

The name stands for Declarative Self-improving Python. The project originated at Stanford NLP and treats prompt engineering as a compilation problem: a program, a small training set and a scoring function go in, and a tuned version of the program comes out.

How DSPy separates program logic from prompts

DSPy works by splitting an application into typed interfaces and learnable parameters. The interface of each step is a signature, such as question -> answer or a class with named input and output fields. The logic is written as ordinary Python that calls modules, and the actual prompt text is generated and refined by an optimizer.

This separation means the same program can be recompiled for a different model without rewriting prompts. When a team moves from a large hosted model to a smaller local one, it reruns the optimizer, and DSPy searches for instructions and demonstrations that work for the new model. The table below lists the core building blocks.

ComponentRoleTypical example
SignatureDeclares input and output fields with short descriptionscontext, question -> answer
ModuleApplies a prompting strategy to a signaturedspy.ChainOfThought
ProgramPython class composing several modulesRetrieve, then answer, then verify
Example and datasetLabeled or partially labeled inputs for tuningdspy.Example with 50 to 300 items
MetricPython function that scores a predictionExact match, F1 or an LLM judge
OptimizerSearches instructions, demonstrations or weightsMIPROv2, BootstrapFewShot
Core building blocks of a DSPy program

Built-in modules and prompting strategies

Modules in DSPy are reusable prompting strategies that wrap any signature. A module holds its own instructions and demonstrations as parameters, which the optimizer later fills in. Developers can nest modules inside custom classes in the same way that layers are nested in PyTorch.

The choice of module determines how much reasoning the model performs before it emits the output fields, and therefore affects both quality and token cost. The following table compares the most common modules.

ModuleBehaviorBest suited for
PredictDirect mapping from inputs to outputsClassification and extraction
ChainOfThoughtAdds a reasoning field before the answerMulti-step questions and analysis
ProgramOfThoughtGenerates and executes code for the answerArithmetic and table calculations
ReActAlternates reasoning with tool callsAgents that search or call APIs
BestOfN and RefineSamples several attempts and keeps the bestOutputs that must satisfy a check
Common DSPy modules and when they fit

Language models are configured through dspy.LM, which routes calls through a provider-agnostic client and supports hosted and local endpoints alike. Responses are parsed into the declared output fields, and typed outputs such as lists, booleans or Pydantic models are validated before the program continues.

How DSPy optimizers tune a program

A DSPy optimizer improves a program by running it on training examples, scoring the results with the metric and keeping the best-scoring variants. Older documentation calls these components teleprompters. Optimizers differ in what they change: demonstrations, instructions or model weights.

Bootstrapping is the central idea. The optimizer executes the program, records the intermediate traces of runs that pass the metric, and turns them into few-shot demonstrations for each module, so labels are needed only for the final output. The table summarizes the main optimizers.

OptimizerWhat it changesTypical data need
LabeledFewShotInserts labeled examples as demonstrationsA handful of labeled items
BootstrapFewShotBuilds demonstrations from successful tracesAbout 10 to 50 examples
BootstrapFewShotWithRandomSearchTries many demonstration sets, keeps the best50 or more examples
MIPROv2Proposes instructions and demonstrations jointly200 or more for best results
GEPAEvolves instructions from textual feedbackMetric that returns feedback text
BootstrapFinetuneDistills traces into model weightsHundreds of examples and a tunable model
Main optimizer families and their requirements

Compilation has a real cost in model calls. A light optimization run may need a few hundred calls, while a thorough instruction search can require several thousand, so teams often optimize with a cheaper model and validate the result on the production model.

DSPy compared with other LLM frameworks

DSPy differs from orchestration frameworks because it optimizes prompts rather than only connecting components. LangChain and LlamaIndex supply integrations, retrievers and chains in which prompt templates remain largely hand-written. LangGraph focuses on stateful agent graphs, and evaluation platforms such as LangSmith measure quality without changing prompts automatically.

ToolPrimary focusPrompt handling
DSPyDeclarative programs with optimizationGenerated and tuned against a metric
LangChainIntegrations and composable chainsTemplates written by developers
LlamaIndexData indexing and retrieval pipelinesDefault templates, manual tuning
LangGraphStateful, cyclic agent workflowsDefined by developers per node
Pydantic AIType-safe agents and structured outputWritten by developers
Positioning of DSPy among neighboring tools

These tools are not mutually exclusive. A DSPy module can run inside a LangGraph node or call a retriever built with LlamaIndex, so DSPy is often adopted as the optimization layer of a larger stack rather than as a replacement.

When to choose DSPy and how to start

DSPy pays off when a task has a measurable notion of quality and the pipeline has several language model steps whose prompts interact. It is less useful for one-off prompts, creative writing without a scoring rule or prototypes where no evaluation data exists yet.

SituationRecommendationReason
Multi-stage RAG with labeled answersAdopt DSPyOptimizer tunes every stage jointly
Frequent model switchingAdopt DSPyRecompilation replaces manual rewriting
Single prompt, no metricWrite the prompt directlyNothing to optimize against
Strict regulatory wordingUse DSPy with fixed instructionsGenerated text needs human review
Decision guide for adopting DSPy
  1. Define signatures for each step and compose them in a module.
  2. Collect a development set of realistic inputs, with labels where possible.
  3. Write a metric that reflects real success, not a convenient proxy.
  4. Run a light optimizer, compare against the unoptimized baseline on a held-out set and save the compiled program.

Limitations and common mistakes

The quality of a DSPy program is bounded by the quality of its metric. A metric that rewards surface overlap can produce prompts that overfit to the training set, and small datasets make the reported gains noisy. Optimized prompts can also be long and hard to read, which complicates audits.

  • Evaluating on training data inflates scores; always keep a separate test split.
  • LLM-judge metrics inherit the bias of the judging model and need spot checks.
  • Underestimating compile cost leads to surprise API bills on large searches.
  • Forgetting to save the compiled program forces repeated, nondeterministic optimization runs.

Put simply

Most AI applications depend on carefully worded instructions, and people usually polish those words by trial and error. DSPy lets developers describe each step in plain terms, such as question in, answer out, and then lets the computer test many wordings and examples to find what works best.

It is similar to a coach who watches practice runs, keeps the moves that score points and drops the rest. The key idea is simple: define the goal and a way to measure it, and DSPy does the tedious tuning.

In practice

A support team at a software company runs a question-answering assistant over about 4,000 help articles. The original pipeline retrieves five passages and asks a hosted model to answer with a hand-written prompt, and an internal review finds that roughly one answer in four cites the wrong article.

An engineer rebuilds the pipeline as a DSPy program with a query-rewriting module, a retriever and a ChainOfThought answer module. She labels 300 past tickets, writes a metric that checks both answer correctness and the cited article, and compiles the program with MIPROv2 using a mid-sized model, which takes a few thousand calls.

On a held-out set of 100 tickets, correct and properly cited answers rise noticeably over the baseline. When the company later moves to a cheaper model, the team recompiles the same program in an afternoon instead of rewriting prompts by hand.