DSPy is an open-source Python framework for building language model applications as declarative programs whose prompts and few-shot examples are optimized automatically against a metric instead of being written by hand. Among AI frameworks & tools in artificial intelligence, it occupies a distinct position: developers describe what each step must do, and DSPy decides how to phrase the instructions for a given model.
The name stands for Declarative Self-improving Python. The project originated at Stanford NLP and treats prompt engineering as a compilation problem: a program, a small training set and a scoring function go in, and a tuned version of the program comes out.
How DSPy separates program logic from prompts
DSPy works by splitting an application into typed interfaces and learnable parameters. The interface of each step is a signature, such as question -> answer or a class with named input and output fields. The logic is written as ordinary Python that calls modules, and the actual prompt text is generated and refined by an optimizer.
This separation means the same program can be recompiled for a different model without rewriting prompts. When a team moves from a large hosted model to a smaller local one, it reruns the optimizer, and DSPy searches for instructions and demonstrations that work for the new model. The table below lists the core building blocks.
| Component | Role | Typical example |
|---|---|---|
| Signature | Declares input and output fields with short descriptions | context, question -> answer |
| Module | Applies a prompting strategy to a signature | dspy.ChainOfThought |
| Program | Python class composing several modules | Retrieve, then answer, then verify |
| Example and dataset | Labeled or partially labeled inputs for tuning | dspy.Example with 50 to 300 items |
| Metric | Python function that scores a prediction | Exact match, F1 or an LLM judge |
| Optimizer | Searches instructions, demonstrations or weights | MIPROv2, BootstrapFewShot |
Built-in modules and prompting strategies
Modules in DSPy are reusable prompting strategies that wrap any signature. A module holds its own instructions and demonstrations as parameters, which the optimizer later fills in. Developers can nest modules inside custom classes in the same way that layers are nested in PyTorch.
The choice of module determines how much reasoning the model performs before it emits the output fields, and therefore affects both quality and token cost. The following table compares the most common modules.
| Module | Behavior | Best suited for |
|---|---|---|
| Predict | Direct mapping from inputs to outputs | Classification and extraction |
| ChainOfThought | Adds a reasoning field before the answer | Multi-step questions and analysis |
| ProgramOfThought | Generates and executes code for the answer | Arithmetic and table calculations |
| ReAct | Alternates reasoning with tool calls | Agents that search or call APIs |
| BestOfN and Refine | Samples several attempts and keeps the best | Outputs that must satisfy a check |
Language models are configured through dspy.LM, which routes calls through a provider-agnostic client and supports hosted and local endpoints alike. Responses are parsed into the declared output fields, and typed outputs such as lists, booleans or Pydantic models are validated before the program continues.
How DSPy optimizers tune a program
A DSPy optimizer improves a program by running it on training examples, scoring the results with the metric and keeping the best-scoring variants. Older documentation calls these components teleprompters. Optimizers differ in what they change: demonstrations, instructions or model weights.
Bootstrapping is the central idea. The optimizer executes the program, records the intermediate traces of runs that pass the metric, and turns them into few-shot demonstrations for each module, so labels are needed only for the final output. The table summarizes the main optimizers.
| Optimizer | What it changes | Typical data need |
|---|---|---|
| LabeledFewShot | Inserts labeled examples as demonstrations | A handful of labeled items |
| BootstrapFewShot | Builds demonstrations from successful traces | About 10 to 50 examples |
| BootstrapFewShotWithRandomSearch | Tries many demonstration sets, keeps the best | 50 or more examples |
| MIPROv2 | Proposes instructions and demonstrations jointly | 200 or more for best results |
| GEPA | Evolves instructions from textual feedback | Metric that returns feedback text |
| BootstrapFinetune | Distills traces into model weights | Hundreds of examples and a tunable model |
Compilation has a real cost in model calls. A light optimization run may need a few hundred calls, while a thorough instruction search can require several thousand, so teams often optimize with a cheaper model and validate the result on the production model.
DSPy compared with other LLM frameworks
DSPy differs from orchestration frameworks because it optimizes prompts rather than only connecting components. LangChain and LlamaIndex supply integrations, retrievers and chains in which prompt templates remain largely hand-written. LangGraph focuses on stateful agent graphs, and evaluation platforms such as LangSmith measure quality without changing prompts automatically.
| Tool | Primary focus | Prompt handling |
|---|---|---|
| DSPy | Declarative programs with optimization | Generated and tuned against a metric |
| LangChain | Integrations and composable chains | Templates written by developers |
| LlamaIndex | Data indexing and retrieval pipelines | Default templates, manual tuning |
| LangGraph | Stateful, cyclic agent workflows | Defined by developers per node |
| Pydantic AI | Type-safe agents and structured output | Written by developers |
These tools are not mutually exclusive. A DSPy module can run inside a LangGraph node or call a retriever built with LlamaIndex, so DSPy is often adopted as the optimization layer of a larger stack rather than as a replacement.
When to choose DSPy and how to start
DSPy pays off when a task has a measurable notion of quality and the pipeline has several language model steps whose prompts interact. It is less useful for one-off prompts, creative writing without a scoring rule or prototypes where no evaluation data exists yet.
| Situation | Recommendation | Reason |
|---|---|---|
| Multi-stage RAG with labeled answers | Adopt DSPy | Optimizer tunes every stage jointly |
| Frequent model switching | Adopt DSPy | Recompilation replaces manual rewriting |
| Single prompt, no metric | Write the prompt directly | Nothing to optimize against |
| Strict regulatory wording | Use DSPy with fixed instructions | Generated text needs human review |
- Define signatures for each step and compose them in a module.
- Collect a development set of realistic inputs, with labels where possible.
- Write a metric that reflects real success, not a convenient proxy.
- Run a light optimizer, compare against the unoptimized baseline on a held-out set and save the compiled program.
Limitations and common mistakes
The quality of a DSPy program is bounded by the quality of its metric. A metric that rewards surface overlap can produce prompts that overfit to the training set, and small datasets make the reported gains noisy. Optimized prompts can also be long and hard to read, which complicates audits.
- Evaluating on training data inflates scores; always keep a separate test split.
- LLM-judge metrics inherit the bias of the judging model and need spot checks.
- Underestimating compile cost leads to surprise API bills on large searches.
- Forgetting to save the compiled program forces repeated, nondeterministic optimization runs.
Put simply
Most AI applications depend on carefully worded instructions, and people usually polish those words by trial and error. DSPy lets developers describe each step in plain terms, such as question in, answer out, and then lets the computer test many wordings and examples to find what works best.
It is similar to a coach who watches practice runs, keeps the moves that score points and drops the rest. The key idea is simple: define the goal and a way to measure it, and DSPy does the tedious tuning.
In practice
A support team at a software company runs a question-answering assistant over about 4,000 help articles. The original pipeline retrieves five passages and asks a hosted model to answer with a hand-written prompt, and an internal review finds that roughly one answer in four cites the wrong article.
An engineer rebuilds the pipeline as a DSPy program with a query-rewriting module, a retriever and a ChainOfThought answer module. She labels 300 past tickets, writes a metric that checks both answer correctness and the cited article, and compiles the program with MIPROv2 using a mid-sized model, which takes a few thousand calls.
On a held-out set of 100 tickets, correct and properly cited answers rise noticeably over the baseline. When the company later moves to a cheaper model, the team recompiles the same program in an afternoon instead of rewriting prompts by hand.