Haystack is an open-source Python framework, maintained by the Berlin company deepset, for building production-ready applications on top of language models, most notably retrieval-augmented generation (RAG), semantic search, question answering and tool-using agents. Among AI frameworks & tools, it occupies the orchestration layer: it does not train models, but connects models, document stores, retrievers and prompts into explicit, inspectable pipelines.
The framework is often called deepset Haystack to distinguish it from unrelated projects with the same name. Its current generation, Haystack 2.x, released in 2024 under the Apache 2.0 license, replaced the older node-based API with a design built on typed components and directed graphs, which makes data flow visible and testable.
How Haystack pipelines connect components
A Haystack application is a pipeline of components in which each component declares typed inputs and outputs, and the pipeline routes data between them. Developers add components to a Pipeline object and then call connect() to link a named output socket to a matching input socket, so the framework can validate types before execution.
Pipelines form a directed graph that may contain loops and branches. Loops allow patterns such as self-correction, where an output validator sends a malformed answer back to the generator, while routers send a query to different branches depending on its language, file type or classification. A custom component is an ordinary Python class marked with the @component decorator and exposing a run() method, which keeps extension simple and framework-neutral.
The following table lists the main component families and what each contributes to a typical application.
| Component family | Typical examples | Role in the pipeline |
|---|---|---|
| Converters | PDF, HTML and DOCX converters | Turn files into Document objects |
| Preprocessors | DocumentCleaner, DocumentSplitter | Clean text and split it into chunks |
| Embedders | Sentence Transformers, OpenAI embedders | Create dense vectors for text |
| Retrievers | BM25 and embedding retrievers | Fetch candidate documents for a query |
| Rankers | Cross-encoder rankers | Reorder candidates by relevance |
| Builders | PromptBuilder, ChatPromptBuilder | Fill Jinja2 prompt templates |
| Generators | Chat generators for hosted or local models | Call the language model |
| Routers | Conditional and file type routers | Choose the next branch |
Indexing and query pipelines in Haystack
Most Haystack projects consist of two separate pipelines: an indexing pipeline that prepares content and a query pipeline that answers requests. Separating them lets teams re-index documents on a schedule without touching the serving path, and lets them change prompts or models without rebuilding the index.
Both pipelines share a Document Store, the abstraction over the database that holds text, metadata and embeddings. Haystack ships an in-memory store for prototyping and maintains integrations for Elasticsearch, OpenSearch, Qdrant, Weaviate, Pinecone, Chroma and PostgreSQL with pgvector. The table compares the two pipeline roles.
| Aspect | Indexing pipeline | Query pipeline |
|---|---|---|
| Trigger | New or changed files | User request |
| Typical steps | Convert, clean, split, embed, write | Embed query, retrieve, rank, prompt, generate |
| Latency budget | Minutes to hours | Usually under a few seconds |
| Main cost driver | Embedding volume | Model tokens per request |
| Failure impact | Stale or missing content | Wrong or slow answers |
Chunking decisions made at indexing time strongly shape answer quality later. Chunks of roughly 150 to 400 words with a small overlap are a common starting point, but the right size depends on document structure and on the context window of the generator.
Haystack compared with LangChain and LlamaIndex
Haystack differs from LangChain and LlamaIndex mainly in its emphasis on explicit pipeline graphs and production stability rather than breadth of integrations. All three frameworks can build RAG systems and agents, so the choice usually rests on team preferences, required integrations and how much control the team wants over data flow.
| Framework | Central abstraction | Typical strength |
|---|---|---|
| Haystack | Typed components in a pipeline graph | Transparent, serializable production pipelines |
| LangChain | Runnables and chains, with LangGraph for agents | Very broad integration catalog |
| LlamaIndex | Indexes, query engines and workflows | Data ingestion and retrieval depth |
| Semantic Kernel | Kernel with plugins and functions | Integration with .NET and enterprise stacks |
A practical advantage of Haystack is pipeline serialization to YAML. A pipeline built in Python can be saved, versioned in Git, reviewed like configuration and loaded again in another environment, which supports reproducible deployments. The companion tool Hayhooks exposes such pipelines as REST endpoints.
Agents, tools and evaluation
Haystack 2.x supports tool-calling agents through an Agent component that wraps a chat generator and a set of tools. A tool can be a Python function, a whole pipeline wrapped as a tool, or a server reached through the Model Context Protocol. The agent loops between model calls and tool executions until an exit condition is met, such as a final text answer.
The framework also includes evaluator components that measure pipeline quality. Retrieval metrics such as recall and mean reciprocal rank check whether the right documents are found, while model-based evaluators judge faithfulness to the retrieved context and answer relevance. The table summarizes common checks.
| Check | What it measures | Needs ground truth |
|---|---|---|
| Document recall | Share of relevant documents retrieved | Yes |
| Mean reciprocal rank | Position of the first relevant document | Yes |
| Faithfulness | Whether claims are supported by context | No |
| Context relevance | Whether retrieved text fits the question | No |
| Semantic answer similarity | Closeness to a reference answer | Yes |
Limitations and common mistakes
Haystack simplifies orchestration, but it does not solve retrieval quality by itself. Poor source documents, careless chunking and missing metadata produce weak answers regardless of the framework. Teams also need their own monitoring, access control and cost tracking, often through tracing integrations such as Langfuse or OpenTelemetry.
- Mixing 1.x and 2.x code: the old
farm-haystackpackage and the currenthaystack-aipackage have incompatible APIs, and many older tutorials still show the legacy nodes. - Using the in-memory store in production: it loses data on restart and does not scale to large collections.
- Skipping a ranker: embedding retrieval alone often returns loosely related chunks, and a cross-encoder ranker usually improves precision.
- No evaluation set: without a few dozen labeled questions, changes to prompts or chunking cannot be judged objectively.
- Embedding model mismatch: documents and queries must be embedded with the same model and dimension, or retrieval fails silently.
Explained without jargon
Haystack is a toolkit for programmers who want an AI assistant to answer questions from their own documents. Think of it as an assembly line for answers. One station reads files. Another cuts them into small pieces. A third finds the pieces that match a question. The last one asks a language model to write a reply from those pieces.
Each station can be swapped or tested on its own. The key takeaway is simple. Haystack does not make the AI smarter. It organizes the steps around the AI, so answers rest on the right information and the system stays easy to maintain.
A typical use case
An engineering team at a mid-sized industrial manufacturer wants service technicians to query about 12,000 pages of equipment manuals in plain language. The team builds an indexing pipeline that converts the PDF manuals, splits them into chunks of about 250 words with 30 words of overlap, embeds them and writes them to an OpenSearch document store together with metadata for machine model and manual version.
The query pipeline combines keyword and embedding retrieval, applies a cross-encoder ranker to keep the five best chunks and fills a prompt that instructs the model to cite the manual section. A metadata filter restricts results to the machine model selected in the technician app. Before launch, the team evaluates the system on 120 real support questions and raises document recall from about 70 to above 90 percent by adding the ranker and hybrid retrieval.
The outcome is a serialized YAML pipeline deployed behind a REST endpoint, with answers that point technicians to the exact manual page and a measurable drop in escalations to senior engineers.