Haystack

Also known as: deepset Haystack

On this page 7

Haystack is an open-source Python framework, maintained by the Berlin company deepset, for building production-ready applications on top of language models, most notably retrieval-augmented generation (RAG), semantic search, question answering and tool-using agents. Among AI frameworks & tools, it occupies the orchestration layer: it does not train models, but connects models, document stores, retrievers and prompts into explicit, inspectable pipelines.

The framework is often called deepset Haystack to distinguish it from unrelated projects with the same name. Its current generation, Haystack 2.x, released in 2024 under the Apache 2.0 license, replaced the older node-based API with a design built on typed components and directed graphs, which makes data flow visible and testable.

How Haystack pipelines connect components

A Haystack application is a pipeline of components in which each component declares typed inputs and outputs, and the pipeline routes data between them. Developers add components to a Pipeline object and then call connect() to link a named output socket to a matching input socket, so the framework can validate types before execution.

Pipelines form a directed graph that may contain loops and branches. Loops allow patterns such as self-correction, where an output validator sends a malformed answer back to the generator, while routers send a query to different branches depending on its language, file type or classification. A custom component is an ordinary Python class marked with the @component decorator and exposing a run() method, which keeps extension simple and framework-neutral.

The following table lists the main component families and what each contributes to a typical application.

Component familyTypical examplesRole in the pipeline
ConvertersPDF, HTML and DOCX convertersTurn files into Document objects
PreprocessorsDocumentCleaner, DocumentSplitterClean text and split it into chunks
EmbeddersSentence Transformers, OpenAI embeddersCreate dense vectors for text
RetrieversBM25 and embedding retrieversFetch candidate documents for a query
RankersCross-encoder rankersReorder candidates by relevance
BuildersPromptBuilder, ChatPromptBuilderFill Jinja2 prompt templates
GeneratorsChat generators for hosted or local modelsCall the language model
RoutersConditional and file type routersChoose the next branch
Core component families in Haystack 2.x

Indexing and query pipelines in Haystack

Most Haystack projects consist of two separate pipelines: an indexing pipeline that prepares content and a query pipeline that answers requests. Separating them lets teams re-index documents on a schedule without touching the serving path, and lets them change prompts or models without rebuilding the index.

Both pipelines share a Document Store, the abstraction over the database that holds text, metadata and embeddings. Haystack ships an in-memory store for prototyping and maintains integrations for Elasticsearch, OpenSearch, Qdrant, Weaviate, Pinecone, Chroma and PostgreSQL with pgvector. The table compares the two pipeline roles.

AspectIndexing pipelineQuery pipeline
TriggerNew or changed filesUser request
Typical stepsConvert, clean, split, embed, writeEmbed query, retrieve, rank, prompt, generate
Latency budgetMinutes to hoursUsually under a few seconds
Main cost driverEmbedding volumeModel tokens per request
Failure impactStale or missing contentWrong or slow answers
Indexing pipeline versus query pipeline

Chunking decisions made at indexing time strongly shape answer quality later. Chunks of roughly 150 to 400 words with a small overlap are a common starting point, but the right size depends on document structure and on the context window of the generator.

Haystack compared with LangChain and LlamaIndex

Haystack differs from LangChain and LlamaIndex mainly in its emphasis on explicit pipeline graphs and production stability rather than breadth of integrations. All three frameworks can build RAG systems and agents, so the choice usually rests on team preferences, required integrations and how much control the team wants over data flow.

FrameworkCentral abstractionTypical strength
HaystackTyped components in a pipeline graphTransparent, serializable production pipelines
LangChainRunnables and chains, with LangGraph for agentsVery broad integration catalog
LlamaIndexIndexes, query engines and workflowsData ingestion and retrieval depth
Semantic KernelKernel with plugins and functionsIntegration with .NET and enterprise stacks
Orchestration frameworks at a glance

A practical advantage of Haystack is pipeline serialization to YAML. A pipeline built in Python can be saved, versioned in Git, reviewed like configuration and loaded again in another environment, which supports reproducible deployments. The companion tool Hayhooks exposes such pipelines as REST endpoints.

Agents, tools and evaluation

Haystack 2.x supports tool-calling agents through an Agent component that wraps a chat generator and a set of tools. A tool can be a Python function, a whole pipeline wrapped as a tool, or a server reached through the Model Context Protocol. The agent loops between model calls and tool executions until an exit condition is met, such as a final text answer.

The framework also includes evaluator components that measure pipeline quality. Retrieval metrics such as recall and mean reciprocal rank check whether the right documents are found, while model-based evaluators judge faithfulness to the retrieved context and answer relevance. The table summarizes common checks.

CheckWhat it measuresNeeds ground truth
Document recallShare of relevant documents retrievedYes
Mean reciprocal rankPosition of the first relevant documentYes
FaithfulnessWhether claims are supported by contextNo
Context relevanceWhether retrieved text fits the questionNo
Semantic answer similarityCloseness to a reference answerYes
Common evaluation checks for Haystack pipelines

Limitations and common mistakes

Haystack simplifies orchestration, but it does not solve retrieval quality by itself. Poor source documents, careless chunking and missing metadata produce weak answers regardless of the framework. Teams also need their own monitoring, access control and cost tracking, often through tracing integrations such as Langfuse or OpenTelemetry.

  • Mixing 1.x and 2.x code: the old farm-haystack package and the current haystack-ai package have incompatible APIs, and many older tutorials still show the legacy nodes.
  • Using the in-memory store in production: it loses data on restart and does not scale to large collections.
  • Skipping a ranker: embedding retrieval alone often returns loosely related chunks, and a cross-encoder ranker usually improves precision.
  • No evaluation set: without a few dozen labeled questions, changes to prompts or chunking cannot be judged objectively.
  • Embedding model mismatch: documents and queries must be embedded with the same model and dimension, or retrieval fails silently.

Explained without jargon

Haystack is a toolkit for programmers who want an AI assistant to answer questions from their own documents. Think of it as an assembly line for answers. One station reads files. Another cuts them into small pieces. A third finds the pieces that match a question. The last one asks a language model to write a reply from those pieces.

Each station can be swapped or tested on its own. The key takeaway is simple. Haystack does not make the AI smarter. It organizes the steps around the AI, so answers rest on the right information and the system stays easy to maintain.

A typical use case

An engineering team at a mid-sized industrial manufacturer wants service technicians to query about 12,000 pages of equipment manuals in plain language. The team builds an indexing pipeline that converts the PDF manuals, splits them into chunks of about 250 words with 30 words of overlap, embeds them and writes them to an OpenSearch document store together with metadata for machine model and manual version.

The query pipeline combines keyword and embedding retrieval, applies a cross-encoder ranker to keep the five best chunks and fills a prompt that instructs the model to cite the manual section. A metadata filter restricts results to the machine model selected in the technician app. Before launch, the team evaluates the system on 120 real support questions and raises document recall from about 70 to above 90 percent by adding the ranker and hybrid retrieval.

The outcome is a serialized YAML pipeline deployed behind a REST endpoint, with answers that point technicians to the exact manual page and a measurable drop in escalations to senior engineers.