LlamaIndex is an open-source data framework for building applications in which a large language model answers questions over private or domain-specific data. Among AI frameworks & tools, it specializes in the data side of generative applications: it loads documents, splits them into nodes, indexes them and retrieves the most relevant context at query time, which makes it a standard toolkit for retrieval-augmented generation (RAG).
The project started in late 2022 under the name GPT Index and was renamed LlamaIndex in 2023. It is available as a Python library and as a TypeScript port, and it is distributed as a small core package plus many integration packages for model providers, vector databases and data loaders. Its scope has since grown from indexing to agents and event-driven workflows, but retrieval over documents remains its center of gravity.
How LlamaIndex turns documents into answers
LlamaIndex works as a two-stage pipeline: an ingestion stage that prepares data once, and a query stage that runs for every user request. During ingestion, readers load raw files into Document objects, node parsers split them into chunks called nodes, and an embedding model converts each node into a vector stored in an index or an external vector store.
At query time, a retriever selects the nodes most similar to the question, optional node postprocessors filter or rerank them, and a response synthesizer sends the selected context together with the question to the language model. The component that wires these steps together is the query engine; a chat engine adds conversation memory on top of it.
The following table lists the core building blocks and the role each one plays in the pipeline.
| Component | Stage | What it does |
|---|---|---|
| Reader (data loader) | Ingestion | Loads PDFs, web pages, databases or APIs into documents |
| Node parser | Ingestion | Splits documents into chunks with metadata and relationships |
| Ingestion pipeline | Ingestion | Chains transformations and caches their results |
| Index | Ingestion | Organizes nodes for efficient lookup |
| Retriever | Query | Returns the top-k relevant nodes for a question |
| Node postprocessor | Query | Reranks, filters or trims retrieved nodes |
| Response synthesizer | Query | Builds prompts and generates the final answer |
Index types and retrieval strategies
An index in LlamaIndex is a data structure that determines how nodes are found, and the choice of index shapes both answer quality and cost. The VectorStoreIndex is the default for most projects, but other index types suit summarization, keyword lookup or relationship-heavy data.
| Index type | Retrieval method | Best suited for |
|---|---|---|
| VectorStoreIndex | Embedding similarity, top-k | Question answering over large corpora |
| SummaryIndex | Reads all nodes in sequence | Summaries of a small document set |
| TreeIndex | Hierarchical summaries, traversal | Layered summarization of long texts |
| KeywordTableIndex | Extracted keyword matching | Exact terms, codes and identifiers |
| PropertyGraphIndex | Graph traversal plus vectors | Entities and relationships across documents |
Beyond the index itself, LlamaIndex offers advanced retrieval patterns. Hybrid search combines dense vectors with keyword scoring such as BM25, recursive retrieval follows links from summaries to detailed chunks, and the router query engine lets the model choose among several indexes. Sub-question decomposition splits a complex question into simpler ones and merges the partial answers.
Response synthesis also has modes. Compact packs as many nodes as fit into each prompt, refine iterates through chunks and improves the answer step by step, and tree_summarize builds the answer bottom-up. The trade-off is consistent: more model calls give more thorough answers but raise latency and cost.
LlamaIndex compared with related frameworks
LlamaIndex overlaps with several orchestration frameworks, and the practical difference lies in their center of gravity. LlamaIndex is data-first, with the deepest set of abstractions for parsing, chunking, indexing and retrieval, while its neighbors emphasize chains, agent graphs or prompt optimization.
| Framework | Primary focus | Typical strength |
|---|---|---|
| LlamaIndex | Data ingestion and retrieval | Rich index types and RAG tooling |
| LangChain | General LLM orchestration | Broad integrations and chain composition |
| LangGraph | Stateful agent graphs | Explicit control flow for agents |
| Haystack | Production search pipelines | Typed, inspectable pipeline components |
| DSPy | Programmatic prompt optimization | Compiling prompts against metrics |
These frameworks are not mutually exclusive. Teams often use LlamaIndex for retrieval and wrap its query engine as a tool inside an agent built with another framework. LlamaIndex also ships its own agent abstractions and Workflows, an event-driven orchestration layer in which steps react to typed events, which supports branching, loops and human review.
Configuring a LlamaIndex application in practice
A minimal LlamaIndex prototype needs only a few lines: load a folder, build a VectorStoreIndex and call as_query_engine(). Production quality, however, depends on deliberate configuration of chunking, embeddings, storage and evaluation. Global defaults for the language model and embedding model are set through the Settings object.
The decision table below summarizes the parameters that most often determine whether a deployment succeeds.
| Parameter | Typical starting point | Effect of changing it |
|---|---|---|
| Chunk size | About 512 to 1,024 tokens | Smaller chunks raise precision but lose context |
| Chunk overlap | About 10 to 20 percent | More overlap preserves continuity but adds storage |
| Similarity top-k | 2 to 5 nodes | Higher values add recall and prompt tokens |
| Vector store | In-memory for tests | External databases add persistence and scale |
| Reranker | Optional cross-encoder | Improves ordering at the cost of latency |
For persistence, LlamaIndex stores documents, indexes and vectors through a storage context, or delegates vectors to external databases through integration packages. Its evaluation modules score faithfulness and relevancy of answers, and observability integrations trace each retrieval and model call, which is essential for diagnosing poor answers.
Limitations and common mistakes
LlamaIndex reduces boilerplate, but it does not guarantee good retrieval. Most failures in LlamaIndex applications come from data preparation and configuration rather than from the framework itself.
- Poor parsing of complex documents: tables and multi-column PDFs extracted as plain text lose structure, so answers miss key figures.
- Default chunking everywhere: one chunk size rarely fits contracts, code and chat logs alike.
- Ignoring metadata: without source, date or section metadata, filtering and citation become unreliable.
- No evaluation set: tuning without a fixed list of test questions leads to changes that feel better but are not measurably better.
- Version drift: the API evolved quickly, and the move to separate integration packages broke many older import paths and tutorials.
Abstraction also has a cost. Deep layering can hide the exact prompts sent to the model, so experienced teams inspect or override prompt templates and keep latency and token budgets visible in tracing. For very simple use cases, a direct call to an embedding API and a vector database can be easier to maintain.
Explained without jargon
A language model knows a great deal about the world, but it has never read your company handbook, your contracts or your research notes. LlamaIndex is a toolkit that helps it read them. It cuts your documents into small pieces, files them in an organized way and, when someone asks a question, quickly pulls out the few pieces that matter.
Think of it as a very fast librarian for an AI assistant. The assistant still writes the answer, but it writes it from the pages the librarian hands over, which makes the answer more accurate and easier to check.
A typical use case
An internal tools team at a mid-sized manufacturing company wants engineers to query about 4,000 maintenance manuals and service bulletins in PDF form. Using LlamaIndex, the team builds an ingestion pipeline that parses the PDFs while keeping tables intact, splits them into chunks of about 800 tokens with 100 tokens of overlap, attaches machine model and revision date as metadata, and stores the vectors in a managed vector database.
At query time, a retriever fetches the top 10 candidates, a reranker keeps the best 4, and a metadata filter restricts results to the machine model the engineer selects. Answers show the source manual and page. After tuning against a test set of 150 real questions, the team reports that engineers find the correct procedure in seconds instead of searching manuals by hand, and wrong-model answers become rare because of the metadata filter.