LlamaIndex

Also known as: GPT Index

On this page 7

LlamaIndex is an open-source data framework for building applications in which a large language model answers questions over private or domain-specific data. Among AI frameworks & tools, it specializes in the data side of generative applications: it loads documents, splits them into nodes, indexes them and retrieves the most relevant context at query time, which makes it a standard toolkit for retrieval-augmented generation (RAG).

The project started in late 2022 under the name GPT Index and was renamed LlamaIndex in 2023. It is available as a Python library and as a TypeScript port, and it is distributed as a small core package plus many integration packages for model providers, vector databases and data loaders. Its scope has since grown from indexing to agents and event-driven workflows, but retrieval over documents remains its center of gravity.

How LlamaIndex turns documents into answers

LlamaIndex works as a two-stage pipeline: an ingestion stage that prepares data once, and a query stage that runs for every user request. During ingestion, readers load raw files into Document objects, node parsers split them into chunks called nodes, and an embedding model converts each node into a vector stored in an index or an external vector store.

At query time, a retriever selects the nodes most similar to the question, optional node postprocessors filter or rerank them, and a response synthesizer sends the selected context together with the question to the language model. The component that wires these steps together is the query engine; a chat engine adds conversation memory on top of it.

The following table lists the core building blocks and the role each one plays in the pipeline.

ComponentStageWhat it does
Reader (data loader)IngestionLoads PDFs, web pages, databases or APIs into documents
Node parserIngestionSplits documents into chunks with metadata and relationships
Ingestion pipelineIngestionChains transformations and caches their results
IndexIngestionOrganizes nodes for efficient lookup
RetrieverQueryReturns the top-k relevant nodes for a question
Node postprocessorQueryReranks, filters or trims retrieved nodes
Response synthesizerQueryBuilds prompts and generates the final answer
Core LlamaIndex components and their roles

Index types and retrieval strategies

An index in LlamaIndex is a data structure that determines how nodes are found, and the choice of index shapes both answer quality and cost. The VectorStoreIndex is the default for most projects, but other index types suit summarization, keyword lookup or relationship-heavy data.

Index typeRetrieval methodBest suited for
VectorStoreIndexEmbedding similarity, top-kQuestion answering over large corpora
SummaryIndexReads all nodes in sequenceSummaries of a small document set
TreeIndexHierarchical summaries, traversalLayered summarization of long texts
KeywordTableIndexExtracted keyword matchingExact terms, codes and identifiers
PropertyGraphIndexGraph traversal plus vectorsEntities and relationships across documents
Common index types and when each applies

Beyond the index itself, LlamaIndex offers advanced retrieval patterns. Hybrid search combines dense vectors with keyword scoring such as BM25, recursive retrieval follows links from summaries to detailed chunks, and the router query engine lets the model choose among several indexes. Sub-question decomposition splits a complex question into simpler ones and merges the partial answers.

Response synthesis also has modes. Compact packs as many nodes as fit into each prompt, refine iterates through chunks and improves the answer step by step, and tree_summarize builds the answer bottom-up. The trade-off is consistent: more model calls give more thorough answers but raise latency and cost.

LlamaIndex overlaps with several orchestration frameworks, and the practical difference lies in their center of gravity. LlamaIndex is data-first, with the deepest set of abstractions for parsing, chunking, indexing and retrieval, while its neighbors emphasize chains, agent graphs or prompt optimization.

FrameworkPrimary focusTypical strength
LlamaIndexData ingestion and retrievalRich index types and RAG tooling
LangChainGeneral LLM orchestrationBroad integrations and chain composition
LangGraphStateful agent graphsExplicit control flow for agents
HaystackProduction search pipelinesTyped, inspectable pipeline components
DSPyProgrammatic prompt optimizationCompiling prompts against metrics
LlamaIndex and neighboring frameworks

These frameworks are not mutually exclusive. Teams often use LlamaIndex for retrieval and wrap its query engine as a tool inside an agent built with another framework. LlamaIndex also ships its own agent abstractions and Workflows, an event-driven orchestration layer in which steps react to typed events, which supports branching, loops and human review.

Configuring a LlamaIndex application in practice

A minimal LlamaIndex prototype needs only a few lines: load a folder, build a VectorStoreIndex and call as_query_engine(). Production quality, however, depends on deliberate configuration of chunking, embeddings, storage and evaluation. Global defaults for the language model and embedding model are set through the Settings object.

The decision table below summarizes the parameters that most often determine whether a deployment succeeds.

ParameterTypical starting pointEffect of changing it
Chunk sizeAbout 512 to 1,024 tokensSmaller chunks raise precision but lose context
Chunk overlapAbout 10 to 20 percentMore overlap preserves continuity but adds storage
Similarity top-k2 to 5 nodesHigher values add recall and prompt tokens
Vector storeIn-memory for testsExternal databases add persistence and scale
RerankerOptional cross-encoderImproves ordering at the cost of latency
Key configuration choices

For persistence, LlamaIndex stores documents, indexes and vectors through a storage context, or delegates vectors to external databases through integration packages. Its evaluation modules score faithfulness and relevancy of answers, and observability integrations trace each retrieval and model call, which is essential for diagnosing poor answers.

Limitations and common mistakes

LlamaIndex reduces boilerplate, but it does not guarantee good retrieval. Most failures in LlamaIndex applications come from data preparation and configuration rather than from the framework itself.

  • Poor parsing of complex documents: tables and multi-column PDFs extracted as plain text lose structure, so answers miss key figures.
  • Default chunking everywhere: one chunk size rarely fits contracts, code and chat logs alike.
  • Ignoring metadata: without source, date or section metadata, filtering and citation become unreliable.
  • No evaluation set: tuning without a fixed list of test questions leads to changes that feel better but are not measurably better.
  • Version drift: the API evolved quickly, and the move to separate integration packages broke many older import paths and tutorials.

Abstraction also has a cost. Deep layering can hide the exact prompts sent to the model, so experienced teams inspect or override prompt templates and keep latency and token budgets visible in tracing. For very simple use cases, a direct call to an embedding API and a vector database can be easier to maintain.

Explained without jargon

A language model knows a great deal about the world, but it has never read your company handbook, your contracts or your research notes. LlamaIndex is a toolkit that helps it read them. It cuts your documents into small pieces, files them in an organized way and, when someone asks a question, quickly pulls out the few pieces that matter.

Think of it as a very fast librarian for an AI assistant. The assistant still writes the answer, but it writes it from the pages the librarian hands over, which makes the answer more accurate and easier to check.

A typical use case

An internal tools team at a mid-sized manufacturing company wants engineers to query about 4,000 maintenance manuals and service bulletins in PDF form. Using LlamaIndex, the team builds an ingestion pipeline that parses the PDFs while keeping tables intact, splits them into chunks of about 800 tokens with 100 tokens of overlap, attaches machine model and revision date as metadata, and stores the vectors in a managed vector database.

At query time, a retriever fetches the top 10 candidates, a reranker keeps the best 4, and a metadata filter restricts results to the machine model the engineer selects. Answers show the source manual and page. After tuning against a test set of 150 real questions, the team reports that engineers find the correct procedure in seconds instead of searching manuals by hand, and wrong-model answers become rare because of the metadata filter.