Chunking is the process of splitting documents into smaller, self-contained passages before they are embedded and indexed, so that a retrieval system can return only the parts relevant to a query. In artificial intelligence retrieval and knowledge systems, chunking is the first design decision of a RAG pipeline, and it shapes everything downstream: what embeddings represent, what semantic search can find and what context a language model finally reads.
A chunk is the unit of retrieval. If chunks are too large, they dilute the meaning captured by a single embedding and waste the model’s context window; if they are too small, they lose the surrounding information needed to answer a question. Good chunking keeps one coherent idea per chunk with enough context to stand on its own.
How chunking works in a retrieval pipeline
Chunking sits between document parsing and embedding. Raw files such as PDFs, web pages or tickets are first converted to clean text with structural markers, then split into chunks, then each chunk is turned into a vector by an embedding model and stored in an index together with its metadata.
- Parse the source and keep structure such as headings, lists, tables and page numbers.
- Split the text into chunks according to a chosen strategy and size limit.
- Enrich each chunk with metadata: document title, section path, date, access rights.
- Embed each chunk and store the vector with a pointer back to the original text.
- Retrieve the most similar chunks at query time and pass them to the model as context.
Because the embedding of a chunk is a single vector for the whole passage, every sentence in the chunk competes to define its meaning. This is why chunk boundaries have a direct effect on recall: a relevant fact buried in an unrelated passage may never surface in vector similarity search.
Chunking strategies
Chunking strategies differ in where they place boundaries. Simple methods cut by length, while more advanced methods follow the document’s structure or its meaning.
| Strategy | How boundaries are chosen | Best suited for |
|---|---|---|
| Fixed-size | Every N tokens or characters, regardless of content | Uniform text, quick baselines |
| Recursive splitting | Tries paragraphs, then sentences, then words until under the limit | General-purpose prose |
| Structure-aware | Follows headings, sections, list items or code blocks | Manuals, documentation, Markdown, HTML |
| Semantic | Splits where embedding similarity between sentences drops | Long narrative text with topic shifts |
| Sentence window | Indexes single sentences, returns surrounding sentences | Precise fact lookup |
| Hierarchical (parent-child) | Small chunks for matching, larger parent for context | Long technical documents |
Recursive splitting is the most widely used default because it respects natural text boundaries while guaranteeing a maximum size. Structure-aware chunking usually gives the best results on well-formatted documentation, because authors already grouped related information under headings.
Chunk size, overlap and other parameters
The two main parameters are chunk size and overlap. Size is usually measured in tokens of the embedding model, and overlap repeats part of the previous chunk at the start of the next one so that a sentence cut at a boundary still appears whole somewhere.
| Parameter | Common range | Effect of increasing it |
|---|---|---|
| Chunk size | 200-1,000 tokens, often 300-512 | More context per chunk, less precise embeddings |
| Overlap | 10-20% of chunk size | Fewer broken ideas, larger index and duplicates |
| Top-k retrieved chunks | 3-20 chunks | Higher recall, more noise and cost in the prompt |
| Embedding model limit | 512 to several thousand tokens | Sets a hard ceiling on chunk size |
These values are starting points, not rules. Question type matters most: short factual questions favor smaller chunks around 200-400 tokens, while questions that need explanation or reasoning favor chunks of 500-1,000 tokens. The reliable way to choose is to evaluate retrieval on real queries and compare hit rates for several settings.
Embedding models also have an effective range below their maximum input length. Text far beyond a few hundred tokens is often compressed into a vague average meaning, so a model that accepts 8,000 tokens does not make 8,000-token chunks a good choice.
Chunking compared with related techniques
Chunking is one of several levers that decide retrieval quality. It defines what can be found, while neighboring techniques decide how it is found and ranked.
| Technique | Question it answers | Relation to chunking |
|---|---|---|
| Chunking | What is the unit stored and retrieved | Sets the granularity of everything else |
| Embeddings | How is meaning encoded as a vector | Embed one vector per chunk |
| Semantic search | Which chunks match the query meaning | Searches across chunk vectors |
| Reranking | Which retrieved chunks are truly most relevant | Reorders chunks after retrieval |
| Contextual enrichment | How does a chunk keep document context | Adds titles or summaries to each chunk |
Contextual enrichment deserves special mention. Prepending a short description such as the document title and section path to each chunk before embedding helps chunks that are ambiguous on their own, for example a paragraph that says only that the limit was raised to 50 units.
Common chunking mistakes
Most chunking problems show up as answers that miss available information or answers built on fragments without context. They are usually cheaper to fix than switching embedding models.
- Splitting tables row by row or mid-row, which separates values from their column headers.
- Cutting code blocks in the middle, so neither half is meaningful.
- Ignoring document structure and mixing the end of one section with the start of the next.
- Dropping metadata, which prevents filtering by date, product version or access rights.
- Using one size for every source, although FAQs, contracts and API references need different settings.
- Never re-chunking after changing the embedding model or its token limit.
Tables and code deserve special handling: keep a table together when possible, or repeat the header row in every table chunk, and split code at function or class boundaries rather than at a token count.
The short version
Chunking means cutting long documents into bite-sized pieces before an AI system searches them. Imagine a library that keeps index cards for every page instead of for every book. When you ask a question, the librarian can hand you exactly the right page.
If the pieces are too big, the right answer hides among unrelated text. If they are too small, the answer loses the words around it that make it clear. Good chunking finds the size where each piece makes sense on its own.
Where it makes a difference
A software company builds an internal assistant over 4,000 pages of product documentation. The first version uses fixed 1,500-token chunks with no overlap. Engineers notice that questions about specific configuration flags often return a general overview page instead of the exact setting.
The team switches to structure-aware chunking at each heading, with a maximum of 400 tokens, a 60-token overlap and the page title and section path prepended to every chunk. Tables are kept whole, and code samples are split only between functions.
On a test set of 200 real support questions, the share of questions where the correct passage appears among the top five results rises noticeably, and the assistant’s answers start quoting the exact flag names and default values. The model itself did not change; better chunk boundaries made the right information findable.