Semantic chunking

On this page 7

Semantic chunking is a document segmentation method that places chunk boundaries where the meaning of the text changes, rather than after a fixed number of characters or tokens. In artificial intelligence, and especially in the retrieval and knowledge layer of retrieval-augmented generation (RAG), it aims to produce chunks that each express one coherent topic, which makes their embeddings sharper and the retrieved context more useful to a language model.

The method sits between ingestion and indexing. A document is split into small units such as sentences, each unit is embedded, and similarity between neighboring units decides whether they belong together. The result is a set of variable-length chunks that follow the natural structure of the argument instead of cutting through it.

How semantic chunking finds topic boundaries

Semantic chunking finds boundaries by measuring how much consecutive passages differ in meaning and cutting where that difference spikes. The most common implementation follows a short pipeline that any embedding model can support.

  1. Split the document into base units, usually sentences, using a sentence tokenizer.
  2. Optionally group each sentence with one or two neighbors into a sliding window to smooth noise from very short sentences.
  3. Embed every window with the same embedding model used later for retrieval.
  4. Compute the cosine distance between each pair of adjacent windows.
  5. Mark a breakpoint wherever the distance exceeds a threshold, then merge the units between breakpoints into chunks.
  6. Enforce minimum and maximum chunk sizes so that no chunk is a single fragment or exceeds the embedding model input limit.

The threshold is the decisive parameter. Fixed absolute values rarely transfer between corpora, so most systems use a relative threshold derived from the distribution of distances within each document. The table below summarizes the usual options.

StrategyHow the cut is decidedTypical settingEffect
PercentileCut at distances above a chosen percentile90th to 95th percentilePredictable number of chunks per document
Standard deviationCut where distance exceeds mean plus k deviationsk between 1 and 3Adapts to how varied the document is
Interquartile rangeCut at outliers beyond the upper quartile range1.5 times the IQRRobust to a few extreme jumps
GradientCut where the change in distance is steepestPercentile on the gradientHelps with dense, uniform technical text
Common breakpoint threshold strategies

Variants of semantic chunking

Several variants share the goal of meaning-aware boundaries but differ in what signal they use. Choosing among them is mostly a question of cost, latency and document type.

VariantBoundary signalRelative costBest fit
Embedding breakpointDistance between adjacent sentence embeddingsLow to moderateGeneral prose, reports, articles
Clustering-basedGroups of sentences with similar embeddingsModerateDocuments that revisit topics out of order
LLM-guidedA language model proposes topic shiftsHighSmall, high-value collections
Structure-aware hybridHeadings first, then embedding distance inside sectionsLow to moderateManuals, policies, documentation
Main variants and their signals

The structure-aware hybrid is often the most reliable in production because explicit headings are a stronger signal than any similarity score. Embedding distance is then used only to split sections that are too long, which keeps chunk sizes within budget without ignoring the author’s own structure.

Semantic chunking compared with other chunking methods

Semantic chunking is one option within the broader practice of chunking, and it competes with simpler and more recent techniques. Its main advantage is topical coherence; its main cost is an extra embedding pass during ingestion.

MethodHow chunks are formedStrengthWeakness
Fixed-size chunkingEqual token counts with optional overlapFast, simple, predictableCuts through sentences and ideas
Recursive character splittingParagraphs, then sentences, then charactersRespects formatting cheaplyIgnores meaning within long paragraphs
Semantic chunkingBreakpoints at shifts in embedding similarityCoherent, topic-focused chunksExtra compute, sensitive to threshold
Late chunkingEmbed the whole document, then pool token vectors per chunkChunks keep document-wide contextNeeds long-context embedding models
Parent document retrievalMatch small chunks, return larger parentsPrecise matching, rich contextMore storage and index logic
Semantic chunking and related approaches

These methods are not mutually exclusive. Teams frequently combine semantic boundaries with contextual retrieval, which prepends a short document summary to each chunk, or with parent document retrieval, so that precise matching and sufficient context for generation come from different layers.

Configuring semantic chunking in a RAG pipeline

Good configuration starts with the retrieval goal, not with the splitter. A chunk should be large enough to answer a question on its own and small enough that its embedding is not diluted by unrelated material. In practice, many pipelines target roughly 200 to 800 tokens per chunk, with hard limits below the embedding model context window.

SituationRecommended settingReason
Short, focused FAQ entriesSkip semantic splitting entirelyEach entry is already one topic
Long narrative reportsPercentile threshold near 95Fewer, broader chunks preserve argument flow
Dense technical manualsHeadings first, then lower thresholdMany small topic shifts within sections
Transcripts and chat logsLarger sliding window of 3 to 5 sentencesSmooths noise from very short utterances
Multilingual corporaMultilingual embedding model for splittingDistances stay comparable across languages
Configuration decisions by situation

The settings should be validated with retrieval metrics such as recall at k and mean reciprocal rank on a labeled question set, and with RAG evaluation of final answers. Visual inspection of a sample of chunks remains a cheap and effective sanity check.

Limitations and common mistakes

Semantic chunking is not a guaranteed improvement. Published comparisons show mixed results: on some corpora it raises retrieval quality, while on others well-tuned fixed-size chunking performs about as well at a fraction of the ingestion cost. The benefit depends heavily on document style and on the embedding model.

  • Threshold drift: a percentile tuned on one document type can create hundreds of tiny chunks on another.
  • Lost references: a chunk that begins with a pronoun or an abbreviation may be unretrievable without context from the previous chunk.
  • Model mismatch: splitting with one embedding model and indexing with another weakens the logic behind the boundaries.
  • Ingestion cost: every sentence is embedded at least once more, which matters for corpora of millions of pages.
  • Ignoring tables and code: sentence tokenizers break structured content, so tables and code blocks need dedicated handling.

Understanding it simply

Imagine cutting a long book into index cards so that someone can later pull out only the cards that answer a question. One way is to cut every page in half, no matter where a thought ends. Semantic chunking instead cuts wherever the subject changes, so each card holds one complete idea.

The computer judges a change of subject by comparing how similar each sentence is to the next one. When two neighboring sentences suddenly look very different in meaning, it draws a line there. The payoff is cleaner cards, which helps an AI assistant find the right passage and answer with fewer mistakes.

Applied in practice

An internal support team at a mid-sized software company builds a RAG assistant over about 4,000 pages of product documentation. With fixed 512-token chunks, the assistant often returns passages that mix installation steps with unrelated licensing notes, because one chunk spans the end of one section and the start of the next.

The team switches to a structure-aware hybrid: it splits first on headings, then applies embedding breakpoints at the 92nd percentile inside sections longer than 800 tokens, with a minimum chunk size of 120 tokens. On a test set of 300 real support questions, recall at 5 rises noticeably and reviewers mark fewer answers as mixing unrelated procedures. Ingestion time roughly doubles, which the team accepts because the documentation is re-indexed only once per release.