Late chunking

On this page 7

Late chunking is a document embedding technique in which a long-context embedding model first encodes the entire document as one sequence, and only afterward are the resulting token embeddings split into chunks and pooled into one vector per chunk. Because every token has already attended to the rest of the text, each chunk vector carries document-level context that conventional chunking discards.

Artificial intelligence systems that answer questions over private documents, especially retrieval-augmented generation (RAG) pipelines in the Retrieval & knowledge domain, use late chunking to fix a familiar weakness: a chunk that says “the company raised its forecast” is hard to retrieve when the company name appeared three paragraphs earlier. Late chunking keeps that missing reference inside the vector without changing the chunk text or adding a language model call.

How late chunking builds context-aware chunk vectors

Late chunking reverses the usual order of split first, embed second. In a standard pipeline, each chunk is sent to the embedding model separately, so the model never sees the surrounding text. In late chunking, the model reads the whole document once, and chunking happens at the pooling stage instead of at the input stage.

The process relies on the internal structure of transformer embedding models. A transformer produces one contextual vector per token, and most text embedding models then apply mean pooling to average those token vectors into a single sentence or document vector. Late chunking simply applies that averaging separately within each chunk span rather than across the whole sequence.

The following table lists the steps of a typical late chunking implementation and what each step contributes.

StepOperationPurpose
TokenizationConvert the full document into tokensPrepare one long input sequence
Full-sequence encodingRun the transformer over all tokens at onceLet every token attend to the whole text
Boundary mappingMap chunk character offsets to token spansDecide which tokens belong to which chunk
Span poolingMean-pool token embeddings inside each spanProduce one context-aware vector per chunk
IndexingStore chunk vectors with chunk text and metadataEnable vector search over chunks
Steps of a late chunking pipeline

Chunk boundaries can come from any strategy, such as fixed token counts, sentence boundaries or semantic chunking. The boundary strategy and the embedding step are decoupled, which makes late chunking easy to add to an existing ingestion pipeline.

Requirements and parameters that govern results

Late chunking works only with models and settings that expose token-level embeddings and accept long inputs. The quality gain depends on several parameters, summarized in the table below with typical values.

ParameterTypical rangeEffect on results
Model context window8,192 tokens or moreLimits how much text shares context
Pooling methodMean poolingCLS-only models need adaptation or fine-tuning
Chunk size128 to 512 tokensSmaller chunks gain more from added context
Macro chunk overlapSeveral hundred tokensPreserves context across window splits
Document lengthUp to the context windowLonger texts require macro chunking
Key parameters for late chunking

For documents longer than the context window, implementations use long late chunking: the text is divided into overlapping macro chunks that fit the model, each macro chunk is encoded, and the overlapping token embeddings are used to keep continuity across macro boundaries. Context beyond one macro chunk is still lost, so the benefit shrinks for very long documents.

The pooling method matters because late chunking assumes that averaging token vectors produces a meaningful chunk vector. Models trained with mean pooling show this property naturally; models that rely on a CLS token may produce weaker chunk vectors without additional training.

Late chunking is one of several answers to the context loss problem in chunked retrieval. It differs from neighboring techniques in where context enters the vector and what it costs, as the table shows.

ApproachHow context is addedMain cost
Naive chunkingNo added contextLowest cost, weakest on references
Late chunkingFull-document attention before poolingLong-context encoding, no extra LLM calls
Contextual retrievalLLM writes a context note per chunkOne generation call per chunk
Parent document retrievalSmall chunks retrieve larger parentsLarger prompts at answer time
Late interaction (ColBERT)Token vectors kept and matched at query timeMuch larger index and scoring cost
Late chunking and alternative approaches

The name invites confusion with late interaction, but the two are different. ColBERT stores many vectors per passage and compares query tokens with document tokens at search time, while late chunking still stores a single vector per chunk and uses ordinary vector similarity, such as cosine similarity, in a standard vector index.

Late chunking and contextual retrieval can also be combined. Contextual retrieval adds explicit text that a reranker and BM25 can read, whereas late chunking improves only the dense vector, so hybrid search systems sometimes use both.

When to use late chunking in a retrieval pipeline

Late chunking pays off most when chunks contain pronouns, abbreviations or implicit references that only make sense with earlier text. Self-contained entries such as product records or FAQ pairs gain little. The decision table below summarizes common situations.

SituationRecommendationReason
Reports, contracts, technical manualsUse late chunkingMany cross-references between sections
Short, independent recordsStandard chunking is enoughLittle context to recover
Embedding API returns pooled vectors onlyUse contextual retrieval insteadToken embeddings are not accessible
Documents far beyond the context windowLong late chunking with overlapContext is kept within each macro chunk
Tight ingestion budgetLate chunking before LLM-based contextNo generation cost per chunk
Decision guide for late chunking

Teams should measure the gain on their own queries with standard RAG evaluation metrics such as recall at k or mean reciprocal rank. Published benchmark gains are typically a few percentage points of retrieval quality, larger for long, reference-heavy documents and close to zero for short ones.

Limitations and common mistakes

Late chunking has practical constraints that are easy to overlook during a first implementation. The most frequent problems are listed below.

  • Misaligned token spans: computing chunk boundaries on characters but slicing token embeddings with the wrong offsets silently corrupts every chunk vector.
  • Exceeding the context window: inputs that are truncated without macro chunking lose the end of the document entirely.
  • Assuming universal model support: hosted embedding services that return only pooled vectors cannot perform late chunking.
  • Context bleed: a chunk vector may absorb topics from neighboring sections, which can slightly lower precision for very specific queries.
  • Higher encoding memory: attention over long sequences needs considerably more GPU memory than encoding short chunks one by one.

Because the chunk text is unchanged, the language model that generates the answer still sees only the raw chunk. Late chunking improves which chunks are found, not how much context the generator receives, so pairing it with grounding checks or parent document retrieval remains useful.

The idea in brief

Imagine cutting a long report into short slips of paper before anyone has read it. A slip that says “it rose by ten percent” means nothing on its own, because nobody knows what “it” is. Normal chunking works like that: each slip is summarized without knowing the rest of the report.

Late chunking reads the whole report first and only then cuts it into slips. Each slip keeps a memory of what came before and after, so a search for the right topic can find it. The key takeaway is simple: understand first, cut second.

A practical scenario

A legal technology team indexes about 20,000 commercial contracts for an internal question-answering assistant. Each contract is split into chunks of roughly 256 tokens. Many clauses refer to “the Supplier” or “the Agreement” without naming the parties, so retrieval often missed the right clause for queries such as”termination notice for the logistics contract”.

The team switches to late chunking with an open embedding model that accepts 8,192 tokens and uses mean pooling. Contracts shorter than the window are encoded in one pass; longer ones are split into macro chunks with a 512-token overlap. The chunk text, chunk size and vector index stay the same.

On a test set of 300 real questions, recall at 10 improves noticeably, mostly for clauses that mention parties only by role. Ingestion takes longer on the GPU, but there are no per-chunk language model calls, so the added cost stays modest.