Prompt compression

On this page 7

Prompt compression is the family of techniques that reduce the number of tokens in a prompt while preserving the information a language model needs to produce the same quality of answer. It works by dropping low-information tokens, summarizing long passages or encoding context into compact learned vectors, and it targets the cost, latency and context-window limits of long inputs.

Prompting techniques usually add text: instructions, examples, retrieved documents and conversation history. Prompt compression runs in the opposite direction and becomes valuable in retrieval-augmented generation, long-running agents and many-shot prompting, where inputs of tens of thousands of tokens are common and most of them contribute little to the final answer.

Why compressing prompts pays off

Compressing prompts pays off because input tokens drive cost and latency in almost every hosted model. Providers bill per token, the attention computation grows with input length, and time to first token rises as the prefill phase processes more text. Shorter prompts therefore reduce spending and make responses feel faster.

Quality is a second motive. Long contexts suffer from the lost-in-the-middle effect, where models attend weakly to information buried in the center of a long input. Removing redundant passages can raise accuracy, not only preserve it, because the relevant evidence becomes easier for the model to locate.

The table below lists the main benefits and the conditions under which each one matters most.

BenefitMechanismMatters most when
Lower costFewer billed input tokensHigh request volume
Lower latencyShorter prefill computationInteractive chat or voice
Fits context limitsMore content per windowLong documents or histories
Better focusLess distracting textRetrieval returns noisy passages
Benefits of prompt compression

How prompt compression methods work

Prompt compression methods work in one of three ways: they delete tokens, they rewrite text more briefly, or they replace text with learned embeddings. The first two produce ordinary text that any model can read, while the third requires a model trained or adapted to accept the compressed representation.

Token-pruning methods score each token or phrase by how much information it carries. Selective Context uses self-information computed by a small language model, and the LLMLingua family ranks tokens by perplexity under a compact model, removing those the target model could predict anyway. LLMLingua-2 reframes the task as token classification, training an encoder to decide which tokens to keep, which makes it faster and less dependent on a causal model.

Abstractive methods use a model to summarize or rewrite context into a shorter form, often conditioned on the user question. Soft compression methods such as gist tokens, AutoCompressors and the in-context autoencoder train a model to condense a long input into a handful of memory vectors that the generator reads in place of the original text.

FamilyOutputTypical ratioMain constraint
Token pruningShortened text2x to 10xText may become ungrammatical
Extractive selectionChosen sentences or chunks2x to 5xNeeds a relevance signal
Abstractive summarizationRewritten text3x to 20xExtra model call, risk of distortion
Soft compressionLearned vectors10x to 100xRequires model access and training
Main families of compression methods

Query-aware and query-agnostic compression

Compression is query-aware when the user question guides what is kept and query-agnostic when the context is compressed once without knowing the question. LongLLMLingua and many retrieval compressors are query-aware, which preserves relevant facts more reliably at high compression ratios.

Query-agnostic compression has a practical advantage: the compressed context can be computed once and reused across many requests, for example a compressed product manual shared by every support query. The choice depends on how often the context changes compared with the question.

SituationRecommended approachReason
RAG with many retrieved chunksQuery-aware pruning or rerankingKeeps evidence for the question
Static system prompt reused oftenManual rewrite plus prompt cachingCaching is cheaper than recompressing
Long agent conversationRolling summarization of old turnsPreserves decisions, drops chatter
Self-hosted model at scaleSoft compression or KV cache methodsAccess to internals allows deeper gains
Choosing a compression strategy

Prompt compression is often confused with neighboring optimizations that also cut cost or length but operate at a different layer of the system. Distinguishing them helps teams combine them rather than duplicate effort.

  • Prompt caching reuses the computed state of an unchanged prefix; it reduces cost without shortening the text, and it combines well with compression.
  • KV cache compression shrinks the internal attention memory during inference and requires control over the serving stack.
  • Chain-of-density prompting asks a model to write denser summaries as an output task, not to shrink its own input.
  • Context engineering decides what enters the context at all, while compression reduces what has already been selected.

In practice the strongest pipelines select first, then compress: retrieval and reranking choose relevant material, and a compressor removes redundancy within it.

Risks and evaluation of compressed prompts

The central risk is silent information loss. A pruned prompt can drop a negation, a number, a unit or a name, and the model then answers confidently from incomplete evidence. Aggressive ratios also damage structured content such as code, JSON, tables and legal wording, where every token can matter.

Compression should therefore be evaluated like any other model change: run a representative test set at several ratios and compare accuracy, faithfulness and cost against the uncompressed baseline. The compressor itself adds latency and expense, so the net saving must be measured end to end.

SymptomLikely causeFix
Wrong numbers in answersDigits pruned as low-informationProtect numerals and entities
Broken code or JSONSyntax tokens removedExclude structured blocks from pruning
Ignored instructionsSystem prompt compressedCompress context only, not instructions
No real speedupCompressor slower than savingsUse a smaller or cached compressor
Failure symptoms and fixes

The idea in brief

Prompt compression is like packing a suitcase carefully. Instead of throwing everything in, it removes items that are not needed for the trip, so the bag is lighter but still holds what matters.

For an AI system, the suitcase is the text sent with each question. Shorter text costs less and gets answered faster. The key takeaway is that good compression keeps the facts the answer depends on and removes only repetition and filler.

A practical scenario

An insurance company runs a support assistant that retrieves ten policy passages of about 600 tokens each for every customer question, producing prompts of roughly 7,000 tokens. Monthly input costs are high, and answers take several seconds to start.

The engineering team adds a query-aware token-pruning step after reranking, targeting a 4x ratio, and excludes the system instructions and any tables of coverage limits from pruning. On a test set of 500 real questions, they compare the compressed and uncompressed pipelines.

Average prompt length falls to about 2,000 tokens, input cost drops by roughly 70 percent after accounting for the compressor, and time to first token improves noticeably. Answer accuracy stays within one percentage point of the baseline, so the team ships the change with continued monitoring.