Context engineering

On this page 7

Context engineering is the discipline of deciding what information enters a language model’s context window, in what form, in what order and at what token cost, so that the model can complete a task reliably. In artificial intelligence, it extends classic prompting from the wording of a single instruction to the whole input assembled at inference time: system instructions, retrieved documents, tool definitions, memory, conversation history, examples and intermediate results.

Prompting focuses on how a request is phrased; context engineering focuses on what the model knows when it answers. The distinction matters most in AI agents and retrieval systems, where the context is rebuilt automatically on every turn and errors come less from bad wording than from missing, stale, noisy or excessive information.

How context engineering shapes model behavior

Context engineering works because a language model has no persistent knowledge of the current task beyond its trained weights and the tokens in its window. Every fact, rule and constraint that should influence the answer must be present in the context at generation time, and anything absent is either guessed or ignored.

In practice, a context pipeline runs before every model call. It gathers candidate information from sources such as a vector index, a database, a tool result or a memory store, filters and ranks that material, compresses what is too long, and assembles the final sequence in a fixed layout. The model then attends to that sequence as one continuous input.

The following table lists the usual components of an engineered context and the role each one plays.

ComponentPurposeTypical source
System instructionsRole, rules, tone and output formatStatic template, versioned in code
Retrieved knowledgeFacts the model does not reliably knowSearch, vector index or database
Tool definitionsNames, parameters and usage rules of callable toolsAgent framework or API schema
Conversation statePrior turns and decisions already madeSession history, often summarized
Long-term memoryUser preferences and durable factsMemory store keyed to the user
ExamplesDemonstrations of the expected answerCurated or dynamically selected shots
Typical components of an assembled context

Context windows, token budgets and attention limits

Every model has a finite context window, commonly from about 8,000 to over 1,000,000 tokens depending on the model, and every token in it costs latency and money. Context engineering therefore treats the window as a budget to be allocated, not a container to be filled.

A larger window does not guarantee better use of its contents. Studies of long-context retrieval show a position effect often called lost in the middle: models tend to use information near the beginning and end of a long input more reliably than material buried in the center. Accuracy can also degrade as irrelevant tokens dilute attention, a pattern practitioners describe as context rot.

The table below summarizes the main budget levers and their trade-offs.

LeverEffectMain risk
Retrieve fewer chunksLower cost, sharper focusMissing a needed fact
Summarize historyKeeps long sessions within limitsLoss of exact wording or details
Place key facts at the edgesBetter recall of critical materialRigid layouts that are hard to maintain
Cache a stable prefixLower latency and cost per callStale content if the prefix changes rarely
Load tools on demandSmaller tool list, fewer wrong callsExtra round trip to discover tools
Budget levers and their trade-offs

Context engineering is a superset of prompt engineering rather than a replacement for it. Techniques such as a prompt template, dynamic few-shot prompting, prompt compression and custom instructions are all building blocks that a context pipeline combines and schedules.

The comparison below shows where the boundaries usually lie.

PracticeMain questionScope
Prompt engineeringHow should the instruction be wordedMostly the text of one request
Retrieval-augmented generationWhich external facts should be fetchedOne input channel of the context
Prompt compressionHow can the same meaning use fewer tokensA transformation step inside the pipeline
Context engineeringWhat should the model see on this callThe full assembled input across turns
How context engineering relates to neighboring practices

Designing a context pipeline in practice

A sound design starts from the task, not from the available data. Teams first define what a correct answer requires, then work backward to the minimum information that guarantees it, and only then add optional material if the budget allows.

  • Separate stable and volatile parts: keep instructions and tool schemas in a cached prefix and append changing data at the end.
  • Label every block with clear delimiters so the model can tell instructions from retrieved text and user input.
  • Isolate subtasks by giving sub-agents their own narrow contexts and returning only condensed results.
  • Write state outside the window, for example to a scratchpad file or notes store, and reload only what the next step needs.
  • Measure with evaluations, comparing answer quality and token cost as context variants change.

The decision table below matches common situations with a suitable strategy.

SituationRecommended strategyReason
Long multi-turn chatRolling summary plus recent turns verbatimPreserves decisions without exceeding the window
Agent with many toolsRetrieve relevant tool definitions per stepReduces wrong or hallucinated tool calls
Question over large documentsRanked retrieval with source labelsGrounds answers and limits noise
Repeated identical setupCached static prefixCuts latency and cost on every call
Choosing a context strategy

Failure modes and common mistakes

Most context failures are silent: the model still produces fluent text, so problems appear only as subtle errors. Experienced practitioners watch for four recurring patterns.

FailureSymptomFix
Context poisoningAn early error is repeated in later stepsValidate outputs before they re-enter the context
Context distractionModel copies history instead of reasoningSummarize or prune old turns
Context confusionIrrelevant tools or documents are usedFilter by relevance before assembly
Context clashContradictory facts produce unstable answersResolve conflicts or mark the authoritative source
Context failures, symptoms and fixes

A further risk is prompt injection through retrieved content. Any document, web page or tool result placed in the window can contain text that imitates instructions, so retrieved material should be clearly marked as data and never granted the authority of the system instructions.

The short version

A language model can only work with what is in front of it at the moment it answers. Context engineering is the job of preparing that desk: putting the right papers on it, in a sensible order, and removing everything that would only get in the way.

Good wording still helps, but the right information matters more. A model given the correct facts, a clear rulebook and a short, relevant history will usually outperform a model given a clever request and a pile of loosely related material.

Where it makes a difference

A customer support team runs an AI assistant that answers questions about insurance policies. The first version placed the full policy handbook of about 150,000 tokens into every request, along with the entire chat history. Answers were slow, expensive and occasionally cited clauses from the wrong policy type.

The team rebuilt the input as an engineered context: a cached block of instructions and tool definitions, the customer’s policy type and three retrieved clauses labeled by section number, a short summary of earlier turns and the last four messages verbatim. The typical request shrank to about 6,000 tokens.

In internal evaluations on a fixed set of real questions, the assistant produced fewer answers citing the wrong clause, responded noticeably faster and cost a fraction of the original version per conversation, without any change to the underlying model.