Context Engineering for Enterprise AI: Why Prompts Are Not Enough

For two years, the enterprise AI playbook had a simple first move: write a better prompt. Teams hired prompt engineers, traded prompt templates, and treated the phrasing of an instruction as the lever that separated a good AI feature from a bad one. That era is closing. As enterprise systems moved from single-shot chat to multi-step agents that retrieve documents, call tools, and run over long time horizons, practitioners discovered that the prompt is a shrinking fraction of what actually determines output quality. The larger, harder problem is everything around the prompt — the documents, tool results, conversation history, memory, and instructions that fill a model's limited attention window at each step. That discipline now has a name: context engineering.

This guide explains what context engineering is, how it differs from prompt engineering, why prompts alone are structurally insufficient for enterprise AI, and what a rigorous approach to enterprise AI context looks like. It is written for teams in the consideration stage — evaluating how to build reliable AI on top of frontier models — and every claim is grounded in the research and industry sources that defined the field in 2025 and 2026.

What context engineering actually is

The term crystallized in mid-2025. On June 18, 2025, Shopify CEO Tobi Lütke posted that he preferred "context engineering" to "prompt engineering," defining it as "the art of providing all the context for the task to be plausibly solvable by the LLM" [1][2]. A week later, Andrej Karpathy — former head of AI at Tesla and an OpenAI founding member — amplified the framing, calling context engineering "the delicate art and science of filling the context window with just the right information for the next step," and noting that in every industrial-strength LLM application this, not prompt-writing, is the core skill [1][2][3]. Simon Willison extended the idea, arguing that context engineering means deliberately assembling system prompts, retrieved documents, conversation history, and tool outputs into one coherent information package [4]. The label was not entirely new — Walden Yan at Cognition (the team behind the Devin coding agent) had been writing about it earlier in the year, and its academic roots reach back to Anind K. Dey's 2001 work on context in ubiquitous computing — but June 2025 is when it went mainstream [2][5].

In September 2025, Anthropic formalized the concept. In its engineering post "Effective context engineering for AI agents," published September 29, 2025, Anthropic defined it as "the set of strategies for curating and maintaining the optimal set of tokens" a model processes during inference — including all the information that lands in the window beyond the prompt itself [3][6]. The company's framing is blunt: as we move from prompts optimized for one-shot tasks toward agents operating over many turns, we need strategies for managing the entire context state — system instructions, tools, external data, the Model Context Protocol, and message history — because an agent running in a loop continuously generates new information that must be refined rather than accumulated [6].

The key conceptual shift is this: prompt engineering is now a subset of context engineering [1][3]. Writing a clear instruction still matters, but in a production agent it is a small share of the total context. The rest is conversation history, retrieved knowledge, tool outputs, agent state, and dynamically assembled data [1]. Karpathy's mental model captures it well: if the LLM is a CPU and its context window is RAM, then context engineering is the operating system that decides what to load into working memory at each step, when to evict stale data, and how to organize it so the processor can execute efficiently [16].

Prompt engineering vs context engineering: the distinction that matters

The clearest way to frame prompt engineering vs context engineering is by the question each asks. Prompt engineering asks: how do I phrase my request? Context engineering asks: what should the model know — in what structure, from which sources, at what moment — before it generates anything? [16] Drew Breunig put the boundary crisply: prompts are instructions, while context is everything the model needs to act on those instructions [12].

That difference is not cosmetic; it reflects what changed in how AI systems are built:

- Prompt engineering treats the input as mostly static and human-authored: a system prompt plus a user message, tuned by hand. It works well for single-turn classification, extraction, or generation.

- Context engineering treats the input as dynamic and machine-assembled: at each step, a system selects, compresses, and orders information drawn from documents, prior turns, tool calls, and memory. It is the discipline required once you have agents, retrieval, and multi-step workflows — which is to say, once you have enterprise AI.

The industry has been unusually direct about the stakes. Cognition described context engineering in mid-2025 as "effectively the #1 job of engineers building AI agents" [7]. Anthropic's guidance makes an even sharper point for anyone deciding where to spend effort: a weaker model given excellent context will often outperform a state-of-the-art model given poor context — but no frontier model can compensate for a badly constructed context [3][6]. In other words, the model is increasingly a commodity input; the context is the differentiator.

Why prompts are not enough for enterprise AI

Consumer chat hides the problem. When a person types a question into a chatbot, the context is small, self-contained, and supplied by the user. Enterprise AI is the opposite. The information a model needs to answer a question about a customer, a contract, a codebase, or a compliance rule does not live in the prompt — it lives across databases, document stores, ticketing systems, and APIs, and it changes constantly. Getting that information into the window correctly, freshly, and safely is the entire job. Three structural realities make prompts alone insufficient.

First, enterprise context is dynamic and multi-source. The relevant facts are scattered, versioned, and time-sensitive. A price retrieved 48 hours ago may have changed; a customer segment from last quarter may be stale; an API schema cached at startup may be deprecated [15]. A static prompt cannot represent a moving world. Retrieval, memory, and tool calls can — but only if a system governs what gets pulled in and when.

Second, the context window is a finite and costly resource, not free space. The arrival of million-token windows created a tempting illusion: just put everything in the prompt [40][42]. But context accumulates fast and expensively in agentic systems. Tool results alone can consume tens of thousands of tokens before an agent has even processed the user's request, and multi-agent systems can use many times more tokens than a simple chat exchange [20]. Unchecked accumulation drives both cost and — as the next section shows — quality degradation.

Third, more context does not mean better answers. This is the finding that reframed the field, and it is worth grounding carefully.

The evidence: context rot and "lost in the middle"

Two bodies of research show that long context is not a free lunch, and that "stuff everything in" is an anti-pattern.

The foundational result is Lost in the Middle by Liu et al., published in the Transactions of the Association for Computational Linguistics (2024). Testing question-answering and retrieval across model families, the authors found a U-shaped performance curve: accuracy is highest when the relevant information sits at the very beginning or end of the context and drops sharply — by more than 30 percentage points in their experiments — when the decisive fact is buried in the middle. The effect replicated across GPT-3.5, GPT-4, Claude, and open models such as MPT-30B and LongChat-13B, and stems partly from architectural properties of attention over long sequences [26][31][34].

The 2025 escalation came from Chroma. In its July 2025 report "Context Rot: How Increasing Input Tokens Impacts LLM Performance," researchers Kelly Hong, Anton Troynikov, and Jeff Huber tested 18 leading models — including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3 — and found that performance degrades as input length grows even on simple tasks, and that the degradation is uneven and often surprising across models [27][28][33][37]. Anthropic's own guidance names the phenomenon directly: despite ever-larger windows, models exhibit "context rot" — as token count rises, the ability to accurately recall information falls — because attention is a finite budget spread across every token, and each token attending to every other creates a quadratic strain on that budget [18][19]. A counterintuitive Chroma finding underlines that this is about information management, not raw size: topically related distractors and even the logical flow of the context can hurt performance, which inverts the tidy-everything-in instinct many retrieval teams start with [28][30].

The practical lesson for enterprise teams evaluating long-context strategies: advertised context length is not usable context length. Analyses cross-checking benchmarks like RULER suggest effective context is often well below the advertised window, and practitioners building agents report frontier models rolling over well before their limits — so the right posture is to measure degradation against your own workload rather than trust a spec sheet [30]. Bigger windows raise the ceiling; they do not remove the discipline.

The four ways context fails

Beyond gradual rot, context breaks in specific, diagnosable ways. Drew Breunig's two-part taxonomy — "How Long Contexts Fail" and "How to Fix Your Context" (June 2025), quickly adopted across the field — names four failure modes that turn a vague "the agent is acting weird" into an actual diagnosis [42][44][37][38]:

1. Context poisoning: a hallucination or error enters the context and is then referenced repeatedly, so the model reasons confidently on top of a false premise. Google DeepMind flagged exactly this in the Gemini 2.5 technical report for game-playing agents — one bad "fact" enters, and every later step inherits the corruption [42][41].

2. Context distraction: the context grows so large that the model over-focuses on its accumulated history and neglects what it learned in training, looping or repeating past actions instead of reasoning fresh [42][44].

3. Context confusion: superfluous or irrelevant content in the window gets used anyway — models feel compelled to draw on what is present — producing low-quality, noisy responses [42][44].

4. Context clash: newly accrued information or tool outputs conflict with earlier instructions or established facts, degrading reasoning. This is especially likely when connecting third-party tools whose descriptions clash with the rest of the prompt [42][44][40].

These are architectural realities, not model defects, and they are preventable through design [20]. Naming them matters for enterprise teams because it converts reliability from a mystery into an engineering checklist.

What enterprise AI context is actually made of

If prompts are not enough, what fills a well-engineered context? Anthropic's guidance describes a deliberate arrangement — system instructions first, then relevant memory, tool definitions, and conversation history — where the ordering and quality of each element determine performance more than any single factor [22]. In practice, enterprise context is assembled from typed, distinct sources, each with its own lifecycle [14]:

- Instructions: the system prompt defining role, capabilities, constraints, and output format. Anthropic advises keeping these focused and avoiding contradictory rules, and — importantly — curating a small set of diverse, canonical few-shot examples rather than stuffing every edge case into the prompt [6].

- Retrieved knowledge: documents and records pulled via retrieval-augmented generation (RAG), selected for relevance rather than dumped wholesale.

- Tool definitions and results: the outputs of API and tool calls, which must be pruned; Anthropic recommends curating a minimal viable set of tools, because if a human engineer cannot say which tool applies in a situation, an agent will not do better [6].

- Memory: durable state carried across turns and sessions — governed, curated recall, not an ever-growing transcript.

- Conversation history: prior turns, compacted so the signal survives without recreating the original bloat.

The engineering discipline is deciding, for each of these, what enters the window, what gets compressed, what is retrieved on demand, and what is dropped entirely [23].

The strategies: how context engineering is done

The field has converged on a compact set of techniques. LangChain's Lance Martin organized them into four verbs that are now widely used [17][8]:

- Write — author clear instructions and persist information outside the window (notes, scratchpads, external files) so it can be recalled without bloating context.

- Select — retrieve only the relevant subset of documents, memory, and tools for the current step.

- Compress — reduce token waste through summarization and compaction of history.

- Isolate — keep unrelated context in separate threads or sub-agents so one task's clutter does not contaminate another.

Breunig's follow-up adds concrete tactics that map onto these: tool loadout (enabling only the subset of tools relevant to a task phase, based on evidence that beyond roughly 20 tools some models get confused), context quarantine (isolating sub-tasks into dedicated threads so an early error cannot poison the whole project), and agentic RAG (letting the agent decide what to retrieve in a reasoning loop rather than relying on a fixed pipeline) [38][44]. Anthropic describes complementary patterns for long-horizon work — context compaction, structured note-taking, and multi-agent architectures where sub-agents operate with clean windows and return distilled results — as the way to maintain coherence at scale [20][6]. A crucial governance note from Breunig: avoid black-box memory; agents need transparent, auditable, and steerable memory to be safe in production [36].

The enterprise layer: governance, freshness, and security

For enterprise AI, context engineering carries obligations that consumer demos never face. Assembling context from internal systems means the context pipeline becomes part of your attack surface and your compliance boundary.

Provenance and freshness. In a multi-agent or retrieval system, facts arrive from many sources that can disagree. Enterprise context needs to track where each piece came from and whether it is still valid — temporal validity and provenance become first-class concerns, because agents otherwise make decisions on stale or unattributed data with no way to detect the error [15].

Security. The Model Context Protocol (MCP), which Anthropic introduced in November 2024 as a standard for connecting models to tools and data, has seen explosive enterprise adoption — thousands of servers listed by 2025, with OpenAI, Google, Meta, and Microsoft participating and Microsoft integrating MCP across Copilot Studio and Azure AI Foundry [46][54][48]. That reach expands the risk surface. OWASP ranks prompt injection as the number-one LLM security risk, and MCP environments amplify its impact: a successful injection can trigger real actions through connected tools, not merely generate malicious text [46][48]. Because untrusted content — a document, a tool result, a web page — enters the same context window as trusted instructions, indirect prompt injection is a structural hazard of context assembly, and traditional input-validation techniques are ineffective when the model itself is the exploited component [48][46]. The protocol is maturing in response: the November 2025 MCP specification formalized OAuth 2.1 for remote servers, but security governance is still catching up to adoption, and real incidents in 2025 (including cross-tenant data exposure and injection against a widely used MCP server) show the gap is concrete [49]. For enterprises, the implication is direct: context engineering is inseparable from context governance.

Context engineering is a system, not a prompt

The through-line of every serious treatment of this topic is the same: reliability comes from the pipeline, not from any single clever input. As one 2026 analysis put it, the gap between a relevant answer and a poor one is not a model problem — a smaller model with well-engineered context can outperform a much larger one without it, and the difference is the discipline of designing what the AI sees before it responds [45]. That is a system: a set of interconnected decisions about what information reaches the model, in what form, at what time, and under what constraints [45].

This is where context engineering meets the harder question enterprise teams are actually in the consideration stage about — not "which model?" but "what surrounds the model?" Selecting, ordering, compressing, isolating, attributing, and securing context across many sources and many steps is orchestration. It requires a control plane that decides what enters each window, monitors for rot and clash, enforces provenance and access rules, and adapts as the underlying data changes. At Etheon, that is the premise we build on: the durable layer of enterprise AI is the system that assembles and governs context, because the model is increasingly interchangeable and the context is what makes it correct, safe, and accountable. Prompts got enterprises to the demo. Context engineering — done as a governed system — is what gets them to production.

FAQ

What is context engineering? Context engineering is the discipline of curating and maintaining the optimal set of tokens a model processes at inference — system instructions, retrieved documents, tool results, memory, and conversation history — so the model has the right information, in the right structure, at the right moment [3][6]. Anthropic formalized the definition in September 2025; the term was popularized by Tobi Lütke and Andrej Karpathy in June 2025 [1][3].

What is the difference between prompt engineering and context engineering? Prompt engineering asks how to phrase an instruction; context engineering asks what the model should know before it generates, from which sources, in what order [16][12]. Prompt engineering is now considered a subset of context engineering — one small part of the total context that fills a production agent's window [1][3].

Why aren't prompts enough for enterprise AI? Enterprise context is dynamic, multi-source, and time-sensitive; the context window is a finite, costly resource; and research shows more context degrades quality ("context rot" and the "lost in the middle" effect), so simply enlarging the prompt fails [26][27][18]. Enterprise AI also requires provenance, freshness, and security controls that a static prompt cannot provide [15][46].

What is context rot? Context rot is the measured decline in a model's ability to use information accurately as the input grows longer. Chroma's July 2025 study found all 18 frontier models tested degrade with length, even on simple tasks, and Liu et al. showed accuracy drops over 30 points when key facts sit in the middle of a long context [27][26].

References

1. Tao An — Context Engineering Is Replacing Prompt Engineering for Production AI (documenting Lütke's June 18, 2025 and Karpathy's June 25, 2025 posts; Anthropic's Sept 2025 formalization). https://tao-hpu.medium.com/context-engineering-is-replacing-prompt-engineering-for-production-ai-02205fad2a7f

2. Glasp — Context Engineering: Why Prompt Engineering Is Dead (And What Replaced It) (Lütke/Karpathy timeline; Cognition/Walden Yan; Gartner briefings; Chroma). https://glasp.co/articles/context-engineering

3. Zep — What Is Context Engineering, Anyway? (Lütke and Karpathy quotes; Anthropic definition). https://blog.getzep.com/what-is-context-engineering/

4. arXiv — Context Engineering: A Practitioner Methodology for Structured Human-AI Collaboration (Karpathy, Lütke, Willison, Breunig framing). https://arxiv.org/pdf/2604.04258

5. decodethefuture.org — What Is Context Engineering? 5 Pillars Behind Reliable AI (Karpathy/Lütke; Anind K. Dey 2001; Context Engineering 2.0). https://decodethefuture.org/en/what-is-context-engineering/

6. Anthropic — Effective Context Engineering for AI Agents (engineering blog, Sept 29, 2025). https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

7. Vorstel — Effective Context Engineering for AI Agents: A Comprehensive Guide (Cognition "#1 job" quote; tool-token and multi-agent token figures). https://vorstel.com/feeds/blog/effective-context-engineering-ai-agents

8. GitHub — langchain-ai/how_to_fix_your_context (Chroma summary; Breunig four failure modes; LangChain techniques). https://github.com/langchain-ai/how_to_fix_your_context

9. arXiv — Context Engineering: A Practitioner Methodology (Breunig: prompts are instructions, context is everything needed to act). https://arxiv.org/html/2604.04258v1

10. Towards Data Science — Context Engineering for RAG: The Four Typed Inputs Behind Every Answer. https://towardsdatascience.com/context-engineering-for-rag-the-four-typed-inputs-behind-every-rag-answer/

11. The Context Graph — Context Engineering in 2026: From Karpathy's Tweet to Production Infrastructure (temporal validity, provenance, inter-agent context). https://thecontextgraph.co/memos/context-engineering-2026-from-tweet-to-infrastructure

12. decodethefuture.org — What Is Context Engineering? (CPU/RAM/operating-system analogy; prompt vs context question). https://decodethefuture.org/en/what-is-context-engineering/

13. arXiv — Interpretable Context Methodology (LangChain / Lance Martin write–select–compress–isolate taxonomy). https://arxiv.org/pdf/2603.16021

14. howaiworks.ai — Anthropic Context Engineering for Agents (context rot; n² attention; Sept 29, 2025 post). https://howaiworks.ai/blog/anthropic-context-engineering-for-agents

15. MarkTechPost — A Guide for Effective Context Engineering for AI Agents (context rot; limited attention). https://www.marktechpost.com/2025/10/20/a-guide-for-effective-context-engineering-for-ai-agents/

16. Vorstel — Effective Context Engineering (failure modes as architectural and preventable; compaction, note-taking, multi-agent). https://vorstel.com/feeds/blog/effective-context-engineering-ai-agents

17. aiwithgrant — Context Engineering for Agents — Anthropic (ordering: system instructions, memory, tools, history). https://www.aiwithgrant.com/guides/anthropic-context-engineering-agents

18. MachineLearningMastery — Effective Context Engineering for AI Agents: A Developer's Guide (what enters/compresses/retrieves/drops). https://machinelearningmastery.com/effective-context-engineering-for-ai-agents-a-developers-guide/

19. arXiv — The Cognitive Divergence (Liu et al. 2024 lost-in-the-middle; U-shape; >30% drop; RoPE decay). https://arxiv.org/pdf/2603.26707

20. Product Talk — Context Rot: Why AI Gets Worse the Longer You Chat (Chroma 2025; Liu 2023; Veseli 2025). https://www.producttalk.org/context-rot/

21. Medium (Quang Tran) — Context Rot: What a 2 AM Night Taught Me (Chroma July 2025, 18 models; Liu et al. TACL 2024). https://medium.com/@trmquang3103/context-rot-what-a-2-am-night-taught-me-that-benchmarks-never-did-dfda7df72f40

22. Particula — Your 1M Context Window Is Lying: What Chroma's Context Rot Study Proves (18 models; 30-point mid-context drop; RULER effective-context estimate). https://particula.tech/blog/chroma-context-rot-long-context-degradation

23. arXiv — Cross-Context Review (Liu et al. 2024 and Hong et al. 2025 context rot across 18 models). https://arxiv.org/html/2603.12123

24. Cobus Greyling — LLM Context Rot (Chroma: GPT-4.1, Claude 4, Gemini 2.5, Qwen3; enterprise implications). https://cobusgreyling.substack.com/p/llm-context-rot

25. arXiv — Lost in the Middle, and In-Between (Liu et al. 2024; NaturalQuestions position experiments). https://arxiv.org/pdf/2412.10079

26. Antoine Buteau — Lessons from Drew Breunig (poisoning, distraction, confusion, clash; tool loadout; quarantine; auditable memory). https://www.antoinebuteau.com/lessons-from-drew-breunig/

27. GitHub — langchain-ai/how_to_fix_your_context (Breunig four modes; Chroma across 18 models). https://github.com/langchain-ai/how_to_fix_your_context

28. Simon Willison — How to Fix Your Context (Breunig four patterns; tool loadout >20 tools). https://simonwillison.net/2025/Jun/29/how-to-fix-your-context/

29. O'Reilly Radar — Working with Contexts (million-token illusion; MCP tool clash; poisoned/distracting/confusing/conflicting). https://www.oreilly.com/radar/working-with-contexts/

30. Medium (Macplanet) — Context Engineering vs. Prompt Engineering (Breunig taxonomy; DeepMind Gemini 2.5 poisoning; measuring thresholds). https://medium.com/@macplanet2012/context-engineering-vs-prompt-engineering-why-your-ai-agent-gets-dumber-the-longer-it-runs-1583d7568e0e

31. Drew Breunig — How Long Contexts Fail (dbreunig.com, June 22, 2025; DeepMind Gemini 2.5 report). https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html

32. Drew Breunig — How to Fix Your Context (dbreunig.com, June 26, 2025; four modes; information management). https://www.dbreunig.com/2025/06/26/how-to-fix-your-context.html

33. Tricky Wombat — Context Engineering (Databricks Mosaic Research on failure types; smaller model + good context beats bigger model; pipeline framing). https://www.trickywombat.ai/signals/context-engineering

34. eSentire — Model Context Protocol Security: Critical Vulnerabilities Every CISO Must Address in 2025 (OWASP #1 prompt injection; MCP amplification; Microsoft integration). https://www.esentire.com/blog/model-context-protocol-security-critical-vulnerabilities-every-ciso-should-address-in-2025

35. arXiv — MCP Threat Modeling and Prompt Injection with Tool Poisoning (18,000+ MCP servers; OpenAI/Google/Meta/Microsoft; validation ineffective when model is the exploited component). https://arxiv.org/html/2603.22489v1

36. Cloud Security Alliance — Agentic MCP Security Best Practices Guide (Nov 2025 spec, OAuth 2.1; 2025 incidents; governance lag). https://labs.cloudsecurityalliance.org/agentic/agentic-mcp-security-best-practices-v1/

37. arXiv — Dependency-Aware Privacy for Multi-turn Agents (Anthropic introduced MCP, Nov 2024; indirect prompt injection references). https://arxiv.org/pdf/2605.03188