State: in-context versus external memory

Where an agent keeps what it knows, and why the choice matters.

5 min read · Lesson 12 of 12 in this domain

An agent has two places to keep state and they have opposite properties. In-context state — the message history you resend each turn — is immediate and needs no tooling, but it is bounded by the window, costs tokens on every single request, and disappears when the session ends. External state — files, a database, a memory directory — is unbounded and durable and costs nothing to hold, but the agent can only see it by spending a tool call to read it. The design question is therefore not "which is better" but "what does the agent need in front of it right now, versus what should it be able to fetch when relevant?" Getting this wrong in either direction is expensive: everything in context burns the window, everything external turns a simple task into a chain of lookups.

• In-context: immediate, no tool call, but window-bounded, re-billed every request, and lost at session end.
• External: durable and unbounded, but requires a tool call to reach and must be found before it can be used.
• Keep hot, decision-relevant state in context; push cold reference material and cross-session knowledge outside.
• The memory tool is the external option that survives process restarts, giving cross-session persistence that compaction and context editing cannot.
• Externalising also makes state inspectable — a checkpoint file can be read, diffed and resumed from, which is what makes crash recovery tractable.
• Never store credentials in memory files: they are replayed verbatim into every future session that mounts the store.

Source: Claude Docs — Memory tool · Independent study aid, not affiliated with or endorsed by Anthropic.