The Four Types of AI Agent Memory

TLDR

AI agents use four memory types from the CoALA framework: working, semantic, procedural, and episodic — each serving a distinct role in agent capability.
Memory architecture is what separates chatbots (context window only) from agents (persistent, accumulated, shaped knowledge).
Not every agent needs all four — match memory types to the agent’s complexity and task scope.

Caveman

Agents have four memory types; match to complexity.

ELI5

AI agents remember in four different ways, just like humans do: what they’re working on right now, facts and rules they know, skills they’ve learned, and lessons from past sessions. Each type is stored and loaded differently. Simple agents only need one or two; complex coding agents need all four.

Trunk → Branches

Trunk: The CoALA framework (Princeton) defines four memory types agents need — working, semantic, procedural, and episodic — each enabling progressively more capable and persistent behavior.

Branches:

Working memory = the context window; everything visible right now (conversation, system prompt, loaded files). Fast but volatile — cleared when the session ends, and performance degrades when overloaded.
Semantic memory = persistent knowledge base (facts, rules, conventions, docs). In production this is often simple Markdown files (e.g., CLAUDE.md) loaded at session start, not necessarily a vector database.
Procedural memory = how the agent does things, encoded as skill.md files. Uses progressive disclosure: lightweight index loaded at start, full instructions pulled only when a task matches a skill.
Episodic memory = distilled record of past sessions and decisions. Production systems compress to useful summaries (“issue was in middleware layer”) rather than storing raw transcripts.
Agent complexity determines which types are needed: reflex/routing agents → working only; simple task agents (e.g., password reset) → working + procedural; full coding agents → all four.
Forgetting is an engineering problem — episodic memory requires deliberate curation of what to keep vs. drop as context changes.

Key Patterns

Pattern: Progressive disclosure for procedural memory (lightweight index → full skill on match) works because it keeps context budget intact while keeping all skills accessible — transfers to any token-budget-constrained system with many available tools or workflows.

Pattern: Semantic memory as static Markdown works in production because the knowledge is stable and always needed — vector DBs add complexity only when the knowledge base is too large to load in full; start simple and upgrade only when hit.

Pattern: Episodic memory should be distilled summaries, not raw logs — compressed insight (“auth bug lived in middleware”) beats a 45-minute transcript — transfers to any human or agent note-taking / after-action system.

Stress Test

Hidden assumption: The four memory types map to four distinct systems. In practice, boundaries blur — episodic and semantic memory may share the same vector store.
Best counterargument: Episodic memory adds meaningful engineering complexity with unclear payoff for most production agents; many teams ship effectively with only working + semantic.
Fails when: The agent needs to unlearn stale or incorrect information — the “forgetting” problem in episodic memory has no clean off-the-shelf solution and must be designed explicitly.
Fact vs opinion: CoALA as a Princeton framework is factual. “Episodic memory is the hardest to get right” is an experience-based opinion, not a measured claim.

In Practice

Reserve working memory for truly active-session data — avoid loading large static knowledge into the context window when it could live in semantic memory instead. ¶2:09
Implement semantic memory as simple Markdown files loaded at session start before reaching for vector databases — simpler and sufficient for many production agents. ¶3:59
Use progressive disclosure for skills: load only name + description at startup (~100 tokens per skill), pull full skill.md instructions only when a task matches. ¶5:29
Distill episodic memory into compressed, decision-relevant notes per session rather than archiving full transcripts. ¶7:12
Match memory type selection to agent complexity — avoid over-engineering simple agents with all four types when fewer suffice. ¶8:22
Treat “what to forget” as an explicit design decision in episodic memory — stale data accumulates and misleads if not actively curated. ¶7:52

Connections

Why RAG Fails on Complex Documents — This note’s advice to implement semantic memory as static Markdown and ‘upgrade to a vector DB only when hit’ mirrors the RAG note’s thesis that curation and appropriate scale matter more than jumping to a bigger retrieval system.
Building a Claude Code Agentic OS — The progressive-disclosure pattern for procedural memory — a lightweight index loaded at start, full skill.md pulled only on match — is the exact skills mechanism the Agentic OS note codifies into an indexed vault plus automations.
Hermes Agent Fundamentals — Both use a four-tier memory taxonomy for agent design — worth checking whether Hermes’s ‘always-on 4-tier memory system’ actually implements CoALA’s working/semantic/procedural/episodic split or just borrows the number four.
How To Think Clearly Using Systems — ‘Match memory types to agent complexity — avoid over-engineering simple agents with all four types’ mirrors that note’s core move of diagnosing which of four system types you’re in before acting: classify the problem’s shape first, then apply only the minimum machinery needed.
Compound Engineering with Claude Code — Episodic memory’s ‘distilled summaries, not raw logs’ is what the compound engineering loop’s Assess/Compound steps operationalize — turning each completed task into a compressed lesson that improves the next run instead of an unprocessed transcript.