Second Brain · Architecture
Mem0 & LLM Memory: The Architecture Behind AI Second Brains
An LLM has no memory of its own — the model is stateless, and every request is answered only from what is in the prompt. "LLM memory" is an external layer that decides what to store, what to retrieve, and what to inject into the next prompt. There are three real approaches: a rolling context window (no persistence), vector recall (RAG-style, persistent but accumulate-only), and agentic memory — explicit add / update / delete operations on a managed store, which is where Mem0 and Letta sit. The architectural problem none of them fully solves is the consolidation step between short-term and long-term memory. Pick by how long facts must live and whether they change.
Why it matters
Why does an AI second brain need a memory layer at all?
Because the model has no memory. A large language model is a stateless function: it reads the prompt, returns a completion, and forgets everything the moment the request ends. Every sense in which an assistant "remembers" you is a layer someone built around that function — capturing what matters, storing it, and feeding it back in on the next request.
This is the part most product copy skips, and skipping it is where the confusion starts. People hear "the AI remembers your preferences" and picture something changing inside the model. Nothing changes inside the model. What changes is a store outside it and a retrieval query that pulls from that store into the next prompt. Once you see memory as a pipeline rather than a property of the model, the design choices get concrete: what do you extract, where do you keep it, and when do you decide a remembered fact has gone stale.
This page is the architecture layer underneath the AI second brain guide. The pillar covers the method and the build-vs-buy call; this one opens up the memory system that makes any of it work — the four kinds of memory, the three ways to implement the long-term layer, and where Mem0 and agentic memory actually fit.
The layers
What are the different types of memory in an LLM system?
"Memory" is one word doing four jobs. Separating them is the single most useful move in this whole topic, because the layer people argue about (long-term recall) is rarely the layer that is actually failing (the consolidation gap). Here are the four, from the weights outward.
Parametric memory
What the model learned during training, baked into the weights. You cannot edit it at inference time, it has a knowledge cutoff, and it is the same for every user. This is not the memory anyone means when they say "give my agent memory" — but it is worth naming, because it is the layer people confuse with the rest.
Short-term (context) memory
The prompt itself — the rolling window of recent turns the model sees on this request. Fast, free to read, and finite. The oldest turns fall off the end silently, which is exactly why a rule set early in a long chat quietly stops being honored.
Long-term (external) memory
The persistent store outside the model: a vector database, a document store, a structured fact table. It survives across sessions and is queried on demand. This is where a memory layer like Mem0 lives — it decides what enters this store and what gets pulled back into the prompt.
Working memory (the gap)
The consolidation step between short-term and long-term — deciding what from the active window is worth promoting, and resolving it against what is already stored. Almost every system has the other three and treats this one as an afterthought. It is the layer to watch.
Almost every memory product on the market addresses layers two and three — a bigger context window, a better vector store. The fourth layer, working memory, is the one that determines whether an assistant feels coherent over a long relationship, and it is the one nobody has fully shipped. Hold onto that distinction; it is the lens for everything below.
The approaches
Rolling window, vector recall, or agentic memory: which do you need?
There are three ways to implement the long-term layer, and they are not interchangeable. They trade off persistence and the ability to correct a fact against complexity. Pick by how long facts must survive and whether they change — not by which framework has the most stars.
| Approach | How it works | Best for | Persistence | Main weakness |
|---|---|---|---|---|
| Rolling context window | Keep replaying recent turns into the prompt; drop the oldest when you run out of room. No external store — memory is just the buffer. | Short single-session chats where nothing needs to persist past the conversation. | None across sessions | Silently forgets early constraints; cost grows with history length; hard cap at the window size. |
| Vector recall (RAG-style) | Embed past messages or extracted facts, store the vectors, and retrieve the top semantic matches per query to inject into the prompt. | Recalling relevant facts from a large history without replaying all of it. | Persistent store | Retrieval can miss or surface stale facts; no built-in notion of updating or deleting a changed fact — it just accumulates. |
| Agentic memory | Treat memory as explicit operations — extract facts from interactions, then add, update, or delete entries in a managed store rather than appending blindly. Mem0 and Letta sit here. | Long-lived assistants where facts change over time and must be corrected, not just accumulated. | Persistent + mutable | More moving parts; extraction and conflict-resolution quality bound the whole system; framework dependency to maintain. |
The line that matters is between vector recall and agentic memory. Vector recall accumulates: every fact you store stays stored, and a corrected fact sits in the index next to the wrong one it was supposed to replace. Agentic memory treats storage as explicit operations — add a new fact, update a changed one, delete a retracted one — so the store reflects the current truth rather than the full history. That difference is the entire reason memory frameworks exist.
The framework
What is Mem0, and where does it fit?
Mem0 is an open-source memory layer for AI agents and assistants. Its job is to sit between your agent and a store, extract the facts worth keeping from each interaction, and retrieve only the relevant ones into the next prompt — rather than replaying an ever-growing transcript that eventually overflows the context window.
The framing that sets it apart from a plain vector store is that memory is a set of operations on a managed store, not an append-only log. New information can add a fact, update one that changed, or be discarded when it adds nothing — so the store trends toward the current state of what is true about a user rather than the full accumulated history. That is the agentic-memory pattern from the table above, made concrete. Mem0 and the broader memory-framework space move fast; verify the current operation set, defaults, and any version-specific behavior against the project's own documentation before you build against a specific claim.
Mem0 is not the only thing in this category. Letta (formerly MemGPT) is a stateful-agent platform that targets the same working-memory problem from the agent side, with memory that persists across sessions. If you have already decided you want a framework and the question is which one, the scored head-to-head is on the WTF Radar comparison of Mem0 vs Letta. This page is deliberately not that ranking — it is the architecture you need to understand before the ranking is a useful question.
A common confusion
Is a memory layer the same as RAG?
No — and conflating them leads to the wrong architecture. RAG retrieves from a static corpus you already have; a memory layer manages information generated during use. They are different inputs to the same prompt, and most serious agents run both.
RAG (retrieval-augmented generation) grounds answers in documents you bring to the system — a wiki, a notes folder, a product manual. The corpus is authored elsewhere and largely fixed; retrieval finds the relevant chunk. A memory layer is concerned with what happens in the conversation: the fact a user just stated, the preference they revealed, the decision made three turns ago. RAG answers "what does my corpus say about X." Memory answers "what has this user told me, and has it changed."
The practical consequence: do not reach for a memory framework to solve a document-retrieval problem, and do not expect a vector store of your PDFs to remember a correction the user made mid-chat. For a personal knowledge base specifically, there is a third option that often beats both — an agent reading plain files on demand. I ran all three head-to-head on the same corpus in the hands-on experiment, where the file-based agent was the most faithful and production RAG confabulated a source that did not exist.
The decision
How do you choose a memory architecture?
Match the architecture to how long facts must live and whether they change — not to the framework with the best landing page. Here is the call by profile.
| Your situation | Architecture | Why |
|---|---|---|
| Single-session chatbot, nothing to persist | Rolling window | A persistent store is overhead you do not need; the context window is the memory, and that is fine for the lifespan of one conversation. |
| Assistant that should recall past facts | Vector recall | Embed and retrieve extracted facts so you recall what matters without replaying everything — but accept that facts only accumulate unless you add update logic. |
| Long-lived assistant where facts change | Agentic memory (Mem0 / Letta) | You need explicit update and delete so a corrected fact overwrites the old one. This is the consolidation logic you would otherwise hand-roll and maintain. |
| Personal knowledge base under ~1M tokens | File-based agent (often) | For a corpus you own, an agent reading plain files on demand is frequently more faithful than a memory framework — I tested this; the experiment is linked below. |
The honest default for most builders: start with vector recall, and only add a memory framework when you hit the specific pain it solves — facts that change and must be corrected, or memory that has to behave the same across several agents. Reaching for the framework first is the same mistake as reaching for a vector database before you have a retrieval problem. Add the layer when the failure mode shows up, not before.
The open problem
What is still unsolved in LLM memory?
Working memory — the consolidation step between the short-term context window and long-term storage. Every system has the buffer and the store; almost none has a principled step in between that decides what to promote, when, and how to reconcile it with what is already saved.
The symptom is concrete and you have probably hit it: set a rule in turn one of a long chat, work through unrelated questions for a few turns, and by turn five the rule is gone. The model did not "forget" in any cognitive sense — the constraint fell out of the rolling history window and was never promoted to a place that would re-inject it. That is a missing consolidation step, not a tuning problem, and it is why "it remembers everything" claims rarely survive contact with a long session.
Agentic-memory frameworks are the closest thing shipping — explicit add / update / delete is the right primitive for it — but the harder question of when to consolidate, ideally on idle rather than on every turn, is still largely open as of mid-2026. The first product that ships a real consolidation step will leapfrog the category. Until then, treat memory as a pipeline you have to reason about, and treat any vendor's "it just remembers" as the marketing it is.
Go deeper
Keep building your second brain
AI Second Brain: The Complete Guide
The pillar this page sits under — the BASB CODE method, RAG vs long-context vs file-based architectures, the tool landscape, and a build-vs-buy verdict for knowledge workers.
The Hands-On Experiment
Three second-brain architectures tested head-to-head on the same corpus and the same seven questions, including a working-memory probe that broke most of them.
Mem0 vs Letta (WTF Radar)
The scored, head-to-head comparison of the two memory frameworks named on this page — if you have already decided you want a framework and need to pick one.
Frequently Asked Questions
What is LLM memory?
What is Mem0 and what problem does it solve?
How is Mem0 different from RAG?
What is the difference between short-term and long-term memory in an LLM system?
Do I need a memory framework, or can I build memory myself?
Does adding memory make an AI agent more reliable?
See the memory architectures break in practice
Theory only gets you so far. The hands-on experiment runs three second-brain memory architectures over the same corpus and the same questions — and shows you which one leaks constraints first.