Let's Build an AI Assistant That Remembers

Sebastian Lund — Fri, 13 Feb 2026 00:00:00 +0000
A founder friend messaged me recently:
When do we trigger compaction? Context is finite, so at some point we have to compress. Priority-based, task-specific, time-based… what have you tried? </blockquote>
These are the questions most people start with. How do I compress? When do I trigger that? How do I retrieve what’s relevant? They’re the right questions, but going from concepts to a working implementation isn’t straightforward.
Let’s build one together: an assistant that remembers everything. It won’t be production-grade. It’s a foundation, enough to understand how the pieces fit together and a starting point for whatever your product needs.
The starting point</h2>
First, we need to get a basic assistant going. We’ll use Next.js</a> for the app, Vercel’s AI SDK</a> for LLM integration, and Cria</a> for prompt composition. AI SDK handles the plumbing and state management. Cria handles the prompt and all future memory components. Together, they keep us from reinventing the wheel while staying easy to customize.
A bare minimum chat route looks like this:
// app/api/chat/route.ts</span></span>
import</span> { streamText }</span> from</span> "ai"</span>;</span></span>
import</span> { createOpenAI }</span> from</span> "@ai-sdk/openai"</span>;</span></span>
</span>
const</span> openai</span> =</span> createOpenAI</span>({ apiKey: process.env.</span>OPENAI_API_KEY</span> });</span></span>
</span>
export async function</span> POST</span>(</span>req</span>:</span> Request</span>) {</span></span>
  const</span> {</span> messages</span> }</span> = await</span> req.</span>json</span>();</span></span>
</span>
  return</span> streamText</span>({</span></span>
    model:</span> openai</span>(</span>"gpt-4o-mini"</span>),</span></span>
    system:</span> "You are a helpful assistant."</span>,</span></span>
    messages,</span></span>
  }).</span>toDataStreamResponse</span>();</span></span>
}</span></span></code></pre>
And a client to use it:</p>
// app/page.tsx</span></span>
"use client"</span>;</span></span>
import</span> { useChat }</span> from</span> "@ai-sdk/react"</span>;</span></span>
</span>
export default function</span> Chat</span>() {</span></span>
  const</span> {</span> messages</span>,</span> input</span>,</span> handleInputChange</span>,</span> handleSubmit</span> }</span> =</span> useChat</span>();</span></span>
</span>
  return</span> (</span></span>
    <</span>div</span>></span></span>
      {messages.</span>map</span>((</span>m</span>)</span> =></span> (</span></span>
        <</span>div</span> key</span>=</span>{m.id}>{m.role</span> ===</span> "user"</span> ?</span> "You: "</span> :</span> "AI: "</span>}{m.content}</</span>div</span>></span></span>
      ))}</span></span>
      <</span>form</span> onSubmit</span>=</span>{handleSubmit}></span></span>
        <</span>input</span> value</span>=</span>{input}</span> onChange</span>=</span>{handleInputChange} /></span></span>
      </</span>form</span>></span></span>
    </</span>div</span>></span></span>
  );</span></span>
}</span></span></code></pre>
That’s a working chat. User sends a message, model streams a response.</p>
Here’s what a stylized version looks like. See the full example on GitHub</a> for the complete code.</em></p>

You now have a basic assistant wired together in minimal code. It won’t remember anything between sessions, but you can talk to it like you would ChatGPT.</p>
Designing the memory lifecycle</h2>
Before we wire memory up, we need to think about how it fits into our product. LLMs can’t keep going forever. They are constrained in several ways:</p>

LLMs have context limits that eventually prevent you from sending more input in the same session.</li>
LLMs pay more attention to the start</em> and end</em> of their context window than to the middle. For an assistant, the initial messages and the latest ones always carry the most weight.</li>
LLMs have diminishing returns past a certain context size (e.g., ~200,000 tokens for OpenAI models), and going beyond that increases the risk of hallucinations.</li>
LLMs are hardwired through reinforcement learning to reach an “end state” where they solve a problem. If you’re a Claude user, you may have noticed how it gets short once it thinks it’s answered your question, and keeping the conversation going past that point almost becomes painful.</li>
</ul>
These constraints mean we need a natural end point for a session. For an assistant, that’s inactivity: when the user stops messaging, we know they’re done, so we end the session and save what happened. Most teams call this compaction.</p>
For users who never go idle, we need to force sessions to end. Teams call this forced compaction: write memories, carry them into a new session, keep going. We want to avoid it when possible, and reinforcement learning helps. Models like Claude Sonnet and Opus naturally steer conversations toward resolution, nudging the user toward a stopping point.</p>
So we know when</em> to write memories. The next question is what</em> to write. We need episodic memory to summarize past conversations, semantic memory to remember durable facts about the user, and a recall mechanism for finding relevant past sessions once there are too many to include directly.</p>
These three memory types let the assistant:</p>

Summarize past sessions, i.e., “what happened before the user went idle”</li>
Maintain a fact sheet about the user, e.g., “what is their name?”</li>
Recall past sessions by topic, i.e., “what did we talk about last week?”</li>
</ol>
Layer</th> Memory type</th> Primitive</th> What it does</th></tr></thead>

Working</td> Current context</td> Recent messages</td> Last N messages kept verbatim</td></tr>
Episodic</td> What happened</td> Summarizer</td> Distills each session into a structured log</td></tr>
Semantic</td> Who this person is</td> Summarizer</td> Extracts durable facts into a structured user profile</td></tr>
Vector recall</td> Specific past conversations</td> Vector DB</td> Indexes session summaries for similarity search</td></tr>
</tbody></table>
I covered LLM memory types</a> in a previous post if you want to know more about how to design this.</em></p>
Notice the summarizer appears twice. Episodic and semantic memory both use summarization, but for different purposes: episodic distills one session</em> into a log of events. Semantic extracts durable facts across many sessions</em> into a structured profile. Same primitive, different keys, different prompts.</p>
The flow:</p>

User chats → conversation accumulates (working memory)</li>
User goes idle → session boundary fires</li>
On boundary → extract episodic, semantic, and vector memories</li>
User returns → fresh session, bootstrapped with stored memories</li>
</ol>
Now we know what to build, so let’s get to it!</p>
Structuring memory</h2>
Multiple parts of the app use memory: The chat route reads it, the save endpoint writes it. Best to wrap it in an interface we can iterate on internally and plug in wherever it’s needed:</p>
// src/lib/memory.ts</span></span>
type</span> SessionMessage</span> =</span> {</span> role</span>:</span> "user"</span> |</span> "assistant"</span>;</span> content</span>:</span> string</span> };</span></span>
</span>
class</span> MemoryManager</span> {</span></span>
  async</span> prompt</span>(</span>userId</span>:</span> string</span>,</span> messages</span>:</span> SessionMessage</span>[])</span>:</span> Promise</span><</span>PromptPlugin</span>> {</span></span>
    // Build a memory-aware prompt for the chat route</span></span>
  }</span></span>
</span>
  async</span> save</span>(</span>userId</span>:</span> string</span>,</span> messages</span>:</span> SessionMessage</span>[])</span>:</span> Promise</span><</span>void</span>> {</span></span>
    // Extract and store memories when a session ends</span></span>
  }</span></span>
}</span></span>
</span>
export const</span> Memory</span> = new</span> MemoryManager</span>();</span></span></code></pre>
Plugging the module into our assistant is straightforward:</p>
// app/api/chat/route.ts</span></span>
...</span></span>
const</span> memory</span> = await</span> Memory.</span>prompt</span>(userId, messages);</span></span>
const</span> rendered</span> = await</span> cria</span></span>
  .</span>prompt</span>(provider)</span></span>
  .</span>system</span>(</span>SYSTEM_PROMPT</span>)</span></span>
  .</span>use</span>(memory)</span></span>
  .</span>render</span>({ budget:</span> 40_000</span> });</span></span>
</span>
return</span> streamText</span>({</span> ...</span>, messages: rendered }).</span>toDataStreamResponse</span>();</span></span>
...</span></span></code></pre>
The other methods wire up to other routes. In the demo, we added a “Save Session” button that forces the session to end. Much easier for testing than waiting minutes for the inactivity trigger.</p>
Layer 1: episodic. What happened</h2>
When a session ends, we have a full conversation sitting in memory. Could be five messages, could be 50. Before we throw it away, we want to save the important pieces: what was the user trying to do, what happened, and what’s left unresolved. That’s episodic memory: a structured log of the session.</p>
We need two things for this: somewhere to store these summaries, and something that can distill a conversation into one. We’ll use SQLite for storage since we’re prototyping, and Cria gives us a summarizer primitive that does the extraction:</p>
const</span> summaryStore</span> = new</span> SqliteStore</span><</span>StoredSummary</span>>({</span></span>
  filename:</span> "./data/cria.sqlite"</span>,</span></span>
  tableName:</span> "cria_summaries"</span>,</span></span>
});</span></span>
</span>
// Factory: same store, different IDs and prompts produce different summaries.</span></span>
const</span> summarizer</span> =</span> (</span>id</span>:</span> string</span>,</span> prompt</span>?:</span> string</span>)</span> =></span></span>
  cria.</span>summarizer</span>({ id, provider, store: summaryStore, priority:</span> 2</span>, prompt });</span></span></code></pre>
A summarizer takes a conversation in, extracts the pieces you care about according to your prompt, and stores the result. Call it again with new input, and it rolls the new content into the existing summary. Our save</code> method is one call:</p>
async</span> save</span>(userId: string, messages: SessionMessage[]) {</span></span>
  const</span> date</span> = new</span> Date</span>().</span>toISOString</span>().</span>split</span>(</span>"T"</span>)[</span>0</span>];</span></span>
  const</span> summary</span> = await</span> summarizer</span>(</span></span>
    `episodic:${</span>userId</span>}`</span>,</span></span>
    `Summarize this conversation into a structured session log.</span></span>
</span>
Date: ${</span>date</span>}</span></span>
Objective: What the user was trying to do.</span></span>
Timeline: Bullet points of what happened, key decisions and discoveries.</span></span>
Outcome: What got done, what's unresolved.</span></span>
</span>
Keep it concise. No markdown formatting. One line per bullet.`</span>,</span></span>
  ).</span>writeNow</span>({ history:</span> toScope</span>(messages) });</span></span>
}</span></span></code></pre>
We customize the prompt so the output is structured and scannable rather than a wall of prose. For example, if the user talks about deploying to Vercel, we may end up with a summary such as:</p>
Date: 2026-02-13</span></span>
Objective: Deploy a Next.js app to Vercel.</span></span>
Timeline:</span></span>
- User confirmed app is running locally and hosted on GitHub.</span></span>
- Discussed managing environment variables for database connections and API keys.</span></span>
- Discovered Vercel supports separate env vars for production and preview deployments.</span></span>
- User noted they could point preview deployments to a staging database.</span></span>
- Explored secrets management for CI pipeline; Vercel integrates with HashiCorp Vault and AWS Secrets Manager.</span></span>
- User flagged potential compliance requirements for health data.</span></span>
Outcome: Deployment approach settled. Pending: security team input on</span></span>
compliance requirements for secrets management.</span></span></code></pre>
The model now knows what the user was trying to do</em>, what they tried</em>, and where they left off</em>.</p>
Now we wire the summary back into the prompt so the model sees it when the user returns:</p>
async</span> prompt</span>(userId: string, messages: SessionMessage[]) {</span></span>
  const</span> previousSession</span> = await</span> summarizer</span>(</span>`episodic:${</span>userId</span>}`</span>).</span>get</span>();</span></span>
</span>
  return</span> cria.</span>prompt</span>(provider)</span></span>
    .</span>system</span>(</span>`## Previous Session</span>\n</span>${</span>previousSession</span>}`</span>)</span></span>
    .</span>use</span>(</span></span>
      // Safeguard: if the conversation outgrows the token budget,</span></span>
      // the summarizer kicks in and condenses older messages.</span></span>
      summarizer</span>(</span>`history:${</span>userId</span>}`</span>).</span>plugin</span>({</span></span>
        history:</span> toScope</span>(messages),</span></span>
      })</span></span>
    );</span></span>
}</span></span></code></pre>
We read the stored summary from last session and inject it as context. We wrap the current conversation in a summarizer plugin that acts as a compaction safeguard: Most of the time messages pass through unchanged, but if the conversation outgrows the token budget, the summarizer kicks in and condenses older turns automatically. The same primitive we use to save memories between sessions also protects us within a single session.</p>
That single layer already lets the user leave and come back the next day with context intact. The problem is that each save overwrites the previous summary. Storing them separately doesn’t help much either: The model would read disconnected session logs. It knows what happened</em> each time but has no coherent picture of who this person is</em>.</p>
Layer 2: who is this person</h2>
After three sessions, the model might know:</p>

Session 1: User asked about Python pandas.</li>
Session 2: User debugged a data pipeline.</li>
Session 3: User asked about deployment.</li>
</ul>
These are isolated events. What we want the model to know is: this is a data engineer who works primarily in Python and is currently building a pipeline they want to deploy. That’s semantic memory. Durable facts that persist regardless of which session they came from.</p>
The approach is simple: take each session summary and feed it into a second summarizer that maintains a rolling user profile. Same tool, different purpose. We extend save</code> to capture the summary and pipe it forward:</p>
async</span> save</span>(userId: string, messages: SessionMessage[]) {</span></span>
  const</span> date</span> = new</span> Date</span>().</span>toISOString</span>().</span>split</span>(</span>"T"</span>)[</span>0</span>];</span></span>
  const</span> summary</span> = await</span> summarizer</span>(</span></span>
    `episodic:${</span>userId</span>}`</span>,</span></span>
    `Summarize this conversation into a structured session log.</span></span>
</span>
Date: ${</span>date</span>}</span></span>
Objective: What the user was trying to do.</span></span>
Timeline: Bullet points of what happened, key decisions and discoveries.</span></span>
Outcome: What got done, what's unresolved.</span></span>
</span>
Keep it concise. No markdown formatting. One line per bullet.`</span>,</span></span>
  ).</span>writeNow</span>({ history:</span> toScope</span>(messages) });</span></span>
</span>
  await</span> summarizer</span>(</span></span>
    `profile:${</span>userId</span>}`</span>,</span></span>
    `Extract durable facts about the user. One fact per line as 'category: value'.</span></span>
Categories: name, role, company, tools, preferences, current projects.</span></span>
Only include facts explicitly stated. Omit unknown categories.</span></span>
When new info contradicts old, keep only the latest.`</span>,</span></span>
  ).</span>writeNow</span>({ history: cria.</span>user</span>(</span>`Session summary:</span>\n</span>${</span>summary</span>}`</span>) });</span></span>
}</span></span></code></pre>
Two summarizers, two keys, two different prompts. The episodic summarizer produces a structured session log. The profile summarizer extracts durable facts and stores them as a flat list. After the first session (deploying to Vercel), the stored profile contains:</p>
tools: GitHub, Vercel, HashiCorp Vault, AWS Secrets Manager</span></span>
preferences: Staging database for preview deployments</span></span>
current projects: Deploying a Next.js app to Vercel</span></span></code></pre>
The user comes back the next day and debugs a React performance issue. After that session ends, the profile has grown:</p>
tools: Next.js, Vercel, GitHub, React</span></span>
preferences: Staging database for preview deployments</span></span>
current projects: Deploying a Next.js app to Vercel, Fixing re-rendering in a React list component</span></span></code></pre>
Same store, same key. The summarizer rolled the new session’s facts into the existing profile. current projects</code> expanded. tools</code> updated. When the user switches from one tool to another, the “contradicts old, keep only the latest” instruction means the profile stays current automatically.</p>
And in prompt</code>, we add the profile at the top:</p>
async</span> prompt</span>(userId: string, messages: SessionMessage[]) {</span></span>
  const</span> userProfile</span> = await</span> summarizer</span>(</span>`profile:${</span>userId</span>}`</span>).</span>get</span>();</span></span>
  const</span> previousSession</span> = await</span> summarizer</span>(</span>`episodic:${</span>userId</span>}`</span>).</span>get</span>();</span></span>
</span>
  return</span> cria.</span>prompt</span>(provider)</span></span>
    .</span>system</span>(</span>`## What You Know About The User</span>\n</span>${</span>userProfile</span>}`</span>)</span></span>
    .</span>system</span>(</span>`## Previous Session</span>\n</span>${</span>previousSession</span>}`</span>)</span></span>
    .</span>use</span>(</span></span>
      summarizer</span>(</span>`history:${</span>userId</span>}`</span>).</span>plugin</span>({</span></span>
        history:</span> toScope</span>(messages),</span></span>
      })</span></span>
    );</span></span>
}</span></span></code></pre>
The prompt is growing. First, the user profile (a structured fact list), then the last session summary (what happened last time), then the current conversation. The model hasn’t seen a single new message yet and already knows the user’s tools, preferences, and current projects.</p>
Layer 3: we talked about this</h2>
The profile tells the model who this person is. The episodic summary tells it what happened last time. But neither helps with “remember that restaurant you mentioned last week?” The profile doesn’t track individual recommendations, and old session summaries aren’t searchable by topic.</p>
Vector search fills that gap. The idea: turn each session summary into an embedding (a list of numbers that captures its meaning) and store it. Later, when the user asks a question, embed that question too and find the past sessions whose embeddings are most similar. “What restaurant?” matches the session where restaurants were discussed, even if the word “restaurant” never appeared in the summary itself.</p>
We set up a vector store backed by the same SQLite database, with OpenAI’s embedding model to convert text into vectors:</p>
const</span> vectorStore</span> = new</span> SqliteVectorStore</span><</span>string</span>>({</span></span>
  filename:</span> "./data/cria.sqlite"</span>,</span></span>
  tableName:</span> "cria_vectors"</span>,</span></span>
  dimensions:</span> 1536</span>,</span></span>
  embed</span>:</span> async</span> (</span>text</span>)</span> =></span> {</span></span>
    const</span> {</span> embedding</span> }</span> = await</span> embed</span>({</span></span>
      model: openai.</span>embedding</span>(</span>"text-embedding-3-small"</span>),</span></span>
      value: text,</span></span>
    });</span></span>
    return</span> embedding;</span></span>
  },</span></span>
  schema: z.</span>string</span>(),</span></span>
});</span></span>
</span>
const</span> vectorDB</span> =</span> cria.</span>vectordb</span>(vectorStore);</span></span></code></pre>
One more addition to save</code>. After updating the profile, we index the session summary so it’s searchable by similarity:</p>
async</span> save</span>(userId: string, messages: SessionMessage[]) {</span></span>
  const</span> date</span> = new</span> Date</span>().</span>toISOString</span>().</span>split</span>(</span>"T"</span>)[</span>0</span>];</span></span>
  const</span> summary</span> = await</span> summarizer</span>(</span></span>
    `episodic:${</span>userId</span>}`</span>,</span></span>
    `Summarize this conversation into a structured session log.</span></span>
</span>
Date: ${</span>date</span>}</span></span>
Objective: What the user was trying to do.</span></span>
Timeline: Bullet points of what happened, key decisions and discoveries.</span></span>
Outcome: What got done, what's unresolved.</span></span>
</span>
Keep it concise. No markdown formatting. One line per bullet.`</span>,</span></span>
  ).</span>writeNow</span>({ history:</span> toScope</span>(messages) });</span></span>
</span>
  await</span> summarizer</span>(</span></span>
    `profile:${</span>userId</span>}`</span>,</span></span>
    `Extract durable facts about the user. One fact per line as 'category: value'.</span></span>
Categories: name, role, company, tools, preferences, current projects.</span></span>
Only include facts explicitly stated. Omit unknown categories.</span></span>
When new info contradicts old, keep only the latest.`</span>,</span></span>
  ).</span>writeNow</span>({ history: cria.</span>user</span>(</span>`Session summary:</span>\n</span>${</span>summary</span>}`</span>) });</span></span>
</span>
  await</span> vectorDB.</span>index</span>({</span></span>
    id:</span> `session:${</span>userId</span>}:${</span>date</span>}:${</span>Date</span>.</span>now</span>()}`</span>,</span></span>
    data: summary,</span></span>
  });</span></span>
}</span></span></code></pre>
That’s the complete save</code>. Three writes when the session ends: distill the conversation into a session log, update the user profile, index for search. The summary already carries its own date and timeline, so the vector index gets rich, structured content to embed. All three happen at the same moment because that’s when the context is complete: The user has signaled “we’re done” through inactivity or clicking save.</p>
Now the final version of prompt</code>. We add vector search using the user’s latest message as the query. And since vector search can surface any relevant past session by similarity, we no longer need the explicit previous session summary. If the last session is relevant to what the user is asking about, it’ll show up in the results.</p>
async</span> prompt</span>(userId: string, messages: SessionMessage[]) {</span></span>
  const</span> userProfile</span> = await</span> summarizer</span>(</span>`profile:${</span>userId</span>}`</span>).</span>get</span>();</span></span>
  const</span> latestQuestion</span> =</span> messages.</span>findLast</span>((</span>m</span>)</span> =></span> m.role</span> ===</span> "user"</span>)?.content</span> ??</span> ""</span>;</span></span>
</span>
  return</span> cria.</span>prompt</span>(provider)</span></span>
    .</span>system</span>(</span>`## What You Know About The User</span>\n</span>${</span>userProfile</span>}`</span>)</span></span>
    .</span>use</span>(vectorDB.</span>plugin</span>({ query: latestQuestion, limit:</span> 4</span> }))</span></span>
    .</span>use</span>(</span></span>
      summarizer</span>(</span>`history:${</span>userId</span>}`</span>).</span>plugin</span>({</span></span>
        history:</span> toScope</span>(messages),</span></span>
      })</span></span>
    );</span></span>
}</span></span></code></pre>
That’s the complete prompt function we’ve been building toward. Three blocks: who this person is, what past sessions are relevant, and the current conversation. When the user asks “what restaurant did you recommend?”, the vector plugin finds the session where food came up, even if it was weeks ago. No explicit tagging, no categories to maintain.</p>
One caveat: Embedding full session summaries works well when sessions are focused on a single topic. It gets weaker when a session covers multiple unrelated topics because the embedding becomes a “semantic average” that matches none of them precisely. Production systems handle this by indexing at topic-level granularity rather than session-level, or by combining semantic search with keyword search to catch what embeddings miss. For a starting point, per-session indexing gets you surprisingly far.</p>
And when the token budget fills up, Cria drops layers in a predictable order:</p>
What drops first</th> What happens</th></tr></thead>

Vector recall</td> “We discussed this last week” stops working</td></tr>
Conversation history</td> Older turns get progressively summarized</td></tr>
User profile</td> Minor friction, user re-explains preferences</td></tr>
</tbody></table>
Everything degrades gracefully. For more on prompt layout</a> and priority-based degradation, see the LLM memory guide</a>.</p>
Wiring it up</h2>
Here’s the full chat route. Compare it to our starting point: The only addition is composing Memory.prompt()</code> into the request.</p>
// app/api/chat/route.ts</span></span>
import</span> { Memory, model, provider }</span> from</span> "@/lib/memory"</span>;</span></span>
</span>
export async function</span> POST</span>(</span>req</span>:</span> Request</span>) {</span></span>
  const</span> {</span> messages</span>,</span> userId</span> }</span> = await</span> req.</span>json</span>();</span></span>
</span>
  const</span> memoryPrompt</span> = await</span> Memory.</span>prompt</span>(userId, messages);</span></span>
</span>
  const</span> rendered</span> = await</span> cria</span></span>
    .</span>prompt</span>(provider)</span></span>
    .</span>system</span>(</span>SYSTEM_PROMPT</span>)</span></span>
    .</span>use</span>(memoryPrompt)</span></span>
    .</span>render</span>({ budget:</span> 40_000</span> });</span></span>
</span>
  return</span> streamText</span>({ model, messages: rendered }).</span>toDataStreamResponse</span>();</span></span>
}</span></span></code></pre>
The app owns the system prompt. Memory is a plugin that slots in via .use()</code>. render({ budget: 40_000 })</code> is where Cria figures out what fits and what gets dropped. The save endpoint is equally simple:</p>
// app/api/save/route.ts</span></span>
export async function</span> POST</span>(</span>req</span>:</span> Request</span>) {</span></span>
  const</span> {</span> userId</span>,</span> messages</span> }</span> = await</span> req.</span>json</span>();</span></span>
  await</span> Memory.</span>save</span>(userId, messages);</span></span>
  return</span> Response.</span>json</span>({ ok:</span> true</span> });</span></span>
}</span></span></code></pre>The result</h2>
The user introduces themselves. They click “Save session.” The system writes episodic, semantic, and vector memories. New session, empty chat. “What do you remember about me?”</p>

The assistant knows their name, location, what they’re working on, and what they wanted help with. Three memory layers, composed into a single prompt builder. The user never re-introduced themselves.</p>
Where to go from here</h2>
What we built is a working memory system, not a finished one. Three directions to explore:</p>
Iterate on the prompts with real conversations.</em> The prompts we used are a starting point. Your product’s conversations will be different. Run a handful of real sessions through the summarizer, read the output, adjust. The shape of what gets stored is entirely determined by the prompt.</p>
Consider atomic facts and knowledge graphs.</em> Our profile summarizer rolls new facts into an existing summary, which works but is lossy. Mem0</a> extracts individual facts and diffs them (add, update, delete) so nothing gets silently dropped. Zep</a> goes further and builds a knowledge graph from conversations, capturing entities and their relationships over time. That precision matters when your product relies on remembering specific details.</p>
Let your product decide what to remember.</em> A coding assistant needs to remember file paths and error patterns. A health app needs to remember medications and allergies. A tutoring app needs to remember what the student already understands. The memory layers are the same. The prompts and what you index are completely different. Start with what your users complain about forgetting.</p>
For the theory behind these layers, see our guide to LLM memory</a> and memory systems explained</a>. And if you want to see where this composable approach goes at scale, read OpenAI’s writeup on their in-house data agent</a>. They use six context layers with the same fundamental pattern: different prompts producing different memory shapes, all composed into a single context window. The fact that this architecture works for an internal tool processing millions of queries is a strong signal that the building blocks we used here are the right ones.</p>

Full code on GitHub</a>. Built with Cria</a>.</p>


Ultimate Guide to LLM Memory
Sebastian Lund — Fri, 16 Jan 2026 00:00:00 +0000
Most LLM memory systems make your product worse.</p>
Engineers add them expecting a database. Instead they get something slow, expensive, and unreliable. Mention memory tools to anyone running agents in production and you get the same reaction: “It’s heavy.” “The latency kills us.” “Great in theory.”</em></p>
The problem isn’t the tools. It’s that there is no universal LLM memory</a>. The industry uses several methods</a> to fake it, and each solves a different problem. Pick the wrong one and you pay for capability you don’t need while missing what you do.</p>
This guide covers the patterns, how to match them to your use case, and how to swap them out when something better drops.</p>

Memory ≠ Database</h2>
Before we get into the details, we need to change your mental model about what “memory” means in the world of AI and LLMs.</p>
Most of us adding memory to agents aren’t LLM experts. We build something with an LLM and want it to remember things between sessions. We search online, find some cool “memory system”, slap it on top, only to discover it makes our product slow and confusing. And it barely remembers anything useful.</p>
When we think of memory in software, we think about databases: we “store things somewhere” and “fetch them later”. Modern databases are robust and deterministic. Fifty years of software engineering has gone into making sure they don’t fail in production. Databases were built to store and retrieve data for which you knew the exact shape, the exact constraints, and wrote the code yourself. They are deterministic by design.</p>
LLMs operate under different constraints than the code you manually write to call a database:</p>

Humans are unpredictable. LLMs are unpredictable. You’re interfacing between two unpredictable systems in a protocol that allows virtually anything (text).</p>
</blockquote>

The type of data you deal with is “random text that contains useful stuff”, but you rarely know what’s useful up-front.</li>
You have an upper limit on the amount of data you can include and shove into an LLM until it becomes slow, costly, and inaccurate (confused).</li>
LLMs are by their very nature non-deterministic. Same input can yield different results.</li>
</ol>
Every memory system is picking a point on this trade-off triangle:</p>
</p>
These constraints mean you need to work around</em> the non-determinism, the limitations of the LLM, and the nature of the input data.</p>
Say you’re building a voice assistant for a hospital and want to store useful data for diagnostics. You cannot do so deterministically. Imagine if you saved data every time a user mentioned “feeling unwell”. You’d quickly find your entire database filled with useless data from patients discussing other people rather than themselves because they’re lonely.</p>
Unpredictable systems are like async/await. Once you introduce it, you can’t escape it. You can only build around it. Memory systems are inherently unpredictable</em> because of the environment they’re deployed and built within. Databases are predictable</em> by design. They are not the same.</p>
What Is “LLM Memory”?</h2>
Because LLMs and humans are both unpredictable, the industry is building memory in layers: working memory, episodic memory, semantic memory, and document memory. Each serves a different purpose and solves a different problem.</p>
A useful mental model:</p>

Working: what I’m thinking</em> right now</li>
Episodic: what I experienced</em> (concrete events)</li>
Semantic: what I know</em> (facts from experience)</li>
Document: what I can look up</em> (external reference)</li>
</ul>
Working Memory</h3>
Working memory keeps the LLM on track</em> with what it’s currently doing. Think of it like a scratch pad the LLM needs to update continuously to track what it’s doing, what it has tried, and what it’s trying to achieve.</p>
Working memory is usually localized to the immediate context of the LLM and is never truncated or messed with. Simple versions include keeping the last 100 messages, storing data in a file and updating continuously, or maintaining a todo list.</p>
LLMs usually take multiple turns to complete a task (making them agents), yet they’re stateless. Working memory helps them continue with a task they started in a previous turn.</p>
LLMs have amnesia (much like Dory in Finding Nemo) and write things down in their working memory to remember next time. It’s the log</em> of what they’re thinking about, what they’re planning to do, etc.</p>
Working memory is a “structured context window” for your LLM.</p>
Episodic Memory</h3>
Working memory tracks what’s happening now. But what about yesterday? Last week? Episodic memory gives your LLM a timeline of past events it can reference.</p>
Think of it as a structured history log. What happened, when, and in what order. “Last Tuesday you asked me to draft that email.” “Earlier in this conversation you mentioned your deadline.” Episodic memory makes these references possible.</p>
Episodic memory works on selective inclusion: you decide what events to remember, how to log them, what details matter. An additional agent or LLM typically handles extraction, deciding what’s worth persisting for future reference.</p>
Episodic memory is usually called “summarization”, and you may see that term used instead. It’s a technique for how to achieve episodic memory in its most simple form, where you “summarise” past messages into a log of useful episodic events. Read more about it here</a>.</p>
</p>
Semantic Memory</h3>
Users interacting across multiple sessions often feed the same data repeatedly into the LLM’s episodic and working memory. If I’m diabetic, I probably have to tell the LLM that every session. How else would it know?</p>
Semantic memory extracts important facts from episodic and working memory and persists them for future sessions. It’s a way to remember user preferences and personalize experiences. Read more about it here</a>.</p>
Semantic memory, like episodic memory, works on selective inclusion. You use another LLM or agent to extract entities, facts, and relationships. Whatever is interesting to remember. These are then selectively persisted</em> for future turns or sessions.</p>
</p>
Document Memory</h3>
Working, episodic, and semantic memory deal with information from the conversation or the user. Document memory is different: It’s reference material. Your knowledge base, your docs, your internal wikis.</p>
The industry calls this RAG (Retrieval-Augmented Generation). You store documents in a searchable index, usually as vector embeddings. When the user sends a message, you search for relevant chunks and inject them into the prompt before calling the LLM. The LLM doesn’t know this information and doesn’t look it up. You do, and you enrich the prompt with what you find.</p>
Say you’re building a customer support bot. The LLM has no idea what your pricing tiers are or how your refund policy works. You index your help docs. When a user asks “how do I get a refund?”, you retrieve the relevant policy and hand it to the LLM. Now it can answer accurately.</p>
When to Use What</h2>
Memory is additive. You start with one layer, need more capability, add another. Each layer solves a different problem. Each can be swapped independently as better solutions emerge.</p>
Most apps start with working memory alone. You add layers as you need new capabilities:</p>
You want to…</th> Add…</th> Now you can…</th></tr></thead>

Have multi-turn conversations</td> Working</td> Track what’s happening right now</td></tr>
Reference things from earlier</td> + Episodic</td> “You mentioned this earlier” works</td></tr>
Remember users across sessions</td> + Semantic</td> Personalize without re-explaining</td></tr>
Answer from your own docs</td> + Document</td> Cite internal knowledge accurately</td></tr>
</tbody></table>
This is a progression, not a menu. You stack layers. A production chat assistant might use all four: working memory for the current turn, episodic summaries for older context, semantic facts for personalization, and document retrieval for domain knowledge.</p>
You can also stack multiples of the same type. Two document stores: one for user uploads, one for your knowledge base. Two episodic memories: one for the current session, one for cross-session history. The layers compose however your use case demands.</p>
Autonomous agents typically need all four</strong>, each tuned differently. Working memory tracks execution state: what step am I on, what have I tried, what’s next. Episodic memory logs what the agent learned across tasks. Semantic memory stores extracted facts about the environment and user. Document memory provides reference material for decision-making. Agents are more demanding than chat assistants because they execute multi-step plans where each layer plays a distinct role.</p>
Latency and Cost</h3>
Every layer adds latency and cost. Working memory scales with context length. Episodic and semantic memory add LLM calls for summarization and extraction. Document memory adds vector search plus the tokens you inject.</p>
We benchmarked this elsewhere</a>. The tradeoffs are real. A voice assistant can’t wait two seconds for semantic memory extraction. A batch agent processing documents overnight doesn’t care.</p>
Failure Semantics</h3>
Remember: LLMs and humans are both unpredictable. Your memory system will fail. The question isn’t if</em>, it’s how</em>. We’ve written about this elsewhere</a>, but here’s the short version:</p>
Layer</th> When it fails…</th> Product impact</th></tr></thead>

Working</strong></td> The LLM forgets what it’s doing mid-task.</td> Task fails. User starts over.</td></tr>
Episodic</strong></td> “We discussed this earlier” stops working.</td> User notices, re-explains.</td></tr>
Semantic</strong></td> The LLM forgets who you are between sessions.</td> Minor friction. User re-explains.</td></tr>
Document</strong></td> The LLM hallucinates instead of citing your docs.</td> User gets wrong answer. Trust erodes.</td></tr>
</tbody></table>
Not all failures are equal. Semantic and episodic failures degrade the experience. Users notice, get annoyed, re-explain themselves. Frustrating, but recoverable.</p>
Working and document failures are different. A task that fails midway wastes user time. A hallucinated answer damages trust, or worse, causes real harm. For these layers, it’s better to error explicitly than to silently produce garbage. If your document retrieval returns nothing relevant, say so. If working memory is corrupted, stop the task. Crashing beats confident hallucination.</p>
Common Mistakes</h2>
These patterns hurt more products than they help:</p>
Using the wrong layer for execution state.</strong> Tools like Mem0 and Zep are built for personalization. They extract facts, build user profiles, remember preferences. Great for “remember I prefer dark mode”. Catastrophic for “remember what step I’m on in this 12-step deployment”.</a> If your agent loses track mid-task, you’re using the wrong memory type.</p>
Adopting a generic “memory solution”.</strong> Off-the-shelf memory tools make decisions for you. What to remember, what to forget, how to retrieve. These decisions should be yours. Your chat assistant and your autonomous agent have different needs. A tool that works for one will produce weird failures</a> in the other.</p>
Adding layers before you need them.</strong> Working memory is powerful on its own. Modern context windows are large, and your LLM can do a lot with just a well-structured prompt. Start there. Add episodic when users need to reference past conversations. Add semantic when personalization matters. Add document when you have a knowledge base worth querying. Each layer adds complexity. Earn that complexity by needing the capability.</p>
Forgetting that layers are swappable.</strong> The point of composability isn’t architectural elegance. It’s survival. The best summarization technique today won’t be the best in six months. The RAG approach you choose now will be obsolete. Build so you can rip out any layer and replace it without rewriting everything else.</p>
How to Add Memory</h2>
Memory systems aren’t mutually exclusive. You compose them.</p>
A chat assistant might need all four types working together:</p>

System prompt + last 100 messages (working)</li>
Summary of older conversation (episodic)</li>
Extracted user facts (semantic)</li>
RAG results from your knowledge base (document)</li>
</ul>
The trap is reaching for an off-the-shelf solution that promises to handle “memory” generically. These produce weird failures</a> because they make decisions for you that should be yours to make.</p>
The better approach: treat your prompt like code. Each memory type is a component. Components can be swapped, removed, or upgraded independently. When a better summarization technique drops, you swap that piece. When your needs change, you rip out what you don’t need. Your prompt structure stays stable while the pieces evolve.</p>
The examples below use Cria</a>, a prompt composition library I built for exactly this problem. The concepts apply regardless of what tools you use.</em></p>
Prompt Layout</h3>
Every piece of memory you pull in competes for space. System instructions, retrieved documents, conversation summaries, user facts, recent messages, the current query. They all need to fit. When you query an LLM, you’re assembling a prompt from these pieces. The layout matters.</p>
LLMs weigh information two ways, and understanding both shapes how you structure prompts.</p>
By position.</strong> LLMs focus on the beginning and end of a prompt. Middle sections fade into background noise. Place stable context (system instructions, user facts) early. Place dynamic context (recent messages, the current query) at the end where it captures attention.</p>
By proportion.</strong> Whichever memory type dominates your prompt gets more of the model’s focus. The needle-in-a-haystack problem works in reverse: a bloated haystack doesn’t just hide needles, it captures attention that should go elsewhere. Pull in 1000 RAG chunks and the model drowns in them, even if five matter. Keep each memory system constrained so none overwhelms the rest.</p>
</p>
Think of your prompt as a layout with a fixed ceiling. Each model has a context window, a hard upper limit on tokens. You know this number for whatever model you use. But filling to the edge is a mistake. Performance degrades well before you hit the limit, sometimes dramatically. The goal is maximizing useful content while staying comfortably below the threshold where quality drops.</p>
Assign each memory system a region. RAG results might get 2000 tokens. Conversation summaries get 1000. User facts get 500. Recent messages get 1500. These numbers are illustrative. Your actual budgets depend on your model’s context window and your use case. The point is giving each memory type a ceiling. System instructions and the current query are sacred, always included. These constraints keep one memory type from eating the haystack.</p>
cria</span></span>
  .</span>prompt</span>()</span></span>
  // working memory: what are we doing?</span></span>
  .</span>system</span>(instructions)</span></span>
  // document memory</span></span>
  .</span>vectorSearch</span>({ store, query, limit:</span> 5</span> })</span></span>
  // semantic memory in another store (tech is used across memory types)</span></span>
  .</span>vectorSearch</span>({ store: factStore, query, limit:</span> 15</span> })</span></span>
  // episodic memory</span></span>
  .</span>summary</span>(messages, { id:</span> "conversation-summary"</span>, store: summaryStore })</span></span>
  // working memory: the last messages for relevance</span></span>
  // which also captures tool calls</span></span>
  .</span>last</span>(messages, { n:</span> 100</span> });</span></span></code></pre>Composing for Different Use Cases</h3>
Different applications need different memory compositions. A chat assistant, an autonomous agent, and a support bot all use the same building blocks but weight them differently.</p>
Chat assistant</strong>: Personalization matters. Semantic memory (user facts) and episodic memory (conversation summaries) take up most of the layout. Document memory is secondary. You might not need RAG at all if the assistant is general-purpose.</p>
cria</span></span>
  .</span>prompt</span>()</span></span>
  .</span>system</span>(assistantInstructions)</span></span>
  // semantic memory dominates</span></span>
  .</span>assistant</span>(userFacts)</span></span>
  .</span>summary</span>(messages, { id:</span> "assistant-summary"</span>, store })</span></span>
  // working memory</span></span>
  .</span>last</span>(messages, { n:</span> 20</span> });</span></span></code></pre>
Autonomous agent</strong>: Execution state is critical. Working memory dominates. The agent needs to track what step it’s on, what it’s tried, what failed. Episodic memory logs task history for learning. Document memory provides reference material for decisions.</p>
cria</span></span>
  .</span>prompt</span>()</span></span>
  .</span>system</span>(agentInstructions)</span></span>
  // working memory dominates</span></span>
  .</span>assistant</span>(currentPlan)</span></span>
  .</span>assistant</span>(executionLog)</span></span>
  // document memory for reference</span></span>
  .</span>vectorSearch</span>({ store: knowledgeBase, query: currentTask, limit:</span> 5</span> })</span></span>
  // need more messages in working memory to</span></span>
  // do longer running work continuously</span></span>
  .</span>last</span>(messages, { n:</span> 200</span> });</span></span></code></pre>
Support bot</strong>: Document memory dominates. The bot needs to cite your help docs, knowledge base, product information. Semantic memory (customer info) helps personalize. Episodic memory matters less since support conversations are usually single-session.</p>
cria</span></span>
  .</span>prompt</span>()</span></span>
  .</span>system</span>(supportInstructions)</span></span>
  // document memory dominates</span></span>
  .</span>vectorSearch</span>({ store: helpDocs, query: customerQuestion, limit:</span> 10</span> })</span></span>
  // semantic memory for personalization</span></span>
  .</span>assistant</span>(customerRecord)</span></span>
  // working memory, don't need that many messages since it's likely</span></span>
  // to be multiple sessions over time that episodic memory captures</span></span>
  .</span>last</span>(messages, { n:</span> 25</span> });</span></span></code></pre>
The composition pattern stays the same. What changes is which layers you include and how much of the layout you give them.</p>
Handling Conflicts</h3>
Memory layers contradict each other. Your semantic memory says “user prefers dark mode”. Your episodic summary says “user switched to light mode yesterday”. What wins?</p>
LLMs handle this well if you capture when</em> something was true. “Preferred dark mode (January 2024)” vs “switched to light mode (December 2025)” gives the model enough to reason about recency. Make sure your memory system records timestamps.</p>
Off-the-shelf tools like Zep</a> and Mem0 handle reconciliation automatically, using an LLM to merge or update facts as they come in. If your use case depends on facts staying accurate over time, pick a memory system that supports this.</p>
Vector search is trickier. RAG can fill your budget with contradictory facts, and you can’t tell which are current. Episodic memory helps because it preserves temporal order. A summary beats a bag of conflicting chunks.</p>
A Complete Example</h3>
Here’s a chat assistant with all four memory types, priority-based fitting, and swappable stores:</p>
const</span> summaryStore</span> = new</span> PostgresStore</span>({ tableName:</span> "summaries"</span> });</span></span>
const</span> vectorStore</span> = new</span> QdrantStore</span>({</span></span>
  client: qdrantClient,</span></span>
  collectionName:</span> "knowledge"</span>,</span></span>
  embed: embedFunction,</span></span>
});</span></span>
</span>
const</span> prompt</span> =</span> cria</span></span>
  .</span>prompt</span>()</span></span>
  .</span>system</span>(</span>"You are a helpful assistant."</span>)</span></span>
  // Document memory: RAG results</span></span>
  .</span>vectorSearch</span>({</span></span>
    store: vectorStore,</span></span>
    query: userMessage,</span></span>
    limit:</span> 5</span>,</span></span>
    priority:</span> 2</span> /* we can live without them */</span>,</span></span>
  })</span></span>
  // Episodic memory: auto-updating summary</span></span>
  .</span>summary</span>(conversation, { id:</span> "conversation-summary"</span>, store: summaryStore, priority:</span> 1</span> })</span></span>
  // Semantic memory: user facts</span></span>
  .</span>omit</span>(</span></span>
    cria.</span>prompt</span>().</span>assistant</span>(</span>await</span> fetchUserFacts</span>(userId)),</span></span>
    { priority:</span> 2</span> /* we can live without them */</span>, id:</span> "user-facts"</span> }</span></span>
  )</span></span>
  // Working memory: recent messages</span></span>
  .</span>last</span>(conversation, { n:</span> 200</span> });</span></span>
</span>
const</span> messages</span> = await</span> prompt.</span>render</span>({</span></span>
  provider: openai,</span></span>
  // cap at 100k because we see accuracy decrease beyond that</span></span>
  budget:</span> 100_000</span>,</span></span>
});</span></span></code></pre>
This example uses Cria</a>, a prompt composition library we’ve been building. If we hit the budget, Cria shrinks what we can live without (semantic and document memory) while keeping the rest intact.</p>
The pattern matters more than the tool. Structure your prompts so you can adapt as the space evolves.</p>
What About Research on Memory for LLMs?</h2>
The future looks promising. Real breakthroughs are happening. At some point “LLM + memory” will become a non-issue. We won’t get there without stepwise improvements in LLM infrastructure, but we’re moving in the right direction.</p>
The pattern worth watching: systems we build around</em> the LLM are slowly getting absorbed into</em> the LLM. This is good news. It means we can pick winning patterns now, knowing the best ones will eventually become native capabilities.</p>
As of early 2026, here’s how recent developments map to the four layers:</p>

DeepSeek Engram</a></strong>: a conditional memory module that adds constant-time (O(1)) lookup into massive N-gram embedding tables, fused into the model via gating. Deterministic addressing enables prefetch and host-memory offload with minimal inference overhead. This is document memory baked into the model.</strong> Similar in spirit to vector search, but without the retrieval latency.</li>
Recursive Language Models (RLMs)</a></strong>: an inference harness that treats long prompts as external environment state (e.g., a Python REPL variable), letting the model inspect it in code and recursively call itself over snippets. This extends working memory.</strong> The agent harnesses we build today, formalized as an inference pattern.</li>
Claude chat search and memory</a></strong>: Claude can search past conversations across sessions and maintain memory summaries across chats and projects. This is episodic and semantic memory as a product feature.</strong> Simple, but often wins in real workflows because it works.</li>
</ul>
These aren’t true memory yet. They don’t give LLMs the kind of episodic recall or semantic persistence we’ve described as fully general capabilities. But they’re absorbing specific layers: Engram eliminates document retrieval latency. RLMs formalize working memory patterns. Claude’s memory handles episodic and semantic for consumer use cases. The layers we build today are scaffolding. Some will become unnecessary as models absorb them. Others will evolve into something better.</p>

The goal isn’t finding the perfect memory system. It’s building something you can evolve.</p>
This space moves fast. The state of the art today won’t be state of the art in three months. Whatever you build needs to adapt, or you’ll be ripping it out and starting over.</p>
That’s the real argument for composability. Don’t settle for whatever’s kinda working today. Build a structure where you can swap out any layer when something better drops. Add capabilities when you actually need them. Evolve with the space instead of getting locked into last month’s best practice.</p>
We’re building Cria</a> to ensure you can swap out whatever you need in the future. If you’re solving similar problems, check it out.</p>


Design Your LLM Memory Around How It Fails
Sebastian Lund — Fri, 05 Dec 2025 00:00:00 +0000
Thursday afternoon. Your security team pings you:</p>

Jon @blue-team: The Next.js RCE vulnerability just dropped. What version were we running on Monday? Were we exposed? Do we need to check logs for exploitation attempts?</p>
</blockquote>

You: Wait what? What versions are affected?</p>
</blockquote>

Jon @blue-team: 15.0.5, 15.1.9, 15.2.6, 15.3.6, 15.4.8, 15.5.7, 15.6.0-canary.58, 16.0.7 have been patched and are safe.</p>
</blockquote>
You ask your dependency auditing agent. It doesn’t know.</p>
</p>
On Monday, it made several tool calls and analyzed 2,400 packages with version numbers (~60k tokens). The memory system summarized the results and discarded most version numbers.</p>
Dependency audit complete: 2,400 packages scanned across 12 workspaces.</span></span>
Key frameworks: Next.js, React, Prisma. 2 moderate vulnerabilities found in dev dependencies.</span></span>
Versions: {"prisma": "6.1.1", "react": "19.2.1", ... }</span></span></code></pre>
Next.js was not in the versions list. The one fact that matters (what version were we running on Monday?) is gone. The agent trusts its own memory. It doesn’t check the logs.</p>
The agent can fabricate a version, admit ignorance, or tell you “Next.js wasn’t flagged, you should be good”. Option three is the disaster. The agent was never supposed to be the source of truth. But it answered confidently, and confidence is contagious. You don’t check the logs. Meanwhile, attackers may have had a window you’ll never know about.</p>
The CVE</a> is real. The scenario is hypothetical. But silent false negatives from lost context</strong> happen constantly.</p>
The Problem: All Context Gets Treated Equally</h2>
Most LLM memory systems work the same way: append everything to a chronological buffer, run some RAG (and pray), and hope the right tokens survive when you hit the limit.</p>
Those version numbers needed to be sacred</strong> or discarded completely. Instead, they got the same treatment as idle chit-chat: summarized into oblivion.</p>
New to LLM memory? Check out LLM Memory Systems Explained</a> for background, or Memory Isn’t One Thing</a> for why generic solutions fail.</em></p>
Failure-First Design</h2>
When people talk about LLM memory, they start with architectures: vector vs. graph, RAG vs. long context, summarization vs. extraction.</p>
That’s the wrong starting point.</p>
How does your system fail when context is missing?</strong> That’s the first question.</p>
Different agents fail in different ways:</p>

A support chatbot</strong> tolerates fuzzy recall. Miss a minor detail, give a decent answer. No one notices.</li>
A vulnerability agent</strong> cannot miss the one critical version number in a pile of false positives. One false negative and the customer walks.</li>
An SRE agent</strong> cannot quietly ignore the only log line that explains why prod is on fire. Waste the on-call’s time once, and they rip it out of the incident channel.</li>
</ul>
Different failure modes require different memory shapes.</p>
</p>
Threat Modeling for Context</h2>
This is effectively threat modeling, but for attention spans instead of security boundaries.</p>
For every piece of data you want to put in the prompt, ask:</p>
Question</th> Policy</th></tr></thead>

Missing context produces dangerous</strong> output?</td> Sacred. Never drop.</td></tr>
Summarization causes false confidence</strong>?</td> Critical. Never compress.</td></tr>
Truncation causes noticeable</strong> quality loss?</td> Important. Compress last.</td></tr>
Removal still leaves useful</strong> output?</td> Expendable. Drop first.</td></tr>
</tbody></table>
Back to the dependency auditor:</p>
Context Type</th> Failure Mode</th> Policy</th></tr></thead>

Version numbers from tool output</td> Silent false negatives</td> Sacred. Never drop.</td></tr>
User’s prioritization decisions</td> Mild confusion on follow-up</td> Important. Compress last.</td></tr>
Reasoning steps</td> None (can be reconstructed)</td> Expendable. Drop first.</td></tr>
</tbody></table>
This exercise doesn’t tell you what</em> to mark sacred. You can’t know a version number matters until a CVE drops. But it forces you to decide what happens when you’re wrong, before you’re wrong.</p>
Once you’ve done this, the architecture follows naturally. Sacred content stays in fixed context, not semantic retrieval. Expendable content can live in RAG and get evicted freely. The hard part isn’t choosing between vector, graph, or long-context. It’s knowing what you can’t afford to lose.</p>
Making Trade-offs Explicit</h2>
Once you know what’s sacred and what’s expendable, enforce it. Several companies building serious LLM applications have already solved this, each in their own way.</p>
One useful mental model comes from the Cursor team, who frame context management as a layout problem</strong>, not a storage problem (Lex Fridman interview</a>). Instead of a flat buffer where everything competes equally, imagine regions with different eviction policies:</p>
cria</span></span>
  .</span>prompt</span>()</span></span>
  // Sacred: never drop</span></span>
  .</span>system</span>(systemPrompt)</span></span>
  .</span>assistant</span>(toolOutputs)</span>  // Version numbers live here</span></span>
</span>
  // High priority: compress last</span></span>
  .</span>last</span>(messages, { n:</span> 50</span>, priority:</span> 1</span> })</span></span>
</span>
  // Low priority: drop if space is tight</span></span>
  .</span>omit</span>(</span></span>
    cria.</span>prompt</span>().</span>assistant</span>(retrievedFacts),</span></span>
    { priority:</span> 2</span>, id:</span> "retrieved-facts"</span> }</span></span>
  )</span></span>
</span>
  // Expendable: drop first</span></span>
  .</span>summary</span>(olderHistory, { id:</span> "older-history"</span>, store, priority:</span> 3</span> });</span></span></code></pre>
When a user pastes a 20k-token stack trace, the layout engine knows what to sacrifice. Sacred regions don’t shrink. Low-priority regions absorb the pressure. If sacred regions alone exceed your budget? Probably better to crash than to silently drop something critical.</p>
The value isn’t the specific implementation. It’s that the trade-offs are explicit in the code. You can look at this and know exactly what gets dropped when space runs out. No implicit decisions buried in a summarization heuristic.</p>
You don’t need a layout engine to apply this. Separate prompt sections. Different summarization policies per source. A simple check that critical data hasn’t been truncated. The point is deciding explicitly, before your agent decides implicitly in production.</p>

Most memory systems are built around a question of capacity: How much can I fit?</em></p>
The better question is: What happens when I can’t fit everything?</strong></p>
If you don’t have an answer, your agent will improvise one. In production, under load, when it matters most.</p>


Universal LLM Memory Does Not Exist
Sebastian Lund — Fri, 21 Nov 2025 00:00:00 +0000
Over the last few weeks, I’ve been digging deep into LLM memory systems. My last post</a> described the techniques they use.</p>
Whenever I mentioned tools like Mem0</a> to engineers running agents in production, I got the same reaction: a collective sigh</em>.</p>
“It’s heavy.” “The latency kills us.” “Great in theory.”</p>
I wanted to understand why.</p>
Systems like Mem0</a> & Zep</a> are sold on the promise of reducing cost by 90%, and cutting latency drastically. Why is there such a disconnect with the reality of these systems vs. how they’re sold?</p>
I took two of the most hyped systems: Zep</strong> (Knowledge Graph) & Mem0</strong> (“Universal Memory”) and ran them against MemBench</a></strong>, a 2025 benchmark designed to test reflective memory and reasoning.</p>
I expected to find some trade-offs. What I found instead was a huge “WTF” moment</strong>.</p>
The Experiment</h2>
I didn’t want to run a standard “retrieval” benchmark. Marketing pages are full of those. I wanted to see the total cost of operation</strong> and latency impact.</p>
I set up a harness running gpt-5-nano</code> against 4,000 conversational cases from MemBench</a></strong>. The goal was simple: have a conversation, extract facts, and recall them later. It’s the same type of benchmark and workload these systems were built to manage, and should be great at.</p>
Here is what happened:</p>
Memory System</th> Precision</th> Avg. Input Tokens</th> Avg. Latency</th> Total Cost (4k cases)</th></tr></thead>

long-context (baseline)</td> 84.6%</td> 4,232</td> 7.8s</td> $1.98</td></tr>
Mem0 (vector)</td> 49.3%</td> 7,319</td> 154.5s</td> $24.88</td></tr>
Zep (graphiti)</td> 51.6%</td> 1.17M</strong>*</td> 224s</td> ~$152.6</strong></td></tr>
</tbody></table>
*Partial run (1,730/4,000 cases). Averaged 1,028 LLM calls per case, 2B input tokens total. Run aborted after 9 hours due to cost (usage alerts triggered in openai).</em></p>
You are reading that correctly. Zep burned 1.17 million tokens per test case.</strong></p>
At first, I thought my harness was broken. How can a simple conversation generate a million tokens of traffic? I dug into the logs, and what I found wasn’t a bug. It was the architecture working exactly as designed.</p>

You can run this experiment yourself using my open sourced tool I built to run benchmarks: pacabench</a>, check examples/membench_qa_test</code> for details on how this was run.</p>
</blockquote>
Architecture: “LLM-on-Write”</h2>
To understand why the latency & cost exploded, we have to look at how these systems work. They don’t just “save” data. They employ a pattern I call LLM-on-Write</strong>.</p>
They intercept every message and spin up background LLM processes to “extract” meaning.</p>
1. Mem0: N+1 summarization & extraction</h3>
Mem0 is based on two main components: facts and summaries. It also supports graphs but we’ll focus on vectors for now, as the graph functionality is disabled by default.</p>
It runs three parallel background LLM processes</strong> on every interaction:</p>

Update a conversation timeline to keep a narrative summary (with an LLM)</em></li>
Identify facts and save them to a vector store (with an LLM)</em></li>
Check for contradictions and update old or remove old facts (with an LLM)</em></li>
</ol>
</p>
For every message your agent sends, Mem0 spins up three separate inference jobs.</p>
2. Zep: Recursive Explosions of LLM calls</h3>
Zep’s Graphiti is a knowledge graph. When you say “I work on Project X “</em> it doesn’t just save the string. It wakes up an extractor LLM to:</p>

Identify the entity “User”.</li>
Identify the entity “Project X”.</li>
Create an edge “Works On”.</li>
</ol>
But it doesn’t stop there. It then traverses the graph to see if this new fact contradicts old facts (e.g., “User is already working on Project Y”). If it finds a conflict or a connection, it triggers another</strong> LLM call to resolve it.</p>
</p>
In my experiment, a single complex reasoning chain triggered a cascade of graph updates. The agent said one thing, which updated a node, which triggered a neighbor update, which triggered a re-summarization of the edge.</p>
The Common Flaw: Fact Extraction</h2>
Despite their differences (Graph vs. Vector), both systems share a fatal architectural flaw when applied to agent execution: Fact Extraction.</strong> They both rely on LLMs to “interpret” raw data into “facts”.</p>
</p>
This works for personalization (“User likes blue”</em> is a safe extraction). If you need a CRM on top of your chat assistant, it’s awesome. If you need to reduce cost and latency of an autonomous agent, it’s not.</p>
Compounding hallucinations by LLMs</h3>
The extractor LLM is non-deterministic. It might rewrite “I was ill last year”</em> as current_status: ill</code>. The error happens at write time</strong>, meaning the data is corrupted before it even hits the database. No amount of retrieval optimization can fix a database filled with hallucinations.</p>
Your primary LLM is at the mercy of the accuracy of the extractor LLM.</strong></p>
N+1 Latency & Cost Tax</h3>
Latency and cost compound as you add more LLM calls to your pipeline. For every message, you are triggering a chain of background inferences—extraction, summarization, graph updates.</p>

We are slapping LLMs on top of LLMs, introducing noise and latency at every layer, and paying a premium for the privilege.</p>
</blockquote>
</p>
Marketing hides the real costs</h2>
So why is everyone buying this?</p>
Because the marketing focuses on the wrong unit of measurement. Memory vendors advertise “Cost per Retrieval”</strong> They show you how cheap it is to fetch a small context window instead of reading the whole history.</p>
But as an engineer or founder, you pay “Cost per Conversation”</strong>.</p>
You pay for the N+1 extraction tax</strong>. You pay for the recursive graph updates. You pay for the debugging time when the system throws away your error logs.</p>
To be fair: Zep is more honest than most. They claim temporal knowledge graphs for personalization and business context, and that’s exactly what they build. The problem is that even for their stated use case (semantic memory), the cost is prohibitive at production scale. And despite this relative clarity, teams still use it for working memory tasks (agent execution state) it was never designed to handle.</p>
The marketing hype feeds into itself. We want “Universal Memory” to be real because it sounds amazing. We want an infinite context window that costs nothing. But the physics of the architecture don’t support it.</p>
Working Memory & Semantic Memory</h2>
</p>
The conclusion from my experiment is clear: Universal Memory does not exist.</strong></p>
We are trying to solve two fundamentally different problems with one tool.</p>

Semantic Memory</strong>, which is for the User</em>. It tracks preferences, long-term history, and rapport. It should</em> be fuzzy, extracted, and graph-based.</li>
Working Memory</strong>, which is for the Agent</em>. It tracks file paths, variable names, and immediate error logs. It must be lossless, temporal, and exact</strong>.</li>
</ul>
When you use a Semantic Memory tool (Zep, Mem0) for Working Memory tasks, you aren’t just making a tradeoff. You are using the wrong architecture. You are trying to run a database on a lossy compression algorithm.</p>
State is the application. You cannot compress your way to reliability.</strong></p>
Semantic memory is brilliant for personalization across sessions. It’s catastrophic for execution state within a task. Treat them as separate systems with separate requirements.</p>


LLM Memory Systems Explained
Sebastian Lund — Fri, 07 Nov 2025 00:00:00 +0000
LLMs don’t have memory. They’re stateless: each response requires resending the entire conversation history.</p>
Yet they reference earlier messages and maintain context across long interactions. How?</p>
LLMs do not remember anything</h2>
</p>
LLMs are stateless. Each inference is independent. Generating a single output token requires processing all preceding tokens as input.</p>
To answer coherently the LLM needs every previous message. This input is the “context window”.</p>
This creates an optimization problem with three competing constraints:</strong></p>

Latency</strong> (most critical): More tokens = slower inference. Users feel every millisecond.</li>
Cost</strong>: More input tokens = higher API costs. Scales linearly with conversation length.</li>
Accuracy</strong>: Finding the right amount of relevant context. Too little causes hallucinations, too much causes perceived hallucinations from noise.</li>
</ol>
</p>
You can’t optimize all three. Sending full history not only reduce accuracy after a certain point but also kills latency and cost. Aggressive compression helps latency and cost but degrades accuracy. Every memory system is picking a point on this trade-off triangle.</p>
Note: Internal caching (KV cache, KV pinning) reduces computation within the LLM, but doesn’t change the external interface. These help reduce latency & avoid reprocessing, and are not memory systems.</em></p>
How LLMs Remember</h2>
We build systems around their current limitation. Three key techniques:</p>

Summarization</em>: Compress old messages to reduce token count</li>
Fact tracking</em>: Extract and store durable information across conversations</li>
Retrieval systems</em>: Store and fetch relevant context on demand</li>
</ol>
These techniques combine to make LLMs appear to remember.</p>
Summarization</h3>
Token-level compression within the context window. Compress old messages to keep only relevant information, preventing the LLM from getting distracted by noise while preserving key events.</p>
</p>
A typical implementation uses a secondary LLM to maintain a running summary as new messages arrive.</p>
Approaches:</strong></p>

Periodic (every N messages)</li>
Sliding window (recent messages + summary of older ones)</li>
Hierarchical (summaries of summaries)</li>
</ul>
Trade-offs:</strong></p>

Reduces token cost significantly</li>
Lossy: important details can be dropped</li>
Adds summarization latency and cost</li>
Quality depends on the summarization model</li>
Can exceed context window limits for very long conversations</li>
</ul>
Fact Tracking Across Conversations</h3>
Extends beyond the current context window. Extract durable facts about users that persist beyond individual conversations. If a user mentions they hate cinnamon in one conversation, that fact is available in all future conversations.</p>
</p>
A typical implementation uses a secondary LLM to monitor conversations and extract facts worth persisting. Unlike summarization</em>, these facts live beyond the conversation scope.</p>
Common patterns:</strong></p>

Continuous extraction (after each message or periodically)</li>
Structured storage (key-value pairs or entity graphs)</li>
Conflict resolution (when facts contradict)</li>
</ul>
Trade-offs:</strong></p>

Personalized experience across conversations</li>
Requires fact extraction inference cost</li>
Needs storage and retrieval infrastructure</li>
Must handle contradictions and updates</li>
Privacy considerations for storing user data</li>
</ul>
Retrieval Systems</h3>
Retrieval means selective inclusion, not persistence. With summarization</em> and fact tracking</em>, we have stored memory, but we can’t send all of it to the LLM without adding noise. When the user asks about “cinnamon”, including “user does not like cinnamon” is useful, including “user has a dog named Oscar” is not.</p>
</p>
Retrieval systems solve this by fetching semantically relevant pieces of memory based on the current context.</p>
How it works:</strong></p>

Generate embeddings (vector representations) of stored information</li>
Store in a vector database</li>
On query: retrieve semantically similar pieces</li>
Inject retrieved context into the prompt</li>
</ol>
Trade-offs:</strong></p>

Adds retrieval latency before inference</li>
Quality depends on embedding model and retrieval strategy</li>
Can miss connections between non-adjacent information</li>
Scales beyond context window limits</li>
</ul>
Other retrieval methods exist (graph databases, keyword search), but vector-based retrieval is common and generalises well between use cases.</p>
Note: Retrieval systems are also widely used for accessing large corpora (RAG - Retrieval-Augmented Generation</a>). That’s a related but different use case. Here we’re focused on retrieving stored memories, not general document knowledge bases.</em></p>
What We Have Today</h2>
Combining these techniques lets us build applications with long conversations, personalization, and large knowledge bases. We can deliver reasonable improvements to user experience.</p>
Systems like Mem0</a> apply these techniques using vector stores for long-term memory. They demonstrate that production implementations are viable, though trade-offs remain.</p>
Every technique forces serious trade-offs. The latency-cost-accuracy triangle is real. There’s no perfect solution out there, and this is very much an unsolved problem.</p>
Open Problems</h2>
The techniques above are starting points, not long-term solutions. Real questions remain:</p>
Lossy compression</strong>: When do critical details get dropped? How do you know what to keep?</p>
Retrieval precision</strong>: Semantic search doesn’t always fetch the right data. How do we measure relevance accurately?</p>
Determinism vs. adaptivity</strong>: Deterministic systems are debuggable but inflexible. Adaptive systems are powerful but unpredictable. Which matters more?</p>
Evaluation</strong>: How do you test memory systems when conversations span thousands of turns and context is deeply nested and subjectively driven by user intent?</p>
Active research areas:</strong></p>

Semantic compaction (compress by importance, not recency)</li>
Learned compression (models that compress their own context)</li>
Multi-modal memory (unified handling of images, audio, text)</li>
Adaptive budgets (dynamic token allocation based on conversation needs)</li>
</ul>
LLMs process fixed-size inputs. How we manage that constraint (balancing latency, cost, and accuracy) is still being figured out.</p>

We’re exploring these problems with context-store</a>, infrastructure for experimenting with memory techniques. Read about why we’re building it</a>.</em></p>


Introducing Context-Store
Sebastian Lund — Fri, 31 Oct 2025 00:00:00 +0000
Every team building LLM apps hits the same wall: users expect full message history, but LLMs have hard context limits.</p>
The Problem</h2>

Users often need to review and trust the full conversation history.</li>
LLMs need compaction to meet latency and token budgets.</li>
</ul>
Those two requirements pull in opposite directions. So we built context-store to improve LLM latency while preserving full history for your users. It’s simple, predictable, and works well for AI assistants.</p>
The Pattern Everyone Rebuilds</h2>
We kept seeing teams build the same stack:</p>

Redis</strong> for hot message storage</li>
Postgres</strong> for persistence</li>
Pub/sub</strong> for real-time updates</li>
Custom compaction logic</strong> to manage the window</li>
</ul>
Every implementation is slightly different. Every team makes similar mistakes. The infrastructure becomes a distraction from the actual product.</p>
What Context-Store Does</h2>
Context-store turns that pattern into a single service. You can choose to back everything up into postgres with a write-behind feature for analytics and to act as cold storage.</p>
You set a token budget. You pick a compaction policy (or write your own). The service handles:</p>

Storing full message history</li>
Enforcing token budgets</li>
Compacting context automatically</li>
Horizontal scaling, add more nodes to accept more data & traffic</li>
Optional cold storage in Postgres</li>
</ul>
const</span> ctx</span> = await</span> fastpaca.</span>context</span>(</span>'chat_42'</span>, {</span></span>
  budget:</span> 1_000_000</span>,</span></span>
  trigger:</span> 0.7</span>,</span></span>
  policy: { strategy:</span> 'last_n'</span>, config: { limit:</span> 400</span> } }</span></span>
});</span></span>
</span>
await</span> ctx.</span>append</span>({ role:</span> 'user'</span>, parts: [{ type:</span> 'text'</span>, text:</span> 'Hi'</span> }] });</span></span>
const</span> {</span> messages</span> }</span> = await</span> ctx.</span>context</span>();</span></span></code></pre>Example: Next.js Chat</h2>
We took this architecture and built a minimal chat app in Next.js that shows the pattern end-to-end: append messages, keep full history, and compact deterministically before calling your model.</p>

Example code →</a></li>
</ul>
</p>
Why Elixir</h2>
We built this in Elixir because:</p>

Raft consensus</strong> for distributed state is table stakes</li>
Process supervision</strong> makes failure handling clean</li>
Hot code upgrades</strong> for zero-downtime deployments</li>
Memory efficiency</strong> at scale</li>
</ol>
We chose Elixir for its concurrency model and reliability characteristics that fit this problem well.</p>
What’s Next</h2>
Context-store is production-ready and Apache 2.0 licensed.</p>
The research continues:</p>

Compression techniques on token-spaces</li>
Semantic compaction (embedding-based strategies)</li>
Multi-modal context handling</li>
</ul>
If you’re building LLM apps and this problem sounds familiar, check out the docs</a> or browse the code</a>.</p>

This is part of the broader fastpaca research into hard problems at the intersection of humans and technology. More posts coming!</em></p>