Don't give your agent memory before you give your user control

I worked through the memory layer for a Bible study agent app recently. The app lets people read without AI, then sign in when they want study agents to help with context, cross-references, language, and reflection.

That made memory tempting right away. A study app could remember preferred translation, recurring topics, passages someone keeps returning to, notes they saved, and the kinds of answers that help them learn.

The first real decision was not which memory provider to use.

It was what the product was allowed to remember.

When should an agent remember something?

That sounds like an implementation detail, but it changes how personal the app feels, how much it costs to run, what users need to trust, what admins need to configure, and what the agent is allowed to bring into future conversations.

The easy version is saving everything. Every chat message, every answer, every preference, every inferred topic. Put it all into a semantic memory service and let retrieval sort it out later.

That is also the version I would avoid first.

The first memory layer in an agent app should be boring, visible, and user-controlled. Especially when the app touches something personal.

Chat history is not memory#

Most AI products already have chat history. That is useful, but it is not the same thing as memory.

Chat history is chronological. It answers, “What happened?”

Memory is selective. It answers, “What should matter later?”

That distinction is where product design starts. A user might say ten things in a session, but only one of them should change how the agent helps them next week. A generated answer might be useful in the moment, but that does not mean it belongs in long-term context. An inferred preference might be correct, but it can still feel wrong if the app quietly stores it without asking.

Memory should not be treated as a transcript with better search.

It is closer to a small shelf of durable notes the user can inspect, edit, and throw away.

The record and the finder#

A useful split is:

• durable memory is the record
• semantic memory is the finder

The record should be plain. It should say what was saved, who saved or confirmed it, where it came from, and when it changed. It should be easy to show in a settings panel. It should be easy to delete.

The finder can be clever. It can use embeddings, similarity search, or a managed memory provider to retrieve related ideas that do not share exact wording.

But the finder should not become the source of truth before the record is trustworthy. If the only memory layer is semantic, the product has no simple answer to basic user questions: What do you remember about me? Why did you use this? How do I remove it?

That is why I like starting with explicit durable memory, then adding semantic retrieval as an upgrade.

Memory belongs in the loop#

Sean’s AI Stories walkthrough on agent harnesses opens in a new tab sharpened this for me. The lesson I took from it was not “add more memory.” It was the opposite: memory is one controlled input inside a larger agent loop.

An agent run has more moving parts than a prompt sent to a model: working context, retrieved memory, tools, stopping rules, traces, and evaluation.

That changed what I wanted in the study app. Instead of “memory” as a chat feature, I wanted the app to separate different jobs:

• procedural memory is how each agent should behave
• durable memory is what the user explicitly saved or confirmed
• episodic history is what happened before, not automatically identity
• retrieval decides whether exact lookup, structured query, or semantic search fits
• tracing explains why a memory or tool result entered the answer

That last part matters. If a memory makes an answer worse, I want to know whether the bug was retrieval, prompting, tool selection, or the loop ending too early.

Memory without that structure is just another place for the agent to become mysterious.

Start with explicit memory#

The first useful memory system can be simple:

• a user saves a preference
• a user saves a note
• a user confirms an insight is worth keeping
• the app records when it was created and where it came from
• the user can delete it later

That gives you a clean promise to the user. The user is not guessing what the agent knows. The agent is not quietly developing a hidden profile. The team is not paying for semantic storage on every stray sentence.

This is especially important for apps that touch reflective, personal, educational, health, financial, or spiritual domains. In those contexts, “the agent remembers you” can be comforting or invasive depending entirely on whether the user has control.

The difference is whether the user agreed to it.

Cheap durable memory is underrated#

Before reaching for a semantic memory provider, I would usually start with normal durable storage.

A small relational table gets you surprisingly far:

memories
- id
- user_id
- kind
- text
- source_run_id
- user_confirmed
- created_at
- updated_at

The important part is not the table. The important part is the shape.

kind lets the product separate preferences from notes, topics, and saved insights. source_run_id gives you traceability. user_confirmed lets the app distinguish explicit memory from something proposed by the agent. Timestamps make cleanup and review possible.

This part is cheap. It is easy to explain. It can be exported. It can be deleted. It can be queried directly.

That matters because users have to trust memory before it can help them.

Add semantic memory when retrieval needs it#

Semantic memory is still useful. I just would not make it the first layer.

A service like Mem0, a vector database, or another embedding-backed memory system becomes valuable when exact lookup stops being enough. If the user saves “I am studying patience and endurance” and later asks about “perseverance,” semantic retrieval can make the experience feel more attentive.

That is the right job for semantic memory:

• find related themes
• retrieve similar preferences
• surface prior insights that do not share exact words
• make the agent feel continuous across sessions

But it is also an extra cost. It can add API calls, storage charges, embedding cost, retrieval latency, and more state to debug. It can also make debugging harder because the answer to “why did the agent use this memory?” may involve scores, transforms, and provider behavior.

So I like a simple three-mode setup:

durable  -> explicit user-visible memory only
semantic -> managed semantic memory only
hybrid   -> durable memory plus semantic retrieval

The control matters more than the label.

If memory has a provider setting, a retrieval limit, and a visible admin page, the product can evolve. If the provider is hardcoded into the app, every future pricing, quality, or privacy decision becomes a code deploy.

There is one rule I would add before turning on hybrid memory:

Delete has to mean delete everywhere.

If the user removes a memory from the durable record, the semantic copy, embedding, or provider-side memory needs to be deleted too. In a hybrid system, the durable record should be the source of truth and the semantic layer should behave like an index. If that index cannot be cleaned up deterministically, I would not promise the user that memory deletion is complete.

Put a budget around memory#

Memory feels like a feature, but it is also context.

Every memory that enters the prompt competes with the user’s current question, the agent’s tool results, system instructions, and any retrieved documents. More memory is not always better. Sometimes it makes the agent less precise. Sometimes it makes the app more expensive. Sometimes it brings old context into a conversation where it no longer belongs.

The memory layer needs limits:

• how many memories can be retrieved
• which kinds of memory are eligible for context
• whether unconfirmed memories can be used
• whether users can see the memories used in an answer
• whether admins can change provider behavior without redeploying

This is where the memory feature becomes part of how the product is run.

Storing memories is only half the feature. You also need to decide when they are allowed to influence the agent.

Trace the memory decision#

The other thing the video pushed up the priority list was tracing.

If an agent uses memory, I want the run to leave behind a small explanation of what happened:

agent_run_trace
- run_id
- agent_id
- selected_memory_ids
- memory_sources
- retrieval_reason
- retrieval_score
- tool_calls
- prompt_inclusion
- end_reason

This does not need to expose private text in an analytics dashboard. It can store ids, kinds, scores, and short reasons.

The point is operational: when an answer feels off, I should be able to tell whether the wrong memory was retrieved, the right memory was ignored, the tool result contradicted the memory, or the loop stopped before the agent had enough context.

That is the difference between “the agent remembered something” and “the system can explain why memory mattered.”

The user should feel recognized, not watched#

The emotional test for memory is simple:

Does the app feel like it is paying attention, or does it feel like it is keeping a file?

That line is thin.

An agent that remembers a user’s preferred reading level can feel generous. An agent that resurfaces a sensitive topic without warning can feel invasive. An agent that says “based on your saved note…” is easier to trust than one that seems to know something without explaining where it came from.

Good memory should be legible:

• show what was remembered
• show why it was used
• allow correction
• allow deletion
• avoid pretending inference is certainty

That last point matters. A user-saved memory and an agent-inferred memory should not carry the same weight. If the system proposes a memory, the user should get to approve it before it becomes durable identity.

What I built first, and what comes next#

For the first version of this app, I started with the boring pieces:

1. Explicit saved memories in ordinary durable storage.
2. A settings panel where users can view and delete those memories.
3. A small retrieval limit for adding memories to agent context.
4. Admin controls for provider, retrieval count, and memory behavior.

The video pushed the next pieces higher:

1. A visible “memories used” section in agent responses.
2. A trace for each run that records which memories and tools influenced the answer.
3. Eval checks for whether memory helped or hurt the response.
4. Semantic retrieval only after exact saved memory starts to feel insufficient.

This sequence is less impressive in a demo than “the agent remembers everything.”

It is also much easier to trust, price, operate, and debug.

The pattern#

My current bias is:

Treat memory as something the user can see and control before treating it as an AI feature.

That shift changes the setup. User control decides what the product is allowed to remember. Admin control decides when that memory is allowed to influence an agent run.

The first question stops being, “Which memory provider should we use?”

You start asking better questions:

• What deserves to be remembered?
• Who approved it?
• Can the user see it?
• Can the user remove it?
• How much does retrieval cost?
• How many memories should enter context?
• Can an admin change this without a deploy?
• Can we explain why this memory was used?

Those questions are less exciting than embeddings and agent frameworks. They are also the questions that decide whether memory makes the app feel personal or careless.

Caveats#

Some products need automatic memory earlier. Support tools, research workspaces, and coding agents may benefit from aggressive capture because the work is more operational and less personal.

Even there, I would separate raw history from durable memory. Store the history if you need it. Promote only selected facts into memory. Give the user or admin a way to inspect what was promoted.

The safer default is “remember deliberately.”

That is the principle I want in the product.

Remember deliberately.

Disclosure#

Drafted with AI assistance, then edited around the actual lesson.