The vector store is useful. The abstraction is the problem.
There is a predictable moment in the development of almost every agent.
The team realizes that the agent needs to remember something beyond the current prompt. A vector store is introduced. Conversations or documents are chunked, embedded, and indexed. Retrieval is added before the model call. The first demo is encouraging: the agent can refer to an earlier conversation, answer questions about an uploaded document, and appear to maintain continuity across interactions.
Then the agent meets real usage.
It retrieves the customer detail from last week instead of the correction made this morning. It surfaces an earlier project state even though the facts have since changed. It carries an assumption from one conversation into another where it no longer applies. A temporary preference expressed for one task is retained as if it were a durable characteristic of the user.
The vector store is not necessarily malfunctioning. It is doing what it was designed to do: finding information that is semantically similar to a query.
The mistake is asking semantic similarity to also determine authority, freshness, ownership, scope, and intent.
I have run into different forms of this problem while working on agent platforms and, in a smaller and more personal setting, while building Engram. It is tempting to think of memory as an infrastructure capability: give the agent somewhere to write information, retrieve the relevant parts later, and gradually improve the retrieval quality.
But “the agent should remember” is not really a requirement. It is the beginning of several different requirements that happen to share the same word.
The model itself does not possess durable runtime memory. It has a context window. Everything beyond that is a system making decisions about what should be retained, how it should be represented, when it should be retrieved, and whether it should still be trusted when it is retrieved.
This makes agent memory less of a storage problem and more of a state-management and governance problem.
The failures are often about freshness and scope
The memory failures I have encountered most frequently are not cases where the system fails to retrieve anything. They are cases where it retrieves information that is relevant but no longer correct.
A user may describe an initial plan, revise it in a later conversation, and eventually settle on something else. All three versions may remain in the memory store. When the information is needed again, semantic retrieval does not inherently understand that the latest statement replaced the earlier ones. It may select the older record because its wording is closer to the current query or because it was stored in a form that happens to rank more strongly.
The agent therefore remembers something meaningful, but remembers the wrong state of the facts.
This is more dangerous than simple forgetting because the answer can still sound coherent. The agent has enough context to produce a plausible response, but the response is built on information that used to be true.
The other recurring problem is scope. A statement may be important for the current conversation without being a lasting fact about the user.
If someone says that they would prefer to stay close to the airport for an upcoming trip, that could be a constraint for this particular itinerary, a temporary preference caused by an early flight, or a broader travel preference that should be retained. The text itself may not make the boundary obvious, and a generic memory system will often preserve the statement without preserving the conditions under which it was true.
The problem is therefore not simply whether the system remembers a fact. It is whether it also remembers when the fact became valid, where it applies, how long it should remain valid, and what should happen when it changes.
A flat collection of textual memories is not enough to represent those distinctions reliably.
Working context
The first memory regime is the state required to keep the current interaction or workflow coherent.
This includes recent conversation turns, the current plan, decisions already made, relevant tool outputs, unresolved questions, pending approvals, and the artifacts the agent is actively manipulating.
This is often described as session memory, although the term can be misleading. A meaningful workflow may span several conversations, browser tabs, context windows, or process restarts. Some of the state must therefore survive beyond the immediate model call.
At the same time, the fact that something needs to survive an interruption does not mean it should become durable user memory.
A temporary assumption made for an analysis may need to remain available for the duration of the task. It should not silently become a permanent fact about the customer, the user, or the organization.
Working context therefore needs both persistence and containment. It has to preserve enough state for the agent to continue the work, while maintaining a clear boundary around information that applies only to that workflow.
Retaining the entire conversation does not solve this problem. As sessions grow, they accumulate abandoned approaches, verbose tool outputs, repeated information, and instructions that were relevant earlier but no longer matter. The agent may technically have the right information available and still fail to prioritize it correctly.
A more reliable representation preserves the structure of the work: what has already been decided, what remains unresolved, which assumptions are temporary, which artifacts are current, and which tool calls produced the information being used.
This begins to look less like conversational memory and more like durable workflow state.
Durable user state
Durable user state includes information that should carry across sessions: stable preferences, recurring behaviors, personal defaults, accessibility requirements, permissions, and facts the user reasonably expects the system to retain.
This is one of the most useful forms of agent memory, but it is also one of the easiest to corrupt.
A simple implementation can extract a possible preference from a conversation and write it into a durable store. That works well for an explicit statement such as “I do not eat seafood.” It becomes much less reliable when a user says something like “I would rather not take an early flight this time.”
The first statement is likely to be stable and broadly applicable. The second is scoped, temporary, and possibly influenced by circumstances that will not apply later.
One approach we have been working with is to allow agents to create durable preference records while attaching a confidence score to each item. The user can review those records, and a periodic cleanup process revisits lower-confidence items. Based on subsequent evidence, the process may increase the confidence, retain the item for further review, or remove it altogether. During recall, confidence acts as one of the filters that determines whether a memory should be surfaced and how strongly it should influence the agent.
This is useful because it avoids pretending that every extracted preference is equally reliable. It also creates a lifecycle around memory instead of treating every write as permanent.
Confidence, however, does not solve the entire problem.
A memory can be high confidence and still be obsolete. The system may have correctly understood a preference that was explicitly stated six months ago, even though the user has since changed their mind. Similarly, a preference can be correctly inferred but apply only to one project, one conversation, or one period of time.
Confidence is therefore one signal among several. Durable memory also needs provenance, scope, timestamps, supersession rules, and a distinction between information that was explicitly stated, inferred by the agent, or confirmed by the user.
The cleanup process also needs care. If another model simply rereads the same ambiguous evidence and assigns a new score, it can reinforce the original mistake rather than correct it. Some transitions need deterministic rules. An explicit correction should generally override an inferred preference. A user-confirmed memory should carry more authority than one automatically extracted from a conversation. A revoked permission should not remain usable simply because the original record had high confidence.
The broader principle is that durable user memory should be treated as managed state rather than as a collection of observations.
It should be inspectable, correctable, and removable by the user. A system that continues to learn without providing a meaningful mechanism to unlearn will eventually accumulate enough stale or incorrectly scoped information to become less useful over time.
Organizational knowledge
Organizational knowledge includes company policies, product definitions, customer information, compliance constraints, entitlement rules, business terminology, and operating procedures.
The important difference is that this information has an authority outside the conversation.
An agent should not learn the current refund policy because somebody described it in a previous chat. It should obtain the policy from a source the organization recognizes as authoritative.
That source may be an API, a policy engine, a configuration service, a transactional database, or a curated and versioned document repository. Semantic retrieval may still be the best way to locate a relevant section within a large body of documentation, but the retrieval result should not automatically become organizational truth.
Retrieval tells the system what information appears relevant. Authority tells the system what information is valid for the current decision.
Those are separate concerns.
A document may be highly relevant and no longer active. A previous conversation may closely match the current question but refer to another customer. A policy excerpt may be correct in general but overridden by a more specific entitlement or contractual rule.
The agent therefore needs more than a matching passage. It needs source identity, version information, effective dates, access controls, and the conditions under which the information applies.
The ranking score of an embedding cannot establish any of those things.
Workflow state and tool artifacts
Agents performing meaningful work produce intermediate state.
A search tool returns a set of records, the agent filters them, and a smaller subset is used in a later decision. A calculation produces a value that becomes an input to another step. A document is drafted but awaits approval. A tool call succeeds partially. A record is fetched from a source system and then changes before the agent acts on it.
This information is sometimes described as tool-result memory, but that framing understates its role.
It is execution state.
Execution state should be tied to the workflow that produced it and stored with enough metadata to determine whether it is still valid. That may include timestamps, parameters, lineage, approval status, idempotency information, and references to the underlying artifact or source record.
The agent should not merely remember that a previous search returned a particular price. It should know which search produced it, when the search ran, what parameters were used, whether the value was subsequently validated, and whether the result is still fresh enough to reuse.
This is another area where semantic retrieval can return something that is highly relevant but operationally wrong.
A previous tool result may match the current task perfectly and still be too old to trust. In such cases, the correct behavior is not to recall the result more effectively. It is to recognize that the result has expired and run the tool again.
Memory should not become a mechanism for avoiding necessary revalidation.
Procedural and episodic memory
There are other useful ways to classify memory.
Procedural memory represents knowledge about how work should be performed. An agent may learn how a user prefers a weekly update to be structured, which steps an organization requires before an action is approved, or which diagnostic sequence tends to work for a recurring issue.
Episodic memory represents prior experiences and outcomes. The agent may retain that a previous approach failed under a particular constraint or that the user rejected a recommendation for a specific reason.
These forms of memory can help agents improve over time, but they introduce the same questions of scope and validity. A procedure may change. A past failure may have been specific to an earlier version of the system. A preference about how to perform a task may apply to one team but not another.
The categories are useful, but no taxonomy removes the need to model ownership, authority, lifetime, and change.
Every memory needs a contract
For each piece of retained information, the platform needs to understand where it applies, where it came from, how long it remains valid, and which source has the authority to change it.
It should know whether the information belongs to one turn, one workflow, one user, one customer, or the organization as a whole. It should preserve whether the information was explicitly stated, inferred by a model, returned by a tool, or extracted from a document. It should also understand whether the record can be updated, superseded, revoked, expired, or consolidated with another record.
The correct access pattern will vary accordingly.
Some information should be retrieved chronologically. Some should be accessed by an exact key. Some belongs in a structured query, some in a graph, and some is genuinely well suited to semantic search.
The same applies to security. A memory system without clear identity, tenant, and authorization boundaries is not merely an imperfect retrieval system. It is a potential data-leak mechanism.
Once these contracts are defined, storage choices become easier. Without them, choosing a database or improving retrieval quality is premature.
The vector store has a job
None of this is an argument against vector databases.
They are useful when information is unstructured, the wording of the query may differ from the wording of the source, and approximate semantic similarity is the right access pattern.
They are well suited to discovering relevant documents, recalling loosely structured discussions, identifying similar cases, and supplementing structured records with contextual evidence.
They are poorly suited to being the sole source of truth for current permissions, exact mutable state, active policy, temporary workflow assumptions, or numerical values that may have changed.
The right architecture is usually hybrid.
The platform first determines what kind of information is needed and which source has authority. It then chooses the access pattern appropriate to that information. Sometimes that is semantic search. Sometimes it is a structured lookup, chronological replay, graph traversal, or a fresh call to the underlying system.
A vector store is an index.
It is not a memory architecture.
Memory writes are a security boundary
Most discussions about agent memory focus on recall quality: whether the agent retrieved the right information.
The write path deserves at least as much attention.
Anything written into durable memory can influence future reasoning and actions. This makes memory a target for accidental contamination, prompt injection, malicious instructions, incorrect inference, and cross-user leakage.
A conversational statement should not become durable truth merely because a model judged it to be noteworthy.
A safer pattern is to separate extraction from acceptance. The model may propose that something be remembered, but a policy layer should determine whether the information is eligible, how it should be scoped, whether it conflicts with existing state, how much confidence should be assigned, and whether user confirmation is required.
Confidence scoring, review mechanisms, and periodic cleanup all help make this process more robust, but consequential memories also need provenance and auditability.
The system should be able to explain where a memory came from, why it was retained, whether it was inferred or confirmed, what later replaced it, and which actions it influenced.
Without those answers, memory becomes invisible mutable state, which is precisely the kind of state reliable systems try to avoid.
The engineering discipline is older than the agent
Agent memory sounds like a new problem because the interface is conversational and the state is often expressed in natural language.
The underlying engineering concerns are familiar: cache invalidation, consistency, identity, tenancy, provenance, lifecycle management, event history, authorization, and the separation of authoritative state from derived views.
The new difficulty is that an agent can make weakly typed and contradictory information appear coherent. It can retrieve an obsolete or incorrectly scoped memory and weave it into an answer that sounds entirely plausible.
Forgetting is inconvenient. The agent may ask the user to repeat something.
False memory is worse. The agent may confidently act on information that used to be true, applied somewhere else, or was never meant to survive the original conversation.
That is why the goal of a memory architecture should not be to maximize how much an agent remembers. It should be to make explicit what the agent is allowed to remember, where that memory applies, how long it remains valid, why it should be trusted, and which source wins when different memories disagree.
The vector store is useful.
The abstraction is the problem.