For the last three or four months, I have been building Engram, a personal knowledge and productivity system.
The motivation was simple enough. I usually have too many threads running at the same time. There is work, a few personal projects, half-formed ideas, things I have promised people, things I am waiting on, and things I am reasonably sure were important when I first wrote them down.
The information itself is rarely lost. It is usually sitting somewhere perfectly safe: Slack, email, Apple Notes, a repository, a document, an agent conversation, or one of the many browser tabs that I am keeping open for “later.”
The harder part is remembering where the information is, whether it is still relevant, and what I was supposed to do with it.
Engram was meant to help with that.
The idea was to give agents and me a shared place where useful context could accumulate over time. It would understand projects, tasks, people, decisions, notes and relationships between them. I could drop in a rough thought or a conversation, and Engram would extract the useful parts, connect them to what it already knew, and help me pick things up later.
Over a few months, it became fairly capable.
It had a React frontend, a Flask backend, PostgreSQL, pgvector, semantic search, full-text search, typed entities, relationships and an MCP server. Agents could query it, update it and use its context in other workflows.
At one point, Engram had enough architecture to support a small organization.
Its main user was still forgetting to open it.
That should have been an early clue.
The original diagnosis
I started with the view that my information problem was mostly about fragmentation.
That was partly true. My project context was spread across several systems, decisions were buried inside conversations, and follow-ups were often stored in the least reliable database available: my own memory.
A central workspace seemed like a reasonable answer.
Once the information was in one place, Engram could classify it, connect related items and make it searchable. A note could belong to a project. A task could be associated with a person. A decision could link back to the conversation where it was made.
The graph became richer. The search got better. The extraction became more accurate.
Engram could remember a lot.
Unfortunately, remembering something and understanding the current situation are quite different jobs.
Suppose I mentioned in a conversation that I was considering a new direction for a project. Engram could extract that correctly. It might even connect the idea to the right project, people and related decisions.
A few days later, I could change my mind.
Now Engram had two pieces of information, both accurately captured, and no reliable way to know which one should guide the next agent interaction.
The problem became more obvious as the system accumulated information.
A project may be active today and paused next week. A deadline may move. Someone may take ownership of a task and later hand it to another person. A decision may begin as a tentative idea, survive two meetings and then quietly disappear without ever becoming official.
Engram was quite good at preserving each of these moments.
It was less good at understanding which moment still mattered.
The trouble with facts
In the early versions, I thought about trust mostly in terms of extraction quality.
If the agent was highly confident, Engram could accept the result. If the confidence was low, it could ask me to review it. Important changes could require approval.
This sounded sensible and gave me several new screens to build.
It also turned out to be an incomplete model of trust.
An extracted fact can be accurate and still be outdated.
It can also be valid only within a specific context. Something discussed while exploring an idea should not automatically become the official direction of the project. A passing complaint should not become a durable truth about a person. A task suggested in one conversation may never have been accepted by anyone.
Confidence scores did not help much here. A model can be extremely confident about something that stopped being true last Tuesday.
Engram needed to understand freshness, supersession and scope. Those concepts sounded straightforward until I tried to define how they should work across real conversations.
How long should a fact remain current?
Which new statement replaces an older one?
Does a contradiction mean the earlier information is wrong, or does it reflect a change?
Should the system update the project automatically, or ask me first?
These are interesting questions when discussing agent memory on a whiteboard. They become less charming when the system confidently reminds you about a decision you abandoned two weeks ago.
The review queue arrives
Human review seemed like the safe answer.
If Engram wanted to update a project, create a task, establish a relationship or store something as durable knowledge, I could approve the change.
This protected the system from making bad assumptions.
It also gave me another inbox.
Engram was meant to reduce the amount of information I had to manage. It now needed me to review the information it had managed on my behalf.
The contradiction was difficult to ignore.
Every additional inference created a potential review item. Every review item required me to reconstruct the context, decide whether the agent had understood it correctly, and determine whether the information was important enough to retain.
The review workflow itself was well intentioned. It was also productivity work wearing a safety vest.
I had two options.
I could keep the queue clean and spend time maintaining Engram.
Or I could ignore it, which is what happened often enough for the system to become less trustworthy.
Neither option produced the calm, capable second brain I had imagined.
A knowledge system that needed constant updates
Engram also depended on information from systems where my work was actually happening.
That included Slack, email, documents, repositories, meeting notes and agent conversations. Without deep integrations, I had to move the relevant information into Engram myself.
This created a slightly ridiculous operating model.
Engram needed current information to help me manage my work.
Keeping Engram current required me to remember what had changed and update it.
Remembering what had changed was one of the problems Engram was supposed to solve.
The loop was technically complete, though perhaps not in the way I intended.
Manual capture was only one part of the issue. Even with automated ingestion, Engram would still have to decide what deserved to become durable knowledge.
Most conversations contain a mixture of useful facts, temporary context, speculation, jokes, abandoned ideas and statements made with great confidence by people who had not yet checked anything.
Persisting all of it would create an impressive memory and a terrible source of truth.
Persisting too little would leave the system with gaps.
The more I worked on this, the clearer it became that a useful memory system must be selective. Selectivity requires judgement, and judgement is exactly where things get complicated.
Engram knew plenty and still could not tell me what to do
The promise of Engram extended beyond storage.
I wanted it to help me answer questions such as:
- What needs my attention?
- What am I waiting on?
- Which commitment is starting to drift?
- What changed while I was focused elsewhere?
- Which project looks active in the database but has quietly stopped moving?
This was the part that could have made Engram genuinely valuable every day.
The system had projects, tasks, dates, statuses and relationships. It could retrieve relevant notes and show me related activity.
The result still depended heavily on the quality of the information underneath.
A task could only appear overdue if the due date had been captured correctly. Engram could only know I was waiting on someone if that dependency had been recorded. It could only identify a stalled project if it had a reliable view of recent progress.
The system could reason over what it knew.
What it knew was partial, occasionally stale and collected through a process I did not follow consistently.
That limited how useful its recommendations could be.
A search system can remain useful even with incomplete information. You ask it a question, and it returns what it has.
A system that tells you what deserves attention needs a much higher level of trust. Missing one important commitment can matter more than correctly surfacing ten minor ones.
Engram had reached the awkward stage where it knew enough to sound helpful and not enough for me to rely on it.
Naturally, I added more features
When Engram fell short, my response was often to improve the system.
If information was difficult to find, I worked on search.
If the context felt disconnected, I added relationships.
If manual organization was tedious, I added agent-assisted extraction.
If agents needed access, I built an MCP server.
If the data model felt restrictive, I introduced richer entity types and workflows.
Each decision made sense at the time. This is one of the reasons overengineering is so easy to miss while you are doing it.
Nobody wakes up and decides to make a personal project unnecessarily complicated.
You add one useful capability, then another, and eventually you have a platform roadmap for a product whose only user is still taking notes somewhere else.
I was solving increasingly sophisticated problems before proving that Engram could complete one simple and useful loop consistently.
The architecture kept moving forward because architecture is comfortable territory. There are clear decisions to make, trade-offs to evaluate and code to write.
The product questions were harder.
Why would I open Engram every day?
What would make me trust it?
Which piece of value would justify the effort required to keep it current?
A new relationship type could be implemented in an afternoon. Those questions could sit unanswered for weeks.
The lesson I nearly reduced to “capture friction”
For a while, I thought Engram’s main problem was that capture was not easy enough.
Capture friction was real. If I had to open another application, review an extraction or decide how something should be classified, Apple Notes usually won.
But that explanation was too shallow.
Even perfect capture would not have solved the full problem.
Engram still needed to understand whether the captured information was temporary, durable, current, superseded or relevant only in one context. It still needed a safe way to update its view of the world without making me approve every small change.
The real challenge was maintaining a trustworthy model of something that was constantly changing.
That is a much larger problem than saving notes.
It also explains why many knowledge systems eventually become archives. They are good at collecting information and less effective at maintaining an accurate view of an evolving situation.
Archives can tolerate contradictions. Operational systems cannot.
I wanted Engram to behave like an operational system for my work and life. I had given it the ingestion model of a notebook.
Starting again, with less ambition and better questions
A new version of Engram is now in the works.
I am keeping it deliberately simple for the moment, which may be the most technically challenging feature I have attempted so far.
I am not rebuilding every entity type, relationship, approval workflow and agent tool from the previous version. Some of those ideas will return if they prove necessary.
The immediate goal is smaller.
Engram should maintain a limited amount of useful, current context and help me understand what deserves attention.
That requires more restraint around memory.
Not every statement should become a fact. Not every conversation should update a project. Some information should remain attached to its source. Some should expire. Some should be treated as tentative until later evidence makes it useful.
Changes also need to be represented as changes.
When something new contradicts an older entry, Engram should not quietly store both and hope semantic search sorts it out. The relationship between them matters. The newer information may replace the old, narrow its scope or record that the situation changed.
I am also reconsidering how review should work.
A large approval queue has already failed its audition. The new system needs to make small, reversible updates safely and interrupt me only when the consequence justifies the interruption.
Most importantly, the new version has to prove itself through use.
Can it bring back an important thread at the right time?
Can it notice that I am waiting on someone?
Can it help me understand what changed?
Can it turn scattered context into a useful next step?
A positive answer to one of those questions would be worth more than another six entity types.
Where Engram stands
Engram remains a personal project, as it has always been.
I am not building it as a commercial product, and there is no launch plan hiding at the end of this article. That gives me the freedom to delete large parts of it without preparing a migration guide or explaining the roadmap to imaginary customers.
The last few months were still worthwhile.
I learned a great deal about agent-assisted capture, semantic retrieval, structured memory and MCP-based access. I also gained a practical understanding of where each one stops helping.
The most valuable lesson came from the part that did not work.
A system can remember every conversation and still misunderstand the situation. It can retrieve relevant information and still surface the wrong version of the truth. It can protect the user through review and quietly create more work than it removes.
The next version of Engram will do less, at least for now.
That feels uncomfortable because less functionality can look like regression when measured through a feature list.
My actual measure is simpler: whether I use it, trust it and miss it when it is unavailable.
I will share more as the new version takes shape, including the parts I keep, the parts I remove and the assumptions that survive contact with daily use.
For now, Engram has taught me that remembering information is relatively easy.
Keeping up with reality is where the trouble begins.