Andrej Karpathy’s LLM Wiki gist landed on 4 April and had an open-source clone 45 minutes later. Since then the “AI second brain” category has grown into 22 profiled products, a curated comparison repo, and a benchmark fight in which every vendor wins its own test. The best-evidenced result in the field is still Letta’s: an agent that just reads files scores 74.0% on LoCoMo, beating Mem0’s 68.5% graph variant. Use the lifecycle taxonomy the vendors published, ignore their rankings, and start with markdown until you can prove you need more.
Karpathy published the LLM Wiki gist at 16:25 UTC on 4 April 2026. The idea is deliberately unimpressive: a git repo of markdown files, an agent that reads new sources and folds them into interlinked pages, version history for free. lucasastorian/llmwiki, an open-source implementation, was created at 17:10 the same day. Forty-five minutes. A second one, llm-wiki-compiler, appeared the next evening.
Three months on, the category has a taxonomy, a shortlist, and an awesome list. It also has a problem, which is that almost nobody can tell you whether any of it works better than the markdown files.
The one genuinely good idea here
The awesome-second-brain repo went up in January and now profiles 22 solutions. Its useful contribution is the lifecycle it grades products against:
| Stage | The question it asks |
|---|---|
| Collect | How does context from chats, docs, apps, calendars, email, and code get in? |
| Organize | Does raw context become structured knowledge instead of a pile of embeddings? |
| Evolve | Does memory improve as new context arrives and old context goes stale? |
| Use | Does the right context show up when someone is doing real work? |
| Govern | Can you inspect, correct, delete, export, and scope what’s in there? |
This beats a feature matrix because it forces the question that actually predicts whether a second brain survives contact with your week. Most of them die at Evolve. Capture is easy and every tool does it; keeping a knowledge base honest as facts change is the hard part, and it’s the part vendors describe with words like “consolidation” and then decline to elaborate on.
The repo’s own capability matrix is refreshingly blunt about this. Under Memory evolution, half the entries read “Partial” or “agent-operated”, which is a polite way of saying you’ll be doing it. Under Activation evidence, the column asking whether a system can prove retrieved memory affected the work, nearly every row says “Partial”, and the caveat is always the same: acted-on proof depends on the agent’s logs, which nobody has.
Read the map, not the recommendations
One thing to know before you use it. The repo is published by Aristo, whose product is Membase, and Membase is listed first in every one of the five lifecycle rows. The prose above the table names it “the default starting point”. That is a vendor-authored map on which the cartographer’s own town is the capital, five times over.
To be fair, the contribution guidelines are stricter than most vendor content: contributors are told to distinguish independent validation from maintainer-published benchmarks, to avoid “best” and “state of the art” unless an independent source backs the exact comparison, and to mark unavailable details as Unknown. Good rules. They’re just applied to everyone except the house.
So take the taxonomy, which is genuinely good, and read the rankings the way you’d read a Gartner quadrant paid for by one of the dots.
Every vendor wins its own benchmark
The field’s benchmark is LoCoMo, 1,540 questions over long synthetic conversations. Here is what happened to it.
Mem0 published a paper claiming state of the art, beating Zep and MemGPT. Zep re-ran it, found the implementation wrong, and published a correction showing Zep ahead by 10%. Letta, the team behind MemGPT, pointed out that they could not work out how Mem0 had run MemGPT at all, since backfilling LoCoMo into it requires a refactor they never saw; their request for clarification went unanswered.
Then Letta ran the experiment that matters. They dumped the LoCoMo conversations into a plain file, gave a GPT-4o-mini agent four tools (grep, search_files, open, close), and let it search until it was ready to answer. Score: 74.0%, against the 68.5% Mem0 reported for its best graph variant. A weaker model, no memory product, no graph, no vector strategy beyond what came free with the file.
Letta’s reading of that result is the honest one: the benchmark mostly measures retrieval, agents are extremely good at filesystem tools because those tools are all over the training data, and an agent that can reformulate its own query and search twice beats a single-hop lookup into a clever index. The scoreboard was measuring whether the harness lets the model look things up more than once.
What the practitioners are saying
The Hacker News thread on Hermes Agent’s “persistent memory” in early June contains the entire debate in one comment:
> persistent memory *looks inside* MEMORY.md
Another commenter in the same thread who’d worked through five harnesses put the objection more usefully: “I’d rather have full session logs rather than these MEMORY.md summaries of what a session did.” That’s the real complaint, and the file format has little to do with it. A summary is a lossy write with no undo. Once the agent decides your project uses Postgres and writes it down, the transcript that said otherwise is gone, and every future session inherits the error with full confidence and no provenance.
Everyone is rebuilding the same substrate to dodge that. In the last month HN has seen a persistent agent memory layer on Elasticsearch (116 points), a proposed Universal Memory Protocol for a shared memory format (41 points, 38 comments of disagreement), and earlier this year an open memory layer aiming to give any agent what ChatGPT and Claude have (185 points). Before that: SQL instead of vectors, git instead of vectors, write-gated memory for Claude Code. The convergent answer, arrived at independently and repeatedly, is boring durable storage the user can read.
The column nobody scores well on
Govern is where this stops being a productivity question. The awesome-second-brain repo has a page on activation evidence which asks whether you can prove which memories crossed into a task and whether the agent relied on them. It proposes a test you can run against any system without knowing its internals: seed one decision with a source, one rejected option, and one stale note, open a clean session, and ask for the current decision plus the stale risk. Then check whether the answer cites activated context or just sounds confident.
Run that test, because a second brain is a write-many, read-many store that an agent both populates and trusts. That is the same shape as the skill-poisoning problem, with a longer fuse. Poison a skill and it fires when invoked. Poison a memory and it becomes background truth, retrieved silently into every future session, with the agent’s own confident voice laundering the provenance away. Add a connector pulling from email or Slack and you have untrusted input, private data, and an agent with tools, which is the lethal trifecta with a persistence layer attached.
Almost none of these products can currently tell you which memory changed an answer. That’s the gap worth watching, and it’s more interesting than another point on LoCoMo.
What to actually do
Start with a directory of markdown files in git, wired to whatever agent you use. Letta’s result says you’ll be close to state of the art on retrieval, and git gives you diff, blame, and revert, which no memory API in the list offers. Adopt a product only when you can name the lifecycle stage it fixes and the failure you hit without it, and treat “Evolve” claims as unproven until you’ve watched the tool correct a fact it got wrong. Keep raw sources separate from synthesised pages so a bad summary is recoverable. Check whether you can see what was retrieved.
The category is real and some of these products will matter. But right now the strongest empirical claim anyone has made is that the filesystem is competitive, and the most sophisticated evaluation dimension anyone has published is one that nearly every product fails. Files until proven otherwise.
Sources
- Andrej Karpathy: LLM Wiki gist (4 April 2026), and implementations llmwiki and llm-wiki-compiler
- Aristo: awesome-second-brain, capability matrix, activation evidence, contributing guidelines
- Letta: Benchmarking AI Agent Memory: Is a Filesystem All You Need?
- Zep: Lies, Damn Lies, & Statistics: Is Mem0 Really SOTA in Agent Memory?
- Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory (arXiv:2504.19413) and the unanswered benchmark issue
- LoCoMo: benchmark site · MemGPT: arXiv:2310.08560
Coverage and discussion
- HN: Hermes Agent: open-source AI agent with persistent memory
- HN: A persistent agent memory layer on Elasticsearch
- HN: Universal Memory Protocol
- HN: Open source memory layer so any AI agent can do what Claude.ai and ChatGPT do
- HN: Everyone’s trying vectors and graphs for AI memory. We went back to SQL
Related on this blog