🧠 Memory Systems
Building Long-Term Context for AI Agents
Jack Coy · SydClaw Meetup · March 2026
Scan to join in
Vote on the poll · Ask & upvote questions · Follow the slides
The Problem
Memory has improved — but it's still not solved
We've made progress
- ChatGPT auto-saves memory bullets
- Claude Code has auto memory + CLAUDE.md
- OpenClaw loads workspace files every session
- Context windows keep getting bigger
These help — but they're shallow, opaque, or manual.
What's still missing
- Switch platforms and your memory is gone
- Long sessions still hit context limits
- Very few systems automatically learn from patterns over time
- Memories lack weight — a stern correction and a casual joke are stored the same way
We want it to feel like a colleague — not a temp with notes.
What's Actually In The Context?
Before you type your first message, here's what the model already sees
💬 ChatGPT
System prompt (OpenAI's, hidden)
Memory bullets (auto-generated)
MCP tool definitions (if configured)
Your message →
You don't control the system prompt or what gets remembered.
💻 Claude Code
System prompt (Anthropic's)
CLAUDE.md chain (org → user → project + parent dirs)
.claude/rules/ (unconditional rules)
@imports (files referenced in CLAUDE.md)
Auto memory (Claude writes, 200 lines)
MCP tool definitions (can be 80K+ tokens!)
Your message →
Layered CLAUDE.md hierarchy. MCP tools can eat huge context.
🦞 OpenClaw
System prompt (OpenClaw's + yours)
SOUL.md personality
AGENTS.md instructions
USER.md · MEMORY.md · TOOLS.md
Skills list (compact index, details loaded on use)
Your message →
You write all workspace files. Skills inject a compact index, not full definitions.
Same model underneath. The starting context is what makes the experience different.
MCP servers (ChatGPT, Claude Code) and Skills (OpenClaw) extend what the agent can do — but consume context differently.
What Happens As You Chat
Context grows with every message — and then it hits a wall
📨 Every message replays everything
System prompt + workspace files
Message 1 + Response 1
Message 2 + Response 2
...
💥 Context window full!
Same everywhere — ChatGPT, Claude, OpenClaw. The full history is sent every call. Eventually, it doesn't fit.
So what happens next?
Every platform handles this differently. It's called compacting (or compaction, context management, summarization — lots of names for the same idea).
The basic idea: take older messages, summarize them, and keep going.
Sounds simple, right? But the details matter a lot:
- What gets summarized vs kept verbatim?
- Are the originals saved anywhere?
- Can you recover lost detail later?
- Does anything get saved for future sessions?
When Context Gets Full: Compacting
Every platform handles this differently — and most don't tell you how
💬 ChatGPT, Claude, Gemini, etc.
These platforms handle compacting for you — but you have no control over how.
- Proprietary algorithms — not published
- Generally summarize or drop older messages
- Claude mentions "organizing its thoughts" on paid plans
- You can't configure when, how, or what gets kept
- It just happens — you trust the platform
🦞 OpenClaw — you have a choice
Transparent and configurable:
- Summarizes older messages, keeps recent ones intact
- Memory flush before compacting — agent saves important notes to disk first
- Choose which model does the summarization
- Plugins can replace the entire compaction engine
- Session history persisted as JSONL — inspectable
By default, compacting is lossy — details get summarized away. And all of this is per-session. Start a new conversation and none of it carries over.
Improving Compaction
One example: Lossless Context Management (LCM)
🌳 What LCM does
- Replaces OpenClaw's default compaction with a DAG-based summarization system
- Every original message is kept in a SQLite database — nothing is ever lost
- Older messages get summarized into a tree of summaries
- Gives the agent tools to drill back into any summary and recover the full original detail
- Feels like talking to an agent that never forgets within a conversation
⚠️ What it doesn't do
- It's per-session only — start a new conversation and you're back to zero
- It doesn't build long-term memory across sessions
- It solves "I forgot what we talked about 2 hours ago" — not "I forgot what we talked about last week"
LCM is one approach to improving within-session compaction — there are others being explored too.
But compaction is a different problem from memory. Let's talk about that next.
Memory Is A Different Problem
You can't just dump everything into context — it won't fit
What fits in one API call
System prompt + skills + tools
Workspace files (each capped at 20K chars)
Conversation history
↕ Room for response
Fixed size. Finite.
vs
Everything you want it to know
Your preferences & personality
Past conversations & decisions
Project knowledge & notes
People, relationships, context
Lessons learned, mistakes to avoid
Changing facts & evolving preferences
Weeks and months of interaction
Calendar events, emails, tasks
Every correction you've ever made
📈 ...and it never stops growing
Unbounded. Always growing.
You can't load it all.
You need a way to retrieve the right memories at the right time — and leave the rest on disk.
Retrieving Memories
Two mechanisms — and one relies on the agent actually doing it
✅ Auto-loaded every session
Injected into context before your first message:
SOUL.md · AGENTS.md · USER.md
MEMORY.md · TOOLS.md · IDENTITY.md
Always there. Costs tokens every turn. This is your baseline — but it's limited to what fits in those files.
🔍 On-demand: memory_search
Semantic vector search over all memory files. But the agent has to choose to call it.
"Mandatory recall step: semantically search MEMORY.md + memory/*.md before answering questions about prior work, decisions, dates, people, preferences, or todos"
↑ This is the actual system prompt instruction. The model is told to search — but it doesn't always listen.
The gap: retrieval depends on the model's judgment call. Sometimes it searches proactively, sometimes as a fallback after failing, sometimes not at all.
How Memory Search Works
A tool call that finds relevant memories by meaning, not keywords
💬 "When's Aom's birthday?"
→
🧠 Model makes a tool call
memory_search("Aom birthday")
🔢 Query → embedding
your words become a vector that captures meaning
→
🔍 Semantic search across all memory files
95% match
📄 people.md · line 14
"Aom (Varunya) — birthday March 3, Thai New Year is important too"
78% match
📄 daily/2026-02-20.md · line 5
"Gender reveal party for Jack & Aom's baby..."
34% match
📄 topics/plex.md
not relevant — filtered out
↑ These snippets, file paths, and line numbers are all returned as tool results — now in the context.
Finds by meaning, not keywords — "Aom birthday" matches "Varunya born on March 3"
📖 Optionally, the model can read more
If the snippet isn't enough, it can just read the full file — a normal file read, nothing special.
→
✅ Memories in context
ready for the response
Semantic Search: Defaults & Limitations
What OpenClaw gives you out of the box — and where it falls short
✅ OpenClaw defaults
- Enabled by default if you have an embedding API key
- Auto-selects provider: local (if configured) → OpenAI → Gemini → Voyage → Mistral
- Indexes
MEMORY.md (already in context) + everything in memory/ folder (daily logs, topic files, etc. — only accessible via search)
- Watches files — automatically re-indexes on change
- Supports fully local/offline embeddings too
⚠️ The limitations
- Can only find what's been written to memory files — if it was never stored, it can't be found
- The model has to decide to search — it's not automatic on every message
- Sometimes the model searches as a fallback, not proactively
- Quality depends on how well memory is organized and chunked
- Each retrieval adds latency and tokens to the turn
Capturing Memories
This is the harder problem — and it's mostly manual today
🦞 OpenClaw defaults
System prompt says "write it down"
The agent is instructed to write decisions, preferences, and facts to memory files. But it's guidance, not enforcement.
Memory flush before compaction
When context is about to be compacted, OpenClaw runs a silent turn reminding the agent: "save durable notes to disk now." A safety net for long sessions.
You can write memory yourself
"Remember that I prefer Thai food" → agent writes to MEMORY.md or a daily file. Works when you remember to ask.
⚠️ The problems
Agents don't reliably follow instructions
The model might forget to write things down, especially when it's focused on a task. Important context slips through.
What gets stored is inconsistent
Sometimes it saves too much detail, sometimes not enough. No systematic process for deciding what matters.
Storage structure is basic
Markdown files in folders. No cross-referencing, no graph structure, no way to connect related facts across files. A decision about a project and a person's preference learned at the same time end up in different places.
The default is "hope the agent writes things down." We can do better.
Inspiration: The Human Brain
What if we built memory the way brains actually work?
🌞
During the day
You don't try to remember — it just happens. Memories are passive.
Your hippocampus quietly encodes experiences in the background.
Emotional weight matters — your boss saying "never do that again" sticks. A casual joke doesn't.
→
🌙
During sleep
Your brain replays the day's events, compressed and reorganized.
Important memories move from temporary (hippocampus) to long-term (neocortex).
Memories transform — "dinner at that Thai place Tuesday" becomes "I enjoy Thai food with Aom."
→
💡
When you need it
One memory triggers related ones — a smell recalls a place, a person, a feeling.
Frequently accessed and emotionally charged memories are stronger.
Unused memories fade — but might still be there, waiting for the right trigger.
The key insight: memory is passive capture, offline consolidation, and associative retrieval. Can we build something similar?
What I Built
Inspired by Total Recall + human sleep consolidation — passive, not active
👀
Observer
Every 15 minutes
Gemini Flash (cheap)
Scans recent activity.
Notes what happened.
Events only, no conclusions.
Most gets discarded.
→
🔄
Reflector
Hourly
Gemini Flash (cheap)
Clusters observations.
Spots patterns.
Suggests routing.
Still non-canonical.
→
💤
REM Cycle
Nightly (2am)
Claude Opus (expensive)
Reviews everything.
Promotes to long-term memory.
Prunes stale entries.
This is where quality matters.
The agent doesn't have to remember anything — the system observes what's happening outside the session and processes it passively.
Heavily inspired by Total Recall (observational memory for OpenClaw) + a custom nightly REM cycle. Still a work in progress.
Where Memory Lives
Plain Markdown files — no database, no magic
memory/
├── daily/ raw daily logs
├── channels/ per-channel context
├── topics/ cross-cutting knowledge
├── rem-logs/ consolidation audit trail
└── MEMORY.md distilled long-term
Daily logs
Raw chronology. Session residue. No elegance required. Ages out.
Channel context
Per-channel memory. What's normal in #home-lab vs #coding. Tiered: daily → weekly → long-term.
Topic files
Cross-cutting project and subject knowledge. "What do I know about the Plex stack?"
MEMORY.md
Distilled canonical memory. Only things that change future behavior. Curated by the REM cycle.
Track your workspace in a private git repo. These files are the soul of your agent — everything it knows about you. Ask your agent to set up a nightly cron job that commits and pushes. If your machine dies, you can restore everything.
What's Still Missing
Taking more inspiration from how the brain actually works
✅ Capture
Passive observation ✅
Like the brain — we don't rely on the agent actively remembering. The observer watches outside the session.
No importance weighting ❌
The brain tags emotional events as more important. A stern correction and a casual joke get stored the same way in our system.
⚠️ Storage
Flat files, not a graph ❌
The brain stores memories as networks — nodes with many connections. We use folders and files. A dinner memory should link to the person, place, food, date, and conversation topic.
No temporal decay ❌
Human memories fade unless refreshed. A 6-month-old preference is just as accessible as yesterday's correction in our system.
❌ Retrieval
No associative chains ❌
Human memory triggers related memories in a chain — a smell recalls a place, a person, a feeling. We just do similarity search on one query.
No strengthening on access ❌
In the brain, recalling a memory makes it stronger. In our system, retrieving a memory doesn't change its weight or priority at all.
We've made progress on capture. Storage and retrieval are wide open for improvement. This is where the interesting work is.
Invest In Your Agent's Memory
The more you invest in telling your agent who you want it to be, how you want it to help you, and what it knows about you — the more personal and useful it becomes.
Platforms will change. Models will change. But the context you build — your preferences, your style, your decisions — that travels with you.
This isn't a solved problem. I've shared some ideas, not a plug-and-play solution. Experiment. Iterate. Make it yours.
github.com/openclaw/openclaw ·
docs.openclaw.ai
🧠
Questions?
Thanks for listening!
References & Further Reading
AI Memory Systems
Letta / MemGPT — sleep-time consolidation
letta.com