How my agents remember: five layers of memory I can read and edit

How 10 agents on one Azure server remember what matters across months of daily work. Small notes I can read and edit keep the active rules fresh, while full searchable history on my own server preserves every past conversation.

Architecture: the five layer hierarchy

Each layer has one specific job, one retention rule, one size budget, and one gate on who may write to it. The model reads only L1 directly. Everything else arrives through the context assembler or a tool call.

The problem: remembering everything makes agents worse

Agents that try to remember everything fail in two quiet ways. The context window fills with stale detail, so the model spends attention on noise. And a store full of unchecked fragments lets the model recall things that were never true. Neither failure raises a clean error.

Cutting old messages blindly is not the answer either: the agent forgets earlier decisions, and a retried task starts from zero. I wanted durable memory small enough to read and audit in full, every durable write reversible, and the bulk of history compressed for the model without ever being thrown away.

How the five layers work

• L1 Session window and system prompt: The live working desk. Holds the system prompt, the agent's role rules, active conversation turns, and tool results. When it grows past the compaction threshold, LCM-X folds the oldest turns into summary nodes and preserves pointers to the original messages.
• L2 Ephemeral scratch: The scratchpad. Each agent's TMPDIR points to its own cache/scratch directory. Downloads, extracted archives, and probe scripts land here. Nothing in scratch is read by subsequent sessions, and entries idle for 24 hours are pruned automatically.
• L3 Curated persistent memory: The personal notebook. Two plain Markdown files per agent, MEMORY.md for system notes and USER.md for my profile, with hard character caps (2,200 and 2,600 characters on the router agent). Both files are loaded in full into every session, eliminating vector ranking latency and missed context. The strict cap forces active curation.
• L4 Curated knowledge base: The reference library. 20 Markdown files in ~/.hermes/knowledge, organized with one topic per file. Contains verified facts about me, systems, and operating boundaries, edited by hand and read on demand per domain.
• L5 Procedural skills library: The workshop tool rack. 980 SKILL.md files (249 distinct skills) across the router and 9 domain agents. Each skill defines activation triggers, step-by-step procedures, exit criteria, and runnable scripts. Skills represent executable procedures rather than raw facts, keeping them separate from L4.

Layer specifications

Curation, human approval gates, and rollback

Writes to persistent memory (L3) never go straight to disk. They pass through a structured validation pipeline with snapshots, so any incorrect fact can be rolled back instantly.

1. Pending queue: Proposed memory writes requiring review are saved as JSON records in ~/.hermes/pending/memory rather than applied directly.
2. Pre-filter: A fast gate checks candidate writes to discard transient session noise before deeper evaluation.
3. Review tier: Fable, the top reasoning tier, evaluates each candidate. Durable operational facts and user preferences are accepted. Routine task progress is logged only, since LCM-X already records full transcripts. Irreversible changes (such as deletions, security boundaries, or suppression lists) wait for my manual confirmation.
4. Snapshot backup: Before modifying memory, the curator copies MEMORY.md and USER.md to timestamped files in memories/backups (3 snapshots on disk today).
5. Atomic apply: Approved updates use atomic file operations (write to temporary file, fsync, and rename) behind file locks, ensuring no agent ever reads a partially written file. The memory tool strictly rejects updates exceeding the character limit until old entries are consolidated.
6. Audit ledger and recovery: A local SQLite ledger tracks the source session and reasoning behind every remembered fact. If an incorrect claim is detected, restoring the previous snapshot takes one command. Explicit forgetting leaves a clean audit tombstone.

In addition, an automated hourly job backs up agent state and memory files, and a daily guard script checks that knowledge edits stay within their assigned topic files.

LCM-X: lossless context memory

LCM-X operates beneath L1. Every message sent or received is recorded once in lcm.db with an immutable store_id. When the active context window fills up, LCM-X compacts older messages into a depth 0 summary node that references the exact store_ids it summarizes. When multiple depth 0 nodes accumulate, they compact into a depth 1 node. This creates a clean directed acyclic graph where summaries always link back to the exact source messages, and compaction never deletes or alters raw history.

• Compaction: The router agent's database currently holds 441,511 messages from 947 sessions and 990 summary nodes (922 at depth 0, 63 at depth 1, 5 at depth 2). Maximum depth currently in use: 2.
• FTS5 search: SQLite FTS5 full-text indexing indexes all messages and summary nodes. Search ranks results by keyword match strength and recency, allowing agents to pinpoint exact filenames, error strings, and identifiers instantly.
• Lossless expansion: If an agent needs full historical context beyond a summary, the lcm_expand tool fetches the exact source messages by store_id or node_id. Compaction reduces token load in the active prompt without losing historical fidelity.
• Concurrent WAL mode: SQLite Write-Ahead Logging allows background compaction jobs to write without blocking active agent reads.

Live telemetry and verification

Every figure on this page is either an architectural design constant or the output of a live, read-only system check probed on 1 Nov 2026.

Integrity check for the lossless DAG claim: every store_id and node_id referenced in a summary node must exist in the database. Result on 1 Nov 2026: 0 dangling references out of 50,569.

SELECT COUNT(*) FROM summary_nodes s, json_each(s.source_ids) j
WHERE (s.source_type = 'messages'
       AND NOT EXISTS (SELECT 1 FROM messages m WHERE m.store_id = j.value))
   OR (s.source_type <> 'messages'
       AND NOT EXISTS (SELECT 1 FROM summary_nodes n WHERE n.node_id = j.value));

Prototype vs production lineage

Key design decisions

• Hard character caps over soft limits: Soft warnings get ignored during complex tasks. Hard limits force active consolidation before new facts are saved.
• Plain Markdown over key-value databases: Human-readable, diffable, and directly editable with any standard text editor.
• SQLite WAL over append-only JSONL files: Transactional safety, non-blocking concurrent reads, and built-in FTS5 search in a single file.
• Exact keyword search over vector embeddings: Most developer and agent lookups require exact file paths, error codes, and entity names, where keyword search provides zero-hallucination precision.
• Preserving raw evidence: Summaries represent compact interpretations. The underlying source messages remain the ground truth evidence.

Failure modes and automated recovery

Stack

Hermes Agent · LCM-X · SQLite WAL · FTS5 · Python · Markdown