The Storage Stack
Three embedded pieces, each owning what it is best at. Markdown is the source of truth; the other two are derived and rebuildable.
| Layer | Backed by | Holds | Rebuildable? |
|---|
| Markdown + YAML frontmatter | plain .md files | the memory content itself — the only portable, human-editable asset | ✗ (it is the truth) |
SQLite (aiosqlite) | .index/sqlite/*.db | system state, audit log, the cascade queue, the boundary buffer, OME engine state | ✓ from markdown |
| LanceDB (Arrow) | .index/lancedb/*.lance | vector + BM25 + scalar columns for retrieval | ✓ from markdown |
Delete the entire .index/ directory and no memory is lost — it rebuilds from the .md tree. There is no separate “export”; the markdown is the export. See Operating It to trigger a rebuild. Storage Paths
The default memory root is ~/.everos/ (override with EVEROS_MEMORY__ROOT or [memory] root in TOML). Configuration (the .env file) is separate from data (the memory root).
Memory is partitioned by <app_id>/<project_id> before the user-visible directories, so different (app, project) spaces never share a directory or cross in search. The reserved id "default" materialises as default_app / default_project on disk.
~/.everos/ ← memory root (EVEROS_MEMORY__ROOT)
├── default_app/ ← <app_id> ("default" → default_app)
│ └── default_project/ ← <project_id> ("default" → default_project)
│ ├── users/
│ │ └── <user_id>/
│ │ ├── user.md single-file (profile)
│ │ ├── episodes/
│ │ │ └── episode-<YYYY-MM-DD>.md daily-log append
│ │ ├── .atomic_facts/ daily-log (hidden)
│ │ │ └── atomic_fact-<YYYY-MM-DD>.md
│ │ └── .foresights/ daily-log (hidden)
│ │ └── foresight-<YYYY-MM-DD>.md
│ ├── agents/
│ │ └── <agent_id>/
│ │ ├── .cases/ daily-log (hidden)
│ │ │ └── agent_case-<YYYY-MM-DD>.md
│ │ └── skills/ skill-named dir
│ │ └── skill_<name>/SKILL.md (+ references/ scripts/)
│ └── knowledge/ ← shared / global (reserved)
│
├── .index/ ← system-managed, rebuildable (gitignore)
│ ├── sqlite/
│ │ ├── system.db state / audit / cascade queue / buffer / LSN
│ │ ├── ome.db Offline Memory Engine state
│ │ ├── ome.aps.db APScheduler jobstore
│ │ └── ome.db.lock OME single-engine guard
│ └── lancedb/
│ └── <kind>.lance/ one Arrow table per kind
│
├── ome.toml ← user-editable OME strategy overrides (hot-reloaded)
└── .tmp/ atomic-write staging
The index dir is .index/ (dot-prefixed), not _index/. The cascade queue and LSN/audit state live in SQLite (system.db, table md_change_state) — there is no .cascade.log or .manifest.json file. There is no everos reindex command — see Operating It. How a Memory Is Born
A message does not become memory immediately — it accumulates, a boundary is detected, an LLM extracts a cell, writers persist markdown, and the index catches up asynchronously.
POST /add ───▶ unprocessed_buffer (SQLite) ← messages accumulate per (session, app, project)
│
├─ boundary detector trips ──┐
POST /flush ─────────┤ (or you force it) │ one LLM call
│ ▼
│ extract MemCell ───▶ memcell row (SQLite)
│ │
│ ───────────────┴───────────────
│ ▼ ▼
│ UserMemoryPipeline (sync) AgentMemoryPipeline (fire-and-forget)
│ writes episode .md NOW emits AgentPipelineStarted
▼ │ │
(response returns once md is on disk) │
▼ ▼
────────────────── Offline Memory Engine (OME) ──────────────────
│ async strategies write derived .md: │
│ atomic_facts · foresight · user profile · agent cases · skills │
└───────────────────────────┬─────────────────────────────────────┘
▼
cascade daemon watches the .md tree
▼
md_change_state queue (SQLite, durable)
▼
rebuild LanceDB rows ───▶ searchable
/add appends messages to a per-(session_id, app_id, project_id) buffer and returns accumulated (or extracted if the boundary tripped on this call)./flush forces the boundary now (one extraction LLM call), used at the end of a chat/agent run.- Episode markdown is written synchronously — when
/flush returns extracted, the episode file is already on disk.
- Everything else (atomic facts, foresight, profile, agent cases/skills) is produced asynchronously by the OME.
- The cascade daemon turns every
.md write into LanceDB rows so the content becomes searchable.
Memory Types & Storage Strategies
Six memory kinds today, each picking one of three on-disk patterns:
| Kind | Owner | Dir / file | Strategy | Produced by |
|---|
| episode | user | episodes/episode-<date>.md | daily-log | extraction (sync) |
| atomic_fact | user | .atomic_facts/atomic_fact-<date>.md (hidden) | daily-log | OME |
| foresight | user | .foresights/foresight-<date>.md (hidden) | daily-log | OME |
| profile | user | user.md | single-file rewrite | OME |
| agent_case | agent | .cases/agent_case-<date>.md (hidden) | daily-log | OME |
| agent_skill | agent | skills/skill_<name>/SKILL.md | skill-named dir | OME (clustering) |
| Strategy | Shape | Why |
|---|
| Daily-log append | <prefix>-<YYYY-MM-DD>.md, one entry appended per memory | collapses thousands of per-entry files into one file per day |
| Single-file rewrite | a fixed filename overwritten in place | for a single evolving document (a user/agent profile) |
| Skill-named dir | one directory per skill | a skill is a richer unit (body + optional references/ scripts/) |
The Cascade Daemon
The cascade subsystem keeps LanceDB in sync with the markdown tree. It runs in-process with the server (a coroutine started by the app lifespan), not as a separate OS daemon.
- A native filesystem watcher (
watchdog: FSEvents on macOS, inotify on Linux) sees a .md create/modify.
- The change is enqueued in the
md_change_state table (SQLite) — durable, so a crash mid-sync replays on restart.
- A worker drains the queue at entry-level granularity: it diffs the file, re-embeds only changed entries (keyed by
content_sha256), and upserts the LanceDB rows.
Because markdown is the source of truth, editing a file directly is fully supported — open an episode in VSCode, Obsidian, or Vim, change an entry, save, and the daemon re-indexes just that entry.
The Offline Memory Engine (OME)
Most memory kinds are not extracted on the request path — they are derived later by the OME, an in-process async strategy engine. When extraction carves a MemCell, it emits an event; OME strategies pick it up and write their markdown when ready:
extract_atomic_facts — single-sentence facts from an episodeextract_foresight — anticipatory notesextract_user_profile — the aggregated user.mdextract_agent_case — a reusable agent trajectoryextract_agent_skill — clusters related cases into a named skill
Strategies are configurable without a code change via ome.toml at the memory root (hot-reloaded within ~2s). Example — disable two strategies:
[strategies.extract_foresight]
enabled = false
[strategies.extract_user_profile]
enabled = false
After /flush returns extracted, the episode is queryable soon (once cascade indexes it), but atomic facts / profile / agent cases appear only after their OME strategy runs — typically seconds later. Poll with backoff if you need them immediately.
Consistency Model
| Path | Guarantee | Detail |
|---|
Write (/add, /flush) | strong | the episode .md is on disk before the call returns extracted; never blocks on LanceDB |
Read (/search, /get) | eventual | reads LanceDB, which lags md by the cascade processing time — sub-second typically, up to ~10–15s under load |
/search immediately after the /flush that produced a record may miss it. The markdown is durable regardless; index lag never loses data. If you need read-your-write, retry with backoff, or force the queue with everos cascade sync.
Zero External Services
No database server, message broker, or vector service to run. Vector ANN, full-text BM25, and scalar filtering all execute inside the embedded LanceDB engine; SQLite is a local file. The whole stack is a single directory you can copy, back up, or check into git (user-visible parts only).
There is no automatic “grep over markdown” search fallback — if the LanceDB index is unavailable, rebuild it from markdown rather than relying on a degraded search path.
Operating It
| Command | What it does |
|---|
everos init | write a starter .env |
everos server start | run the HTTP API (cascade + OME start with it) |
everos cascade status | queue / LSN summary |
everos cascade sync | drain the cascade queue now (force md → LanceDB) |
everos cascade fix | list failed rows / re-enqueue retryable ones |
There is no everos reindex or everos flush CLI command.
- Reindex — the index is rebuildable: stop the server,
rm -rf <memory-root>/.index/lancedb, restart — the cascade rebuilds from markdown. For an incremental catch-up, use everos cascade sync.
- Flush is an HTTP endpoint (
POST /api/v1/memory/flush), not a CLI command — it forces extraction of the session buffer, which is different from forcing index sync (cascade sync).
