Dashboard/AI-OS (Second-Brain)/myos/Storage-Architecture.md
AI-OS (Second-Brain)

myos/Storage-Architecture.md

MyOS — Storage Architecture (decision)

Decided 2026-07-01 from research (high confidence). Answers "how does personal Hermes store data, and how should MyOS store its knowledge + dashboard data." Companion to index and My-OS-Specs.

How Hermes itself stores data (fact, sourced)

Hermes = NousResearch/hermes-agent (github.com/NousResearch/hermes-agent). Persistence: - SQLite at ~/.hermes/state.db (WAL): sessions + messages (+ FTS5 tables for search). The only built-in store. - Memory = flat markdown at ~/.hermes/memories/ (MEMORY.md, USER.md), injected as a frozen snapshot into the system prompt at session start (not queried live). This is why editing MEMORY.md only affects NEW sessions, and why hermes sessions delete <id> resets a persona (the SOUL is baked into the session's messages). Matches memory feedback-hermes-soul-frozen-per-session. - Skills/config = plain files under ~/.hermes/ (HERMES_HOME overrides base). Plugin state = each plugin manages its own files (no built-in API). - Vector store = optional, OFF by default (9 pluggable providers: Mem0/Qdrant/pgvector/Hindsight/etc., one at a time). VPS hermes-prism runs default SQLite only.

So Hermes gives us sessions/messages (SQLite) + frozen-markdown memory. It does NOT give us a dashboard-grade queryable store or a vector index out of the box. MyOS must add that.

Markdown canonical + Postgres projection + pgvector, synced by a watcher, multi-tenant via RLS.

  1. Source of truth = Obsidian markdown (git-versioned). Humans edit it directly; agents write markdown first. Do NOT make a DB the sole source (that's the Notion/Linear model, wrong when knowledge starts in Obsidian and humans edit raw files). Do NOT run dashboard queries over flat files.
  2. Postgres = the queryable projection the dashboard reads. Relational tables: project status (project, stage enum, health enum, last_updated, owner, tenant_id) + a time-series progress log (one row per status change = the graph's data).
  3. pgvector on the SAME Postgres for the agent's retrieval (one row per doc chunk: embedding, tenant_id, content, source_path). No second vector DB below ~100M vectors. Embeddings: text-embedding-3-small (1536-dim) or nomic-embed-text if fully self-hosted.
  4. Multi-tenancy = shared schema + tenant_id column + Postgres Row-Level Security (SET app.current_tenant). Not schema-per-tenant, not DB-per-tenant (only for later regulatory contracts). Prefix the pgvector index with tenant_id. Clean migration path to promote a big tenant later.
  5. Sync = a file-watcher daemon (Node chokidar / Python watchfiles) beside the service: watch the vault, debounce ~1-2s, parse frontmatter+content, upsert to Postgres. The DB is a rebuildable projection (wipe + re-index from the vault in seconds). NOT write-through (couples every write to DB uptime), NOT git hooks (humans/agents don't commit on save).

Net flow: Obsidian markdown (git) = human truth → watcher daemon → Postgres + pgvector (RLS by tenant_id) → dashboard reads SQL, Hermes/Athena reads pgvector.

Why this fits us specifically

  • Reuses PIP's stack. PIP (prism_platform) already runs Postgres + alembic. MyOS's Postgres + the watcher can follow the same pattern, not a new stack.
  • Keeps the vault as the human face. Arijit lives in Obsidian; that stays the truth. The DB is derived, disposable, rebuildable.
  • Multi-tenant from day one, cheaply. tenant_id + RLS costs almost nothing now and avoids a rewrite when MyOS gets its second tenant.

To verify before building (caveat)

The exact state.db table/column names above come from Hermes public docs, not our VPS install. Before writing code against them, run on the box:

sqlite3 <HERMES_HOME>/state.db ".schema"     # e.g. ~/.hermes-prism/state.db

Sources

NousResearch/hermes-agent + hermes-agent.nousresearch.com/docs (session-storage, memory-providers); bytebase.com (multi-tenant DB patterns); tigerdata.com (multi-tenant RAG on Postgres); obsidian-index-service, obsidian-graph, MindMatrix (vault→DB/pgvector sync implementations).