Dashboard/AI-OS (Second-Brain)/myos/hermes/hermes-docs-overview.md
AI-OS (Second-Brain)

myos/hermes/hermes-docs-overview.md

Hermes Agent — Docs Overview

Karpathy-wiki style: plain language, faithful to source, every fact tagged with where it came from.

Sources used: - [LOCAL] = /Users/arijitchowdhury/Dropbox/AI-Development/Personal/ChowMes/reference/hermes-agent-docs/ (snapshot downloaded 2026-06-15 — llms.txt, llms-full.txt, skills-catalogs.md). This is the reuse-first source per the project's own lookup policy. - [LIVE] = fetched this session via WebFetch from hermes-agent.nousresearch.com/docs/user-stories and .../developer-guide/architecture. Both pages exist in the local index (llms.txt) but their full body text isn't in the local snapshot beyond the architecture summary already captured under Developer Guide, so these two were fetched live as instructed.

Where a live fetch runs content through a small summarizing model, treat wording as paraphrase, not verbatim quote, even when it reads like a quote.

1. What Hermes Is

Hermes Agent is Nous Research's self-improving, terminal-native, autonomous coding-and-task agent. [LOCAL]

Its core pitch, verbatim from the local index:

"The self-improving AI agent built by Nous Research. A terminal-native autonomous coding and task agent with persistent memory, agent-created skills, and a messaging gateway that lives on 21+ messaging platforms — 19 native to the gateway plus IRC and Microsoft Teams via plugins (Telegram, Discord, Slack, SMS, Matrix, ...). Runs on local, Docker, SSH, Daytona, Modal, or Singularity backends. Works with Nous Portal, OpenRouter, OpenAI, Anthropic, Google, or any OpenAI-compatible endpoint." [LOCAL]

Repo: github.com/NousResearch/hermes-agent. Install via a one-line curl installer on Linux/macOS/WSL2/Termux. [LOCAL]

The defining traits, per the docs: - Persistent memory across sessions (MEMORY.md, USER.md, session search, and pluggable external memory providers like Honcho, Mem0, Supermemory). [LOCAL] - Agent-created skills — Hermes can write its own on-demand knowledge documents ("skills") and a background "Curator" process maintains them (usage tracking, staleness detection, archival). [LOCAL] - A messaging gateway so the same agent identity is reachable from Telegram, Discord, Slack, WhatsApp, Signal, SMS, Email, Matrix, Mattermost, Home Assistant, DingTalk, Yuanbao, Microsoft Teams, LINE, webhooks, or a custom OpenAI-compatible frontend. [LOCAL] - Multiple run targets: local machine, Docker, SSH into a remote box, Daytona, Modal, or Singularity (HPC-container) backends. [LOCAL] - Provider-agnostic: Nous Portal, OpenRouter, OpenAI, Anthropic, Google Vertex, or any OpenAI-compatible endpoint (self-hosted vLLM/Ollama included). [LOCAL]

Think of Hermes as: Claude Code's terminal-agent model, but built to be provider-neutral, multi-platform (it lives inside your group chats, not just your terminal), and self-extending (it writes its own skills and plugins over time).

2. Architecture

2.1 The core loop [LIVE — paraphrased from architecture page]

Everything routes through one orchestration object: AIAgent, implemented in run_agent.py. Three entry points feed it:

  1. CLI — interactive terminal session
  2. Gateway — the messaging-platform layer (Telegram, Discord, etc.)
  3. ACP — Agent Context Protocol, for editor integrations (VS Code, Zed, JetBrains)

Whatever the entry point, the same AIAgent.run_conversation() loop handles: provider selection → prompt assembly → the API call → the tool-dispatch loop → response delivery → session persistence. This is the "platform-agnostic core" design principle — one agent brain, many doors in.

2.2 Directory layout [LIVE]

Path What lives there
run_agent.py The main AIAgent conversation loop
agent/ Internal subsystems — prompt building, context compression, caching
tools/ Tool implementations + the central tool registry
hermes_cli/ CLI commands, configuration, auth
gateway/ ~20 messaging-platform adapters + session persistence
acp_adapter/ Editor integration
plugins/ Memory providers and context engines (plus general/platform/backend plugins — see §3)
cron/ The job scheduler for first-class scheduled agent tasks

2.3 Data flow, three ways [LIVE]

  • CLI session: user input → HermesCLI → AIAgent.run_conversation() → prompt assembly → provider resolution → API call → tool-dispatch loop → response display → session storage.
  • Gateway message: platform event → adapter → authorization check → session resolution → a fresh AIAgent instantiated with that session's history → response delivered back to the platform.
  • Cron job: scheduler tick → load job definition → fresh AIAgent → skill injection → execution → delivery to whatever platform the job targets.

2.4 Major subsystems [LIVE]

  • Agent Loop — the synchronous orchestration engine. Handles provider selection, prompt construction, tool execution, retries, fallback, callbacks, compression, persistence.
  • Prompt System — builds the system prompt in ordered tiers (identity/tools/skills → context files → memory/profile/timestamp blocks), using Anthropic-style prefix caching and lossy summarization once context crosses a threshold. [LOCAL confirms]: config docs describe a "Prompt Assembly" page and context-compression settings with a configurable auxiliary summarization model.
  • Provider Resolution — a shared runtime resolver that maps (provider, model) to (api_mode, api_key, base_url). Handles 18+ providers, OAuth, credential pools, alias resolution. [LOCAL] confirms a "Provider Runtime" developer-guide page and a "Credential Pools" feature.
  • Tool System — a central registry (tools/registry.py) holding 70+ tools across ~28 toolsets; each tool file self-registers on import. [LIVE]
  • Session Persistence — SQLite with FTS5 full-text search, lineage tracking across context-compression events, atomic writes with contention handling. [LIVE]; [LOCAL] independently documents session resume, export/prune commands, and per-platform session tracking.
  • Messaging Gateway — long-running process; 20 platform adapters; unified session routing; user authorization via allowlists + DM pairing codes; slash-command dispatch; the hook system; cron ticking; background maintenance. [LIVE], cross-checked against [LOCAL]'s Security page (pairing-code approval flow) and Messaging overview.
  • Plugin System — three discovery sources register tools, hooks, and CLI commands; memory providers and context engines are special single-select plugin types. [LIVE]; matched by the much more detailed [LOCAL] Plugins page (see §3).

2.5 Design principles, as stated [LIVE]

  • Prompt stability: the system prompt doesn't change mid-conversation except on explicit user action (this is what makes prefix caching pay off).
  • Observable execution: every tool call is visible via callbacks.
  • Interruptible: API calls and tool execution can be cancelled mid-flight.
  • Platform-agnostic core: one AIAgent serves CLI, gateway, and ACP alike.
  • Loose coupling: optional subsystems (memory providers, context engines, image/video backends) use registry patterns with feature-gating checks.
  • Profile isolation: each "profile" (see local docs on Profiles) keeps separate state and can run its own concurrent gateway process.

3. Plugin & Hook System [LOCAL — this is the most detailed section in the local snapshot]

3.1 What a plugin is

A plugin is a directory dropped into ~/.hermes/plugins/<name>/ with a plugin.yaml manifest plus Python code (__init__.py defines register(ctx); conventionally split into schemas.py / tools.py). Restart Hermes and the model can call any new tools immediately.

Discovery sources, later overriding earlier on name collision:

Source Path
Bundled <repo>/plugins/
User ~/.hermes/plugins/
Project .hermes/plugins/ (needs HERMES_ENABLE_PROJECT_PLUGINS=true)
pip hermes_agent.plugins entry points
Nix services.hermes-agent.extraPlugins

Security gate: general plugins and user-installed backends are disabled by default. Discovery finds them (they show up in hermes plugins / /plugins) but nothing with hooks or tools runs until its name is added to plugins.enabled in config.yaml. Bundled "always-works" infrastructure (default platform adapters, default image-gen backend, the single active memory provider, the single active context engine) is exempt from this gate — only arbitrary third-party code needs opt-in.

3.2 The ctx API inside register(ctx)

A plugin can: register a tool, register a hook, register a slash command, dispatch a tool programmatically, add a CLI subcommand, inject a message into the live conversation, ship data files, bundle a namespaced skill, require env vars (prompted at install), distribute via pip, register a whole gateway platform adapter, or register an image/video-generation backend, a context-compression engine, a memory backend, or an inference-provider backend.

3.3 The hook list

Hook Fires Return value used?
pre_tool_call Before any tool executes Yes — {"action": "block", "message": str} vetoes the call
post_tool_call After any tool returns No (observer)
pre_llm_call Once per turn, before the tool-calling loop Yes — {"context": str} gets appended to the user message
post_llm_call Once per turn, after the tool-calling loop (successful turns only) No
on_session_start New session, first turn only No
on_session_end End of every run_conversation() call + CLI exit No
on_session_finalize CLI/gateway tears down a session (/new, GC, quit) No
on_session_reset Gateway swaps in a fresh session key No
subagent_stop Once per child after delegate_task finishes No
pre_gateway_dispatch Gateway receives a message, before auth+dispatch Yes — {"action": "skip"\|"rewrite"\|"allow", ...}
pre_approval_request Dangerous command needs user approval, before the prompt is sent No
post_approval_response User responded to (or timed out on) an approval prompt No
transform_tool_result After any tool returns, before the result reaches the model Yes — str replaces the result, None leaves it
transform_terminal_output Inside the terminal tool, before truncation/ANSI-strip/redact Yes — str replaces raw output
transform_llm_output After the tool-calling loop completes, before the final response is delivered to the user Yes — str replaces the response text

General rule: callbacks take **kwargs for forward compatibility; if a callback crashes it's logged and skipped — a broken plugin can never break the agent.

Deep dive — pre_llm_call (the one used for memory/RAG injection):

def my_callback(session_id: str, user_message: str, conversation_history: list,
                is_first_turn: bool, model: str, platform: str, **kwargs):

Fires once per user turn (not once per API call inside a multi-tool-call loop), after context compression, before the main tool loop. Its return — a dict {"context": "..."} or a plain string — gets appended to the user message, never the system prompt, which is the trick that keeps prompt-cache hit rates high (system prompt stays byte-identical across turns). The injected text is ephemeral: never written back into session history, never persisted to disk. If multiple plugins return context, outputs are joined with double newlines in alphabetical plugin-discovery order. Canonical use case shown in docs: a memory-recall plugin that POSTs the user's message to an external memory API and returns the top hits as injected context.

Deep dive — transform_llm_output (the one used for output-rewriting / house-style / persona filters):

def my_callback(response_text: str, session_id: str, model: str, platform: str, **kwargs) -> str | None:

Fires once per turn, after the tool-calling loop completes and the model has produced its final text — before that text reaches the user on any surface (CLI, gateway, or a programmatic caller). Returning a non-empty string replaces the response text with no extra inference call; returning None/empty leaves it alone. First non-empty string wins across multiple registered plugins. Documented use cases: personality/vocabulary transforms (the docs' own example is a pirate/Spongebob filter), redacting user-identifying info from the final text, appending a signature footer, enforcing a house style without spending tokens on SOUL-file instructions. Guarded so it never fires on interrupts or empty turns; exceptions are logged as warnings and don't break the run.

Two more transform hooks worth flagging: transform_tool_result (rewrite any tool's return value before the model sees it) and transform_terminal_output (rewrite raw terminal output before Hermes's own truncation/ANSI-strip/redaction logic runs — good for collapsing a 50k-line find output into a 40-line summary).

3.4 Plugin types (4 kinds)

Type What Selection
General plugins tools, hooks, slash/CLI commands multi-select (enable/disable list)
Memory providers replace/augment built-in memory single-select
Context engines replace the built-in context compressor single-select
Model providers declare an inference backend multi-register, picked per-call via --provider/config

Two more extension surfaces sit outside the Python plugin system: gateway event hooks (drop a HOOK.yaml + handler.py into ~/.hermes/hooks/<name>/, fire on gateway:startup, session:start, agent:end, command:*) and shell hooks (declare a shell command under hooks: in config.yaml — no Python required, good for desktop notifications or audit logs).

3.5 Bundled plugins shipped in-tree

disk-cleanup (hooks + slash command, auto-cleans ephemeral files on session end), security-guidance (hooks; pattern-matches dangerous code on write_file/patch, 25 rules forked from Anthropic's claude-plugins-official), observability/langfuse and observability/nemo_relay (trace turns/LLM calls/tools to an external observability endpoint, fail-open if unconfigured).

4. The API

Hermes exposes an OpenAI-compatible HTTP API server so any frontend that speaks the OpenAI chat format (Open WebUI, LobeChat, LibreChat, NextChat, ChatBox, etc.) can use Hermes as a backend, with its full toolset (terminal, files, web search, memory, skills) behind the scenes. [LOCAL]

Quick start: set API_SERVER_ENABLED=true and API_SERVER_KEY=... in ~/.hermes/.env, run hermes gateway, and it listens on http://127.0.0.1:8642. [LOCAL]

POST /v1/chat/completions is the standard OpenAI Chat Completions shape — stateless, full conversation passed in messages each call, standard usage block returned. It also accepts inline image input (both http(s) and data:image/... URLs) in the content array. [LOCAL]

Streaming mode surfaces tool-progress indicators inline so a frontend can show what the agent is doing mid-turn. [LOCAL]

Beyond the API server, Hermes is also usable as an embeddable Python library (AIAgent directly, no CLI) and as an MCP client — it can connect to external MCP tool servers and auto-register their tools, with config-level filtering over which tools load (user-guide/features/mcp, reference/mcp-config-reference). [LOCAL] One of the live-fetched user stories also describes the inverse: wrapping Hermes itself as an MCP server so Claude Desktop/Cursor can call it (see §5, Dev Workflow #13).

5. Integrations

Grouped by surface, per the local index: [LOCAL]

  • Messaging platforms: Telegram, Discord, Slack (Socket Mode), WhatsApp (Baileys bridge), Signal (signal-cli), Email (IMAP/SMTP), SMS (Twilio), Matrix, Mattermost, Home Assistant, Webhooks (GitHub/GitLab event triggers), plus DingTalk, Yuanbao, Microsoft Teams, LINE per the top-level pitch.
  • Model providers: Nous Portal, OpenRouter, OpenAI, Anthropic, Google, Azure, Vertex AI, plus any OpenAI-compatible endpoint (self-hosted vLLM, Ollama, llama.cpp/MLX on Apple Silicon).
  • MCP (Model Context Protocol): Hermes as an MCP client, connecting to and filtering tools from external MCP servers.
  • ACP (Agent Context Protocol): Hermes inside ACP-compatible editors — VS Code, Zed, JetBrains.
  • Memory providers (pluggable, single-active): Honcho, OpenViking, Mem0, Hindsight, Holographic, RetainDB, ByteRover, Supermemory.
  • Skills catalog (bundled, ~90 skills) spans: Apple ecosystem (Notes/Reminders/FindMy/iMessage/computer-use), other coding agents as delegate targets (Claude Code, Codex, OpenCode), creative tooling (Excalidraw, p5.js, ComfyUI, manim), GitHub workflow, Google Workspace, Notion/Airtable/PowerPoint, research (arXiv, Polymarket, an "llm-wiki" skill directly analogous to this vault's own Karpathy-wiki pattern), and software-development discipline skills (TDD, systematic-debugging, code-review) that mirror this project's own superpowers:* skill set almost 1:1. A parallel optional skills catalog (~60 more) adds blockchain, finance/Excel modeling, MLOps, security/pentest, and more — installed on demand via hermes skills install official/<category>/<skill>. [LOCAL — skills-catalogs.md]

6. User Stories — Distilled (the inspiration list)

Arijit flagged these as very important for inspiration — this is the full list as extracted live from the docs, organized by the site's own category headings. Treat each line as a pattern to consider for AIOS, not a literal spec. [LIVE — summarized by WebFetch's model, so wording is paraphrase]

The live page is long and its final section (a short, low-signal "General" grouping) was truncated by the fetch tool's summarization step — flagged here rather than silently dropped.

Personal Assistant (24 stories) — patterns for AIOS

  • Scheduled digest/briefing generation delivered to chat (inbox summary at 9am → Slack; Turkish market/news briefing cards; daily research brief to Discord/Slack/Notion)
  • Agent as the front-end to a self-hosted stack (Nextcloud+LibreOffice as a Drive replacement)
  • Web research → build artifact → deploy loop (research a topic, build a landing page, ssh it to a VPS)
  • Health/journaling pipelines writing into Obsidian, using local (not cloud) models for privacy
  • Cross-tool task management spanning Obsidian + Apple Calendar + Signal, or Google Tasks
  • Voice-first, long-horizon personal coach (fitness: learns body/training/nutrition/recovery over time)
  • Weighted-scoring natural-language planners (meal planning by constraint scoring, not just chat)
  • Task-centric memory scoped to a specific operational domain (a printing factory's ops)
  • Proactive check-ins — the agent asks you "anything to watch this afternoon?" instead of waiting
  • Sensor/wearable ingestion (Whoop, Android Health Connect, an iOS companion app streaming health/location/voice)
  • Community/group bot with per-subgroup specialization (a horse-racing Telegram community)
  • Recurring generative task on a timer (twice-daily playlist curation)
  • ADHD-aware executive-function support — nudges, and a PM-style agent running literal daily standups with the user
  • One agent instance serving multiple family members with different individual use cases
  • "Always-on presence" via a device-native channel (iMessage tied to a always-on Mac Studio)

Dev Workflow (35 stories) — patterns for AIOS

  • Agent-to-agent supervision: one agent (Codex) watches another live agent's workflow and intervenes
  • A "converse mode" — agent thinks/proposes and requires human approval before acting, as an explicit low-autonomy mode
  • Rigorous token/cost profiling as its own discipline (one story found 73% overhead and fixed it)
  • Custom long-horizon memory kernels — several independent, serious builds: a 22k-line kernel compiling conversation into a temporal graph; a 3-layer memory system (Hindsight + Graphiti + MemPalace); a "no disk writes, straight to SQLite FTS5" variant
  • Exposing the agent itself as an MCP server so IDE-native tools (Claude Desktop, Cursor) can call it — inverts the usual "Hermes calls MCP servers" direction
  • Sub-agent delegation with a formal identity/token-exchange layer (RFC 8693-style "ZeroID")
  • Agent learning a developer's own code-review preferences over time (file patterns, flags) — a personalization loop, not a static ruleset
  • Full autonomous pipeline: plan → code → QA → ship, multi-agent
  • A dedicated "session compression" plugin whose only job is preserving the thread of ongoing work across a long session — distinct from generic context compression
  • A "memory compiler" that intercepts the LLM's own intermediate thoughts and compiles them into a structured DB (not just final outputs)
  • Kanban-style multi-agent coordination: parent posts cards, children pull/work/report
  • Self-improvement loop running literally as a cron job — a "skill audit skill" that reviews and refines other skills on a timer
  • Hooks used specifically to swap in a better tool at every opportunity the agent has to act — treating tool selection itself as improvable
  • Auto-restart-on-deploy via a local Gitea + Watchtower combo (10-minute SLA after a main-branch commit)
  • A "skill factory" plugin that watches live workflows and auto-generates new SKILL.md + plugin code from what it observes
  • A guardrail skill (STANDING.md) whose explicit job is stopping the agent from guessing — force a docs-check first
  • Background reasoning while idle, MCTS-powered ("Dream Auto")
  • Explicit narrative arc from a competing tool (OpenClaw) to converging on Hermes's own architecture — worth reading as a case study in what made the switch worth it

Integrations (19 stories) — notable for AIOS

  • Wrapping third-party code-intelligence tooling (tree-sitter via jMunch, 52 tools) directly into the agent's toolset
  • Using a sales-enrichment MCP (Composio → Hunter.io) for outreach — the agent doing real revenue-adjacent work, not just internal tasks
  • Cloud sandbox backends with snapshot persistence (Vercel Sandbox) — ephemeral compute that still remembers state
  • On-chain identity for the agent itself (Ethereum Attestation Service on Base mainnet) — agent-as-verifiable-entity
  • A dedicated computer-use module (NoVNC desktop + screenshots + mouse/keyboard) as a first-class integration, not a hack

Content Creation (13), Research (9), Enterprise (7), Business Ops (9), Cost Optimization (7), Trading & Markets (3), Marketing (3), Creative (6), Meta & Ecosystem (15), Messaging (6), Privacy & Self-Hosted (5), General (4)

Full titles preserved in the raw fetch (available on request); highest-signal patterns for AIOS from these buckets: - Content: learning a specific person's writing voice from their back catalog, then drafting new content in that voice (scripts → tweets → LinkedIn posts, chained) - Research: a "self-improving LLM Wiki" running on its own VPS with a Telegram bot and a public static site — structurally identical to what record-knowledge/llm-wiki already do here - Enterprise: session continuity across Kubernetes pod restarts (pod-hop handoff) — a durability pattern worth stealing regardless of platform - Business Ops: a "chief of staff" multi-agent pattern — one main agent + per-project sub-agents, exactly the shape of this CLAUDE.md's own skill-routing model - Cost Optimization: explicit smart-routing-by-tier saving real dollars — directly validates this project's own T1–T4 model-routing table - Meta/Ecosystem: a HUD-style TUI dashboard for watching the agent think in real time, and per-tool-call telemetry piped to Grafana — both are observability patterns AIOS currently lacks - Privacy: an explicit sandboxing recommendation ("run within sandbox, not free reign") as a named user story, not just a security appendix

7. What This Means for AIOS (synthesis, not from source)

A few structural ideas Hermes has already shipped that AIOS doesn't yet, worth stealing outright: 1. A formal hook taxonomy (pre_llm_call for context injection, transform_llm_output for output rewriting) is a cleaner separation of concerns than jamming everything into a system prompt. AIOS's CLAUDE.md-based routing is doing pre_llm_call's job by hand, every session, in natural language. 2. Ephemeral, cache-safe context injection (append to user message, never touch system prompt) is the specific mechanism that makes memory recall compatible with prompt caching. Any AIOS memory-recall feature should copy this shape. 3. A self-auditing skill running on a cron job (Dev Workflow #22) is exactly what code-auditor + /loop already approximate — validates that direction. 4. The "chief of staff" multi-agent shape (Business Ops #6) matches AIOS's own skill-routing table structurally — good external validation, not a new idea.

Everything else in §6 is raw material — pull specific stories back out when scoping a concrete AIOS feature.