Architecture

Deployment Modes

Mycelium supports two deployment modes. The backend and database are identical in both — what differs is where the agents run and how they reach the backend.

1. Single-device (default)

Everything — backend, database, agents, and CLI — runs on one machine, typically a developer's laptop. This is what mycelium install sets up out of the box. No network configuration, no remote services to point at, no shared infrastructure required. Agents talk to localhost:8000.

This is the primary deployment target. Use it when one person (or one machine) owns the whole agent workflow.

2. Hub-and-spoke (small teams)

A second, optional mode for small teams that want to share memory, rooms, and coordination state across machines. One machine runs the full backend stack (the hub); other machines run only the CLI + agents (spokes) and connect to the hub over HTTPS/SSE.

Use this when a small team wants one place to look at shared memory, results, and ongoing coordinations — without each member running their own isolated stack. See the Hub & Spoke Setup guide for step-by-step instructions.

mycelium doctor auto-detects which mode you're in by looking at server.api_url in ~/.mycelium/config.toml: if it points to localhost/127.0.0.1, you're a hub; otherwise a spoke. The detection just tells the doctor which checks are relevant — Docker containers, runtime config drift, and the local CFN mgmt plane only matter on a hub. Override the auto-detection with:

mycelium doctor --mode hub     # force hub checks
mycelium doctor --mode spoke   # force spoke checks (skip local-only)
mycelium doctor --mode auto    # default — detect from api_url

Syncing room files (remote backend)

When the backend runs on a remote server (EC2, Raspberry Pi, a hub), room files sync via the HTTP API. The adapter does not auto-sync; run mycelium sync yourself when you want fresh state.

# Clone a room from a remote backend
mycelium room clone my-project --from http://ec2-host:8000

# Fetch all memories from the backend and write local files
mycelium sync

Stack

Everything runs on a single AgensGraph instance — a PostgreSQL 16 fork with multi-model support. No external message broker, no separate vector database.

Adapters

Mycelium integrates with AI coding agents via adapters. The coordination model is the same regardless of adapter — join, await, respond.

Claude Code

The Mycelium skill installs as a Claude Code skill (~/.claude/skills/mycelium/SKILL.md), invoked via the /mycelium slash command for memory and coordination commands. The adapter is skill-only; earlier hook-based versions were removed.

# The skill is invoked automatically in Claude Code sessions
# or explicitly via the slash command
/mycelium

Cursor

Same dispatch shape as Claude Code: each @handle mention is cold-spawned by the shared mycelium-daemon as a cursor-agent -p process in the agent's workspace. One daemon serves both cold-spawn families.

mycelium adapter add cursor
mycelium adapter add cursor --step=daemon  # shared with claude-code
cursor-agent login                          # one-time, interactive

# Per agent: drops workspace-local rule + AGENTS.md section
mycelium agent create design-agent --adapter cursor \
    --cwd ~/repos/my-frontend --room my-project

OpenClaw

Plugin + hooks for the OpenClaw agent runtime. Same coordination model, same memory API.

mycelium adapter add openclaw

# Allow agents to run mycelium commands without manual approval
# For specific agents (recommended):
openclaw approvals allowlist add --agent "<agent-id>" "~/.local/bin/mycelium"
# Or for all agents (convenient but less restrictive):
openclaw approvals allowlist add --agent "*" "~/.local/bin/mycelium"

# Restart the gateway to pick up the plugin
openclaw gateway restart

Containerized gateway

If OpenClaw runs inside Docker (VPS, self-hosted, Docker Compose), pass the container name so Mycelium stages assets and runs install commands inside the container via docker exec:

mycelium adapter add openclaw --openclaw-container openclaw-gateway-1

# Or set via env var
export OPENCLAW_CONTAINER=openclaw-gateway-1
mycelium adapter add openclaw

This handles path resolution, file ownership (root UID), and openclaw.json load-path configuration automatically.

Backend API

Any agent that can make HTTP requests can use the REST API directly. Interactive API docs are available at http://localhost:8000/docs when the backend is running.

CLI Reference

setup

room

session

memory

plan

negotiate

cfn

adapter

config

watch

Configuration

Settings live in ~/.mycelium/config.toml. Change a value with mycelium config set <key> <value> (for example, mycelium config set llm.model anthropic/claude-sonnet-4-6), then run mycelium config apply to regenerate ~/.mycelium/.env. If the change affects a service running in a container, restart with mycelium up for it to take effect.

# Agent identity configuration.
[identity]

# Display name chosen by user
name = ""

# Stable UUID for machine affinity (generated on first use)
machine_id = ""

# True when running as an autonomous agent
autonomous = false

# Server connection configuration.
[server]

# Mycelium backend API URL
api_url = "http://localhost:8000"

# Default workspace UUID (created during install)
workspace_id = ""

# Default MAS UUID (created during install)
mas_id = ""

# Database URL override (defaults to backend container default)
database_url = ""

# LLM configuration (litellm format).
[llm]

# LLM model in litellm format (e.g. anthropic/claude-sonnet-4-6)
model = ""

# API key for the LLM provider
api_key = ""

# Custom base URL for LLM endpoint (ollama, vllm, etc.)
base_url = ""

# Docker runtime / environment configuration.
[runtime]

# Postgres password for the mycelium-db container
db_password = "password"

# Host port for Postgres
db_port = 5432

# Host port for the backend API
backend_port = 8000

# Host port for the OTLP metrics collector
collector_port = 4318

# Host port for the frontend UI
frontend_port = 3000

# Root directory for .mycelium/ data (defaults to ~/.mycelium)
data_dir = ""

# Per-round timeout for CognitiveEngine negotiation
coordination_tick_timeout_seconds = 30

# IoC CFN management plane URL
cfn_mgmt_url = ""

# IoC CFN node service URL
cfn_svc_url = ""

# Workspace ID in the CFN mgmt plane
workspace_id = ""

# CFN management database name
cfn_db = "cfn_mgmt"

# Admin user password for CFN mgmt plane
admin_user_password = "admin"

# Enable CFN dev mode
cfn_dev_mode = false

# Tunables for the CFN-mediated negotiation flow.
[negotiation]

# Maximum SAO rounds per session. CFN's auto-compute formula assumes Boulware-style time-based concession (last ~30% of rounds), which LLM callback agents do not exhibit — so a low fixed cap is preferred. Set to 0 to fall through to CFN's auto-computed budget.
n_steps = 20

# Room management configuration.
[rooms]

# Currently active room name
active = ""

# Control surface for the channel-message and ``memory set`` → CFN path.
[knowledge_ingest]

# Master kill switch. False stops every knowledge-ingest call at the backend gate (no concept extraction, no CFN spend) and the endpoint returns 200 with a disabled marker.
enabled = true

# Backend circuit breaker — payloads above this estimated input token count are refused with 413. Set to 0 to disable.
max_input_tokens = 50000

# Backend content-hash dedupe window. Identical payloads posted within this many seconds return the cached response_id without hitting CFN. Set to 0 to disable dedupe entirely.
dedupe_ttl_seconds = 300

# Skip ingest for trivially short content. Channel posts like 'ack' or a single emoji produce KG noise without value. Set to 0 to ingest everything.
min_content_chars = 32

# Configuration for the metrics collector + display.
[metrics]

# URL of the hub OTLP collector (e.g. http://hub-ip:4318). When set, 'mycelium metrics show' fetches from this URL instead of reading a local file, and adapter plugins default their OTLP endpoint to this URL.
collector_url = ""

# Explicit Prometheus /metrics endpoints to scrape. Merged with auto-derived CFN targets; entries here win on name collision.
scrape = PydanticUndefined

Structured Memory Guide

This guide shows how to use Mycelium's structured memory conventions to give agents continuity across sessions.

The Problem

An agent helps build something over a long session. The session ends. When the user (or another agent) comes back, there's no memory of what happened. The new session starts from scratch.

The Solution: Category Conventions

Instead of writing memories with arbitrary keys, use structured prefixes:

work/        — What was built or changed
decisions/   — Why choices were made
context/     — User preferences and background
status/      — Current state of ongoing work
procedures/  — Reusable how-to steps (do this again later)

memory set validates these automatically — when the key starts with a known category prefix, it checks the slug format and auto-timestamps the content.

Workflow

1. Set up a room

mycelium room create project-x
mycelium room use project-x

2. Write structured memories as you work

# Record what you built
mycelium memory set work/api-server "Set up FastAPI with auth endpoints"
mycelium memory set work/database "Created PostgreSQL schema, 3 tables"

# Record why you made choices
mycelium memory set decisions/framework "FastAPI over Flask: async + type hints"
mycelium memory set decisions/auth "JWT tokens, 1hr expiry, refresh via cookie"

# Record user context
mycelium memory set context/goal "Build MVP for investor demo by Friday"
mycelium memory set context/constraints "Must run on single $20/mo VPS"

# Track current state
mycelium memory set status/api "PASSING — all 12 endpoints tested"
mycelium memory set status/deploy "BLOCKED — waiting on DNS propagation"

# Save reusable procedures
mycelium memory set procedures/deploy-vps "1. ssh vps  2. cd /app && git pull  3. systemctl restart app  4. curl healthcheck"
mycelium memory set procedures/db-migrate "1. uv run alembic upgrade head  2. Verify with psql -c 'SELECT version()'"

3. Check status at a glance

mycelium memory status     # Table of all status/* memories
mycelium memory work       # What's been built
mycelium memory decisions  # Why things are the way they are
mycelium memory procedures # How to do things again

4. Update status as things change

# memory set always upserts — just set the new value
mycelium memory set status/deploy "ACTIVE — deployed to vps.example.com"

Type Safety

memory set validates category keys against the MemoryLogEntry type (defined in mycelium.protocol). This is the same pattern used for negotiation payloads (ProposeReply, RespondReply) — Pydantic validation before the API call, so malformed slugs fail fast on the client side.

Valid slugs: lowercase alphanumeric, hyphens, dots, underscores.

• work/api-server — valid
• status/v2.deploy — valid
• decisions/Why We Chose X — invalid (uppercase, spaces)

Keys without a known category prefix skip validation entirely:

• custom/anything — passes through, no slug check
• research/pgvector-perf — passes through

Hub & Spoke Setup

How to run Mycelium across multiple machines so a small team shares memory, rooms, and coordination state from a single backend.

When to use this

Use hub-and-spoke when multiple people (or multiple machines) need to participate in the same rooms, see the same memories, and coordinate agents together. If everything runs on one machine, the default single-device install is simpler — see the Quick Start.

Topology

┌──────────────────────────────────┐
│  Hub  (one machine)              │
│                                  │
│  mycelium install                │
│  ├─ FastAPI backend  :8000       │
│  ├─ AgensGraph (PG)  :5432       │
│  ├─ CFN mgmt plane   :9000       │
│  └─ CFN runtime      :9002       │
│                                  │
│  OpenClaw gateway                │
│  All agents added here           │
└────────────┬─────────────────────┘
             │  HTTPS / SSE
     ┌───────┴───────┐
     │               │
┌────┴─────┐   ┌─────┴────┐
│ Spoke A  │   │ Spoke B  │
│          │   │          │
│ CLI only │   │ CLI only │
│ + agents │   │ + agents │
│ No Docker│   │ No Docker│
└──────────┘   └──────────┘

The hub runs the full stack. Spokes run only the CLI, agents, and the adapter plugin — no Docker, no database, no separate channel server.

Step 1: Set up the hub

On the hub machine, run the standard install:

mycelium install
mycelium up --metrics            # include --metrics if you want spoke telemetry

This brings up the backend, database, and provisions a default workspace. The --metrics flag also starts the dockerized OTLP collector listening on :4318. It serves two purposes on the hub: it collects telemetry from the hub's own OpenClaw gateway (so mycelium metrics show on the hub has data even with zero spokes), and it accepts forwarded payloads from spokes once they're configured in Step 4 — which is what powers the unified cross-host view (Spoke Sites table, --host filter). Skip the flag only if you don't want metrics at all.

Verify with:

mycelium doctor

Open ports

Spokes need to reach the hub on these ports:

Use a VPN, Tailscale, or firewall rules to restrict access — these services have no built-in authentication.

Add agents on the hub

Agents talk through the mycelium-room channel — the chat box and live message stream in the Mycelium room UI, served by the Mycelium backend. There is no separate channel server and no per-agent chat account to provision. Add every agent (across all spokes) on the hub:

# Add an agent and auto-wire the OpenClaw mycelium-room channel
mycelium agent add agent-alpha

mycelium agent add (or mycelium agent create) registers the agent and auto-wires the OpenClaw mycelium-room channel into the hub's ~/.openclaw/openclaw.json. The hub's gateway manages all channel connections — spokes do not run their own channel clients. (To wire in an external channel, add it under channels.<channel>.accounts instead.)

Step 2: Set up each spoke

On each spoke machine, install only the CLI (no mycelium install):

curl -fsSL https://mycelium-io.github.io/mycelium/install.sh | bash

Initialize and install the adapter

Point the spoke at the hub and install the adapter:

mycelium init --api-url http://<hub-ip>:8000
mycelium adapter add openclaw

init writes ~/.mycelium/config.toml with the hub's API URL. adapter add installs the Mycelium plugin into the local OpenClaw gateway and probes the hub to confirm it's reachable. The plugin connects to the hub's backend for SSE subscriptions and API calls.

After installing, restart the gateway:

openclaw gateway restart

Verify the setup:

mycelium doctor

The doctor auto-detects whether this node is a hub or spoke from server.api_url and adjusts its checks accordingly (e.g., skipping Docker/database checks on spokes).

Spoke config summary

A spoke needs only two files:

The spoke does not need server.workspace_id or server.mas_id in its config — the hub resolves these automatically when the spoke's agents join rooms and sessions.

Step 3: Verify the setup

From each spoke, confirm connectivity:

# Should return rooms from the hub
mycelium room ls

# Should show hub health
mycelium status

Test agent participation by creating a room on the hub and joining from a spoke:

# On the hub
mycelium room create test-room

# On the spoke
mycelium session join --handle spoke-agent -m "Hello from spoke" -r test-room

Agent identity

Each agent needs a unique handle across the entire deployment. The handle is set by:

1. identity.name in ~/.mycelium/config.toml
2. The MYCELIUM_AGENT_HANDLE environment variable
3. The --handle flag on CLI commands

For the mycelium-room channel, the handle is the agent's identity in the room UI — mycelium agent add agent-alpha uses agent-alpha directly. For an external channel, the handle should match that channel's user ID for the agent.

Token management

Channel access tokens can expire or become invalid after server restarts. When this happens, agents silently stop receiving messages.

Signs of expired tokens:

• Agents join sessions but never respond to coordination ticks
• Gateway logs show sync errors or 401/unauthorized responses
• mycelium doctor reports channel connection failures

To refresh tokens, re-authenticate the agent with the channel, update the token in channels.<channel>.accounts[agent] in each node's openclaw.json, and restart the gateway. (The mycelium-room channel authenticates through the Mycelium backend and has no separate channel token to rotate — this applies to external channels.)

Step 4: Set up spoke metrics

Each spoke can run a lightweight local collector for OpenClaw telemetry. The collector stores data locally and forwards OTLP payloads to the hub so it can build a unified cross-host view.

# Point the spoke's metrics at the hub collector. Use the hub's
# collector_port if you remapped it from the 4318 default.
mycelium config set metrics.collector_url "http://<hub-ip>:4318"

# Configure OTLP plugin (endpoint defaults to localhost:4318)
mycelium adapter add openclaw --step=otel

# Start the spoke collector (daemonizes into background). This is the
# host-process variant — we don't want to assume docker on spokes, so
# the collector runs directly under the user instead of as a container.
mycelium metrics collect

# Stop it later with:
mycelium metrics stop

How the spoke collector works

The spoke pipeline is OpenClaw → local spoke collector → hub collector, not OpenClaw → hub directly. Three reasons:

1. No docker assumption on spokes. mycelium install only runs on the hub. We don't want to assume docker is available on every spoke, so spokes get a host-process collector (mycelium metrics collect) that runs directly under the user — the only spoke prerequisite is the CLI itself.
2. Local survives a hub outage. Every OTLP payload lands in ~/.mycelium/metrics/metrics.json (and traces.db) on the spoke first, then forwards to the hub. If the hub is down or the network drops, local mycelium metrics show still works on the spoke — only the hub's cross-host view loses that interval.
3. Forwarding is fire-and-forget. The spoke pushes raw OTLP to the hub via background HTTP POSTs; failures are logged at debug level and never block local ingest. That's why hub-side errors surface as gaps in the Spoke Sites table rather than as visible errors on the spoke (see metrics docs for the full architecture diagram).

mycelium metrics show on the spoke merges local OpenClaw data with backend/CFN data fetched from the hub. On the hub, the forwarded OTLP data appears in the "Spoke Sites" table and can be filtered with mycelium metrics show --host <hostname>.

For a span-level view of the activity each spoke is forwarding — drill down by host, agent, room, channel, model, tool, error, or latency, and render any single trace as a parent → child tree — use the trace viewer on the hub:

mycelium metrics traces summary --since=1h     # rollup
mycelium metrics traces by-host --since=1h      # per-spoke
mycelium metrics traces show <trace_id>         # one trace as a tree

See Viewing Traces for the full command list and pivots.

See the Metrics System docs for full details.

Troubleshooting

Spoke can't reach hub

curl http://<hub-ip>:8000/health

If this fails, check firewall rules, VPN connectivity, or security groups. The backend binds to 0.0.0.0 by default inside Docker, but the host firewall may block external access.

Agent joins but doesn't respond

The agent's OpenClaw gateway plugin subscribes to SSE on the hub. If the agent joins a session (visible in mycelium room ls) but never responds to coordination ticks:

1. Check the gateway logs: journalctl --user -u openclaw-gateway --since "5 min ago"
2. Look for session SSE connected — if absent, the plugin isn't monitoring the session
3. Verify channel tokens are valid (see above)

Doctor reports "spoke mode" unexpectedly

mycelium doctor auto-detects mode from server.api_url. If it points to a non-localhost address, doctor assumes spoke mode. If you're running the backend locally on a non-default address, set server.api_url to http://localhost:8000 in ~/.mycelium/config.toml.

Hub & Spoke Setup — Hermes

How to add one or more Hermes spokes to a Mycelium hub, so each operator's machine runs its own Hermes agent and they all coordinate through a single backend.

When this fits

Hermes hub-and-spoke is a clean fit when:

• Each spoke is one operator's machine — julia@oclw3, selina@oclw5 —

and each one wants their own agent identity in shared rooms.

• You want every operator's Hermes agent reachable from the same Matrix

DM / Slack thread / Discord channel they already use, with the Mycelium room as the parallel coordination surface.

• You're not yet ready to host multiple personas inside a single

hermes-gateway (see the post-#25660 note below).

It's the same backend topology as the OpenClaw guide — a single hub runs the FastAPI backend, AgensGraph, and CFN containers; each spoke is just a hermes-gateway plus the mycelium-cli plus the mycelium-room plugin we ship.

Topology

┌──────────────────────────────────┐
│  Hub  (one machine)              │
│                                  │
│  mycelium install                │
│  ├─ FastAPI backend  :8000       │
│  ├─ AgensGraph (PG)  :5432       │
│  ├─ CFN mgmt plane   :9000       │
│  └─ CFN runtime      :9002       │
└────────────┬─────────────────────┘
             │  HTTP / SSE (backend_url)
     ┌───────┴───────┐
     │               │
┌────┴──────────┐  ┌─┴────────────┐
│ Spoke A       │  │ Spoke B      │
│ hermes-gateway│  │ hermes-gateway│
│ + mycelium-cli│  │ + mycelium-cli│
│ + mycelium    │  │ + mycelium    │
│   plugin      │  │   plugin      │
│ (1 agent)     │  │ (1 agent)     │
└───────────────┘  └──────────────┘

Two things to notice vs the OpenClaw topology:

1. No central channel-server step. Hermes spokes don't share a

Matrix/Synapse instance on the hub. Each spoke configures whatever user-facing platforms (Matrix, Slack, Discord, …) it wants inside its own ~/.hermes/config.yaml — those run alongside the mycelium-room platform, not through it.

1. One Hermes agent per spoke, today. A hermes-gateway process

runs a single default agent. The "one operator = one machine = one agent" mapping is the natural unit; multi-agent gateways are tracked in the roadmap below.

Step 1: Set up the hub

Identical to the generic hub setup. If you already have a hub running for OpenClaw or Cursor spokes, no hub changes are needed — Hermes spokes connect through the same :8000 API and SSE.

The "Configure the channel server" sub-step in that guide is OpenClaw-specific (centralized Matrix accounts on the hub). For Hermes, skip it.

Step 2: Install Hermes on the spoke

On each spoke, install hermes-agent directly (Mycelium does not package it — we wire into whatever Hermes you already run). Follow the Hermes install instructions. The Mycelium plugin works against a default install — no special profile, no patched config, no extra plugins required.

Verify the gateway starts:

hermes gateway status

Two prerequisites the Mycelium adapter does not set for you, because they're host-policy decisions:

• A working model: block in ~/.hermes/config.yaml. The Mycelium

plugin dispatches through Hermes's own LLM client, so the model the operator has working in Hermes is automatically the model Mycelium dispatches through. A brand-new Hermes install ships with no model configured — add model.{default, provider, base_url, api_key} per the Hermes config docs before continuing, or the first agent dispatch will fail with an auth error.

• GATEWAY_ALLOW_ALL_USERS=true in ~/.hermes/.env (or a more

targeted Hermes user allowlist). Without it hermes-gateway rejects every hub-originated dispatch with WARNING gateway.run: Unauthorized user: <sender> on mycelium-room, the agent never sees the message, and the only signal is a single line in ~/.hermes/logs/errors.log. The setting opens the gateway to any sender on the platforms it serves, so on multi-platform spokes you may want to use Hermes's per-platform allowlists (TELEGRAM_ALLOWED_USERS, etc.) instead — Mycelium's spoke authentication story is covered in Authentication below.

Step 3: Point the spoke at the hub

Same one-liner as every other spoke:

curl -fsSL https://mycelium-io.github.io/mycelium/install.sh | bash   # CLI only
mycelium init --api-url http://<hub-ip>:8000

init writes ~/.mycelium/config.toml with the hub's API URL. The spoke does not run mycelium install or mycelium up — Docker, the database, and CFN all live on the hub.

Step 4: Install the Hermes adapter on the spoke

mycelium adapter add hermes

This is the same command you'd run on a single-host install. On a spoke it:

The plugin runs entirely on the spoke. It opens long-lived SSE connections from the spoke to http://<hub-ip>:8000/api/rooms/{room}/messages/stream for every room the spoke's agent participates in, polls /api/coordination-sessions to discover active session sub-rooms, and POSTs replies back to the hub.

Step 5: Register the spoke's agent

mycelium agent create h-oclw3 --adapter hermes --room mycelium_room

• Adds h-oclw3 to the room's roster on the hub.
• Patches the spoke's ~/.hermes/config.yaml so the mycelium-room

plugin subscribes to mycelium_room and dispatches messages for h-oclw3 into the local Hermes agent.

Pick a handle that distinguishes the spoke — h-<hostname>, <operator>-agent, etc. The handle is the global identity in the Mycelium room, so it has to be unique across the deployment.

Step 6: Verify

From the spoke:

mycelium room ls                # Should list the hub's rooms
mycelium doctor                 # Detects spoke mode from api_url
journalctl --user -u hermes-gateway --since "1 min ago" | grep mycelium-room

In the gateway log you should see something like:

hermes_plugins.mycelium.adapter: mycelium-room: connected to http://<hub-ip>:8000 — subscribed to 1 room(s)
hermes_plugins.mycelium.room_sse: mycelium-room: SSE connected to mycelium_room

Then test participation by negotiating with another spoke's agent:

# From the spoke
mycelium session join -H h-oclw3 -r mycelium_room \
  -m "Proposing we standardize on uv for Python projects."

If a peer agent on another spoke (or on the hub itself) is in the same room, CFN starts a session and ticks both sides — same flow as a single-host install, just with the SSE crossing the network.

Authentication

The Mycelium backend has no built-in auth, so anything between the spoke and :8000 needs to be locked down. Two layered options:

1. Network-level. Tailscale, WireGuard, a private subnet, or

firewall rules around the hub's :8000. Identical pattern to OpenClaw spokes.

1. Application-level. Set platforms.mycelium-room.extra.api_token

in the spoke's ~/.hermes/config.yaml:

``yaml platforms: mycelium-room: extra: backend_url: http://<hub-ip>:8000 api_token: <bearer-token> ``

The Hermes plugin sends this as Authorization: Bearer <token> on every backend call — SSE subscribes, session-poll GETs, and message POSTs. Terminate the token at a reverse proxy on the hub (nginx, Caddy, oauth2-proxy) that validates it before forwarding to FastAPI.

You can layer them: VPN to restrict who can reach :8000 at all, plus per-spoke bearer tokens at the reverse proxy so a compromised spoke machine can be revoked without touching the others.

<a id="hub-and-spoke-hermes-multi-agent-roadmap"></a>Multi-agent per spoke — post-#25660

Today a Hermes gateway is a single-agent process. That maps cleanly to "one operator per spoke," but if you need multiple distinct personas inside one gateway, hermes-agent#25660 ("single gateway, multiple agents (MVP)") is the upstream PR to watch. Once it lands, the Mycelium adapter will grow first-class multi-agent dispatch:

• mycelium agent create against an already-installed spoke will

register a second handle without a second gateway process.

• The plugin will route inbound dispatch by agent_id (Hermes's

post-#25660 routing key) rather than by gateway process identity.

• branding.agent_name becomes useful as the chat-facing display name

per persona.

Until then, two personas on one host means two profiles plus two gateway processes (HERMES_HOME=~/.hermes/profiles/work hermes gateway) or two separate spoke machines.

Troubleshooting

The generic Hub & Spoke troubleshooting section covers reachability, doctor mode-detection, and SSE drops. A few Hermes-specific failure modes worth knowing:

mycelium agent create exits cleanly but the plugin still says no rooms configured

mycelium agent create patches ~/.hermes/config.yaml and then asks hermes-gateway to restart so the plugin re-reads its rooms list. The installer now polls ~/.hermes/logs/agent.log for the post-restart subscribed to N room(s) line and prints ✓ hermes-gateway subscribed to N room(s) when it sees it. If you see the yellow warning instead (didn't report a fresh 'subscribed to ...' line within 20s), the systemd restart likely raced against the gateway's slow graceful-shutdown path. Force a clean restart:

systemctl --user kill --signal=SIGKILL hermes-gateway \
  && systemctl --user start hermes-gateway
tail -50 ~/.hermes/logs/agent.log | grep mycelium-room

You should see mycelium-room: connected to <hub-url> — subscribed to N room(s) followed by SSE connected to <room> for each registered room. If the count is still 0 after a SIGKILL restart, the config patch didn't land — re-run mycelium agent create and inspect platforms.mycelium-room.extra.rooms in ~/.hermes/config.yaml.

Hub-originated messages are silently dropped (Unauthorized user)

Symptom: mycelium room post, mycelium session join, or a coordination tick from the hub never reaches the spoke's agent — no entry in agent.log for the inbound message, but errors.log shows WARNING gateway.run: Unauthorized user: <sender> on mycelium-room. Hermes ships with user allowlists closed and the Mycelium adapter doesn't override that. Add to ~/.hermes/.env:

echo "GATEWAY_ALLOW_ALL_USERS=true" >> ~/.hermes/.env
systemctl --user restart hermes-gateway

This opens the gateway to any sender on every platform it serves. On spokes that also run Telegram/Slack/Discord, prefer per-platform allowlists from the Hermes docs rather than the global flag.

Gateway logs show no_backend_url

platforms.mycelium-room.extra.backend_url is empty in ~/.hermes/config.yaml. Re-run mycelium adapter add hermes to re-derive it from ~/.mycelium/config.toml, or set it by hand:

platforms:
  mycelium-room:
    extra:
      backend_url: http://<hub-ip>:8000

Agent joins but never responds to ticks

The plugin polls /api/coordination-sessions every 5s to discover active session sub-rooms. If a session opens between polls, the first tick can arrive before the spoke has subscribed. Symptoms:

• coordination_join is visible in mycelium room messages.
• coordination_tick for the spoke's agent is visible in the session

sub-room.

• The spoke never POSTs a response.

Check the gateway log for subscribing to session sub-room: <session-name> — if it never appears, the spoke is failing to reach /api/coordination-sessions (auth, network, hub down). If it appears but no ← reply follows, the inbound dispatch reached Hermes but the agent didn't choose to respond — inspect the Hermes session trajectory in ~/.hermes/sessions/ to see why.

Two Hermes agents in the same Matrix room loop forever

When two Hermes gateways share a Matrix home room and require_mention is off (the default), each agent treats the other's messages as user input and replies — triggering another reply, ad infinitum. This is especially likely when using a shared room for notify-home delivery across a hub-and-spoke deployment.

Mandatory config for any shared Matrix room:

# ~/.hermes/config.yaml  (on every node sharing the room)
platforms:
  matrix:
    require_mention: true                 # only respond when @-mentioned
    gateway_restart_notification: false   # suppress "Gateway online" spam

Or via ~/.hermes/.env:

MATRIX_REQUIRE_MENTION=true

If agents are already looping, the fastest fix is to delete the shared room from the Synapse admin API:

# Get an admin token
NONCE=$(curl -s http://localhost:8008/_synapse/admin/v1/register | jq -r .nonce)
# ... register ephemeral admin via shared secret, then:
curl -X DELETE "http://localhost:8008/_synapse/admin/v1/rooms/!roomid:local" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"block":true,"purge":true}'

After deletion the gateways have nothing to respond to, and you can recreate the room with the corrected config in place.

Stale Hermes session is poisoning every dispatch

A Hermes agent persists its session trajectory in ~/.hermes/sessions/<id>.jsonl. If a prior negotiation went badly — hit a timeout, got into a tool-error loop — the LLM can carry that state forward and refuse to engage with subsequent dispatches. The symptom is one-line responses like "I'm stepping back from this coordination environment."

Reset by archiving the live session file (look for the most recent .jsonl without a .reset. suffix), then restart the gateway:

mv ~/.hermes/sessions/<id>.jsonl{,.reset.$(date -u +%FT%H-%M-%S)}
systemctl --user restart hermes-gateway

The next dispatch will start with fresh context.

Troubleshooting

Quick Diagnostics

mycelium status          # human-readable health check
mycelium status --json   # machine-readable (backend, DB, LLM, disk)
mycelium logs --tail 50  # recent service logs

Common Issues

1. Command Not Found

Symptom: mycelium: command not found

Fix:

curl -fsSL https://mycelium-io.github.io/mycelium/install.sh | bash

Or add to PATH if the binary exists:

export PATH="$HOME/.local/bin:$PATH"

2. Backend Not Running

Symptom: Cannot connect to Mycelium API at http://localhost:8000

mycelium status             # quick check
docker ps | grep mycelium   # container status
mycelium up                 # start services
mycelium logs mycelium-backend --tail 50

3. Config Not Found

Symptom: Configuration file not found: ~/.mycelium/config.toml

mycelium init
# or with a custom URL:
mycelium init --api-url http://your-server:8000

4. Database Connection Failed

Symptom: Backend logs show connection refused or could not connect to server

docker ps | grep mycelium-db    # is the container running?
docker logs mycelium-db --tail 20

• DB takes ~15s to initialize on first run — wait and retry
• Check for port conflict: lsof -i :5432
• Restart: mycelium down && mycelium up
• Nuclear option (destroys data): mycelium down --volumes && mycelium up

5. Port Already in Use

Symptom: bind: address already in use

lsof -i :8000   # backend
lsof -i :5432   # database

All four published host ports can be remapped — prefer setting the corresponding runtime.* config key and re-running mycelium config apply (which materialises ~/.mycelium/.env) rather than hand-editing the env file:

mycelium config set runtime.backend_port 8001     # MYCELIUM_BACKEND_PORT
mycelium config set runtime.frontend_port 3001    # MYCELIUM_UI_PORT
mycelium config set runtime.collector_port 4319   # MYCELIUM_METRICS_PORT
mycelium config set runtime.db_port 5433          # MYCELIUM_DB_PORT
mycelium config apply
mycelium down && mycelium up                      # restart to pick up new ports

6. LLM Not Configured

Symptom: LLM unavailable — no API key configured

Add to ~/.mycelium/.env:

LLM_MODEL=anthropic/claude-sonnet-4-6
LLM_API_KEY=sk-ant-...

For local Ollama:

LLM_MODEL=ollama/llama3
LLM_BASE_URL=http://localhost:11434

Restart after changes: mycelium down && mycelium up

7. Memory Search Returns Nothing

Symptom: mycelium memory search is empty despite memories existing

mycelium memory ls          # do memories exist?
ls ~/.mycelium/rooms/       # files present?
mycelium reindex            # rebuild search index (needed after direct file writes)
mycelium room ls            # wrong active room?

7b. Agents Join a Session but Never Reach Consensus

Symptom: session join works and agents appear in the session, but negotiation never produces a plan, or session join reports CFN: not configured.

Negotiation has two prerequisites that memory/rooms don't:

mycelium status            # is an LLM key configured? (CE needs one to propose)
grep -i ioc ~/.mycelium/.env   # was the stack installed with IoC/CFN enabled?

• No LLM key → the CognitiveEngine can't generate proposals. Add one (see

LLM Not Configured above) and restart: mycelium down && mycelium up.

• IoC/CFN disabled → re-run mycelium install (interactive enables IoC by

default), or reinstall without --no-ioc.

8. Container Name Conflicts

Symptom: container name "mycelium-db" is already in use

The CLI handles this automatically, but if it persists:

docker rm -f mycelium-db mycelium-backend
mycelium up

9. Migration Failures

Symptom: alembic.util.exc.CommandError or schema mismatch errors in logs

Migrations run automatically on container start. If they fail:

mycelium logs mycelium-backend --tail 100   # check startup errors
mycelium down && mycelium up                # restart often fixes it

If the schema is corrupted (destroys data):

mycelium down --volumes && mycelium up

10. No Active Room

Symptom: No active room. Use 'mycelium room use <name>'

mycelium room ls
mycelium room use <name>
# or pass room explicitly:
mycelium memory ls --room <name>

11. OpenClaw Agents Prompt for Approval on Mycelium Commands

Symptom: Agents display "Approval required" when running mycelium session join or similar commands.

Fix: Add mycelium to OpenClaw's exec approvals allowlist:

# For specific agents (recommended):
openclaw approvals allowlist add --agent "<agent-id>" "~/.local/bin/mycelium"

# Or for all agents (convenient but less restrictive):
openclaw approvals allowlist add --agent "*" "~/.local/bin/mycelium"

# Restart the gateway
openclaw gateway restart

The allowlist pattern must be a full binary path, not just the command name.

12. OpenClaw CLI Fails with "pairing required"

Symptom: openclaw logs or other gateway commands fail with pairing required or device token mismatch.

Fix: Approve the pending device pairing request:

openclaw devices list
openclaw devices approve <requestId>
# Or approve the most recent:
openclaw devices approve --latest

13. OpenClaw Adapter Fails on Containerized Gateway

Symptom: mycelium adapter add openclaw --openclaw-container <name> fails with No running container matched "<name>" under podman or docker, even though docker exec <name> openclaw status works fine.

Cause: Mycelium routes install commands through docker exec to avoid OpenClaw's --container flag, which uses docker inspect for container-name resolution. If you see this error, you may be running an older version of the CLI that still uses openclaw --container.

Fix: Upgrade to the latest Mycelium CLI:

curl -fsSL https://mycelium-io.github.io/mycelium/install.sh | bash

Verify the container is reachable:

# Get the exact container name
docker ps --format "{{.Names}}" | grep -i openclaw

# Verify connectivity
docker exec <container-name> openclaw status

# Install with container flag
mycelium adapter add openclaw --openclaw-container <container-name>

You can also set OPENCLAW_CONTAINER as an environment variable instead of passing --openclaw-container every time.

14. Agents Join Sessions but Never Respond (Expired Channel Tokens)

Symptom: An agent appears in mycelium room ls as a session participant, but never responds to coordination ticks. No error in mycelium logs.

Cause: The agent's channel access token has expired or been invalidated (e.g., after a server restart). The OpenClaw gateway silently drops the channel sync connection without surfacing an error to Mycelium.

Diagnosis:

# Check gateway logs for channel sync errors
journalctl --user -u openclaw-gateway --since "10 min ago" | grep -i "sync\|401\|unauthorized"

# Or on the hub
openclaw logs | grep -i "sync\|401\|unauthorized"

Fix: Re-authenticate the agent with the channel server and update the token in ~/.openclaw/openclaw.json under the corresponding channels.<channel>.accounts.<agent> section. Then restart the gateway:

openclaw gateway restart

In a hub-and-spoke setup, update tokens on every node that runs agents.

16. Spoke Cannot Reach Hub Backend

Symptom: mycelium status or mycelium room ls from a spoke returns a connection error pointing at the hub's URL.

Diagnosis:

# Test raw connectivity
curl http://<hub-ip>:8000/health

# Check what the spoke is configured to use
grep api_url ~/.mycelium/config.toml

Common causes:

• Firewall or security group blocks port 8000
• Hub backend isn't running (mycelium up on the hub)
• VPN/Tailscale not connected
• Wrong IP or port in config.toml

Fix: Ensure the hub is running and the spoke can reach it, then re-initialise if the URL is wrong:

mycelium init --api-url http://<correct-hub-ip>:8000

Configuration Reference

CLI settings — ~/.mycelium/config.toml

Backend settings — ~/.mycelium/.env

All of these are written by mycelium config apply from the matching runtime.* config keys — don't edit .env by hand.

Agent environment variables

Read by the CLI and adapters at runtime to identify the agent and locate the backend:

Knowledge-ingest cost controls

Overrides for [knowledge_ingest] in ~/.mycelium/config.toml. Every key below has a matching env var for ephemeral changes (no config edit needed). Forwarding to the CFN graph is off by default.

Log Locations

mycelium logs                       # all services
mycelium logs mycelium-backend      # backend only
mycelium logs mycelium-db           # database only
mycelium --verbose status           # CLI debug output

Reset Everything

mycelium down --volumes   # stop and delete all data
rm -rf ~/.mycelium        # remove all config
mycelium install          # fresh install

Getting Help

Report issues at https://github.com/mycelium-io/mycelium/issues