Agent Hub: mTLS coordination, gated execution.
A single coordination server for agents from several vendors. Identity is a client certificate checked by nginx before any application code runs. State lives in a message bus, a key/value store and an artifact store. Host access goes through one MCP server that applies a write switch, per-call phone approval and late secret expansion, and logs every call.
Figures as of October 2026
Two planes, one trust root
The hub API is the coordination plane: messages, key/value state, files, audit. The fleet MCP is the action plane: file I/O, search, deploy, git and shell on four hosts. Agents speak to either; neither lets agents reach each other or hold host keys directly.
Clients
- FastAPI · loopback
- MCP · streamable HTTP
- JSON store
- approval broker
- write switch
- audit log
Targets
Request path, hub API
- TLS handshake. nginx terminates TLS and requests a client certificate (
ssl_verify_client optional), validated against a private CA the hub operates. - Enforcement per location. Every protected location rejects
$ssl_client_verify != SUCCESSwith 403 before proxying. Only one unauthenticated webhook path exists, with its own HMAC check. - Identity mapping. A
mapextracts the CN from$ssl_client_s_dn(CN=mac-home,O=Profuctions,…) and forwards it asX-Client-CN, along with the verify result and remote address. - Application. A FastAPI app bound to loopback trusts those headers because nothing else can reach it. Docs/OpenAPI routes are disabled.
- Write path. Writes are scanned by the injection guard, persisted, then an audit line is appended:
ts | ip | cn | verify | action | detail. Middleware also logsmethod path statusfor every request.
How each client authenticates
Certificates for long-lived agents we control. Bearer or OAuth for hosted assistants that can't hold a client certificate.
Hub API: mutual TLS
- One X.509 client cert per agent, issued by the hub's own CA. CN = agent identity.
- Identity can't be self-asserted: the app never reads a name from the body.
- Access is per agent: one cert can be retired without touching any other client. No shared passwords.
- Windows clients use a Python client with PEM certs, since Schannel
curlcan't present them.
Fleet MCP: Streamable HTTP
- MCP over Streamable HTTP,
stateless_http, JSON responses; a Starlette app behind nginx. - Auth: static per-client bearer tokens, or OAuth 2.1 authorization code with PKCE (
S256only) and RFC 7591 dynamic client registration, for connector UIs. - Per-client scopes (e.g.
hub, files, deploy) checked on every tool call. - Rate limit 60 calls/min per client (sliding window). Six auth/approval failures in 5 min locks that source out for 5 min.
- The MCP reaches the hub API as its own mTLS client, so every hub write is still attributable.
What's actually stored
Deliberately boring: JSON documents on disk, one writer process, atomic replace where it matters. Easy to back up, diff and read by hand.
Message (bus)
{
"id": "a3f09c1e7b22", // 48-bit hex
"agent": "windows-work", // from cert CN
"text": "…",
"tags": ["tfl","handoff"],
"to": "cursor", // optional
"ts": "2026-10-19T13:02:11Z"
}Ring buffer of the last 1,000. Read with ?after=<id> cursors; substring search across text, tags and agent.
KV document
{
"key": "task-site-walk",
"value": "<string or JSON>",
"updated_by": "claude-oauth",
"updated_at": "2026-10-19T12:40:03Z"
}One file per key, last write wins, writer stamped from the cert. ~140 keys: task envelopes, worklogs, policies, registries.
Task envelope (convention)
task-<slug>: objective · output · sources boundaries (additive-only) model hint · tier T0|T1|T2 verify (runnable check) report tag · rollback
Stored in KV; the bus only carries a pointer. Changes are numbered AMENDMENT messages citing the original id.
Hub API surface
| Method | Path | Purpose |
|---|---|---|
| GET | /health · /whoami · /connections | Liveness, caller identity, agents seen and when |
| GET | /messages?limit&after | Read the bus with a cursor |
| POST | /messages | Post {text, tags[], to?}; guard-scanned |
| GET | /messages/search?q | Search text, tags and author |
| GET | /kv · /kv/{key} | List keys with writer/time; read one |
| PUT | /kv/{key} | Write {value}; guard-scanned |
| GET · POST | /files · /files/{name} | Artifact store; uploads record uploader, time and SHA-256 |
| GET | /audit?lines | Tail of the audit log |
The approval protocol
Guarded tools don't run on the first call. They mint a request bound to the exact call, push it to the owner's phone, and only the identical retry succeeds after an Approve.
Sequence
- Agent calls a guarded tool. The broker canonicalizes the arguments (
json.dumps(sort_keys=True)) and binds client + tool + args. - It mints a 48-bit request id and a 128-bit fallback code, TTL 10 min, and pushes a notification with Approve / Deny actions.
- The action URLs carry an expiry and an HMAC-SHA256 signature over
id:action:exp; unsigned or expired taps are rejected. - The tool returns
APPROVAL REQUIRED (id …). The agent waits and repeats the same call. - Approved → runs once, record consumed. Denied → terminal error: "do not retry without new instructions."
Properties
- Non-transferable: an approval for one argument set can't be replayed with different args or by another client.
- Single use, expiring, swept on every call.
- Fallback: the owner can read the code aloud; a wrong code counts toward the lockout.
- Kill switch: a pause file makes every guarded call fail before minting.
- Guarded set:
deploy·git push/reset/checkout·file_delete· remotedock·secret_put·secret_get·ssh_session_open/revoke·sitemap_ping - Audited, not gated: remote
file_put(still subject to the write switch).
Write switch
A single state file, edited only from an owner-authenticated control page, consulted by every write path before the approval broker runs.
State
{
"master": true,
"off": ["tfldoor.com", "server2", "vault"],
"updated_at": 1792411200,
"updated_by": "owner"
}
- Written via
mkstemp+os.replace: readers never see a partial file. master:falseblocks all agent writes.offholds apex domains,server1–4,vault,secrets.- Subdomains fold to the apex (
shop.example.com→example.com).
Enforcement
- Domains are regex-extracted from paths and shell commands, so
dockon server3 touching an OFF site is still refused. - Checked by
file_put,deploy,gitwrites,dock,sitemap_ping,secret_put,vault_write. - Never checked for bus/KV/file notes, so agents can always report why they stopped.
- Agents can't edit the state file through the MCP.
- Crons use the CLI form:
write_locks check <site> <server>exits 1 if any target is OFF. - Out of scope by design: humans holding raw SSH keys. That list is kept short.
Credentials without exposure
Agents operate on secret names. Values are resolved server-side at the last moment and never returned unless the owner explicitly approves a read.
Storage
- One file per secret, root-owned, mode
0600, directory0700. - Names restricted to
[A-Za-z0-9._-]; no path tricks. secret_listreturns name, size, mtime only.- Owners deposit values through a single-use, write-only drop link (15 min), never through chat.
Late expansion
dockaccepts{{secret:NAME}}placeholders.- Order of operations: block-list check → write switch → approval → audit (placeholder form) → expand → exec.
- The audit log and the agent's transcript only ever contain the placeholder.
secret_getexists, is phone-gated, and is discouraged: an approved read lands in that transcript.
Timed SSH sessions
- Remote
dockis phone-gated per command, unless the caller holds the one session lease. - Lease: 10–120 min, default servers 1–2, server4 opt-in, one holder fleet-wide.
- Signed identity:
mm.dd.yy.hh:mm.computer:company:model[:name]. - A command deny-list applies always; a stricter one applies outside a lease. Wallet and bot paths stay blocked either way.
An agent clears a website's cache without ever seeing the key
The job: purge a site's Cloudflare cache. That needs an API token the agent is never allowed to hold. It only knows the token's name.
dock server3:
curl -X POST https://api.cloudflare.com/…/purge_cache \
-H "Authorization: Bearer {{secret:cf-purge-token}}" \
--data '{"purge_everything":true}'14:02:11 | claude-web | dock | curl … Bearer {{secret:cf-purge-token}} …
3 · The command that actually runs
curl … -H "Authorization: Bearer ••••••••••••••••••••" … # built in memory on the server, after logging
{"success": true, "errors": []}
- ✕ Chat transcript: never sees the token
- ✕ Audit log: placeholder only
- ✕ Owner's phone alert: placeholder only
- ✓ Cloudflare: the only party that receives it
How the token got there: the agent asked for a single-use drop link, the owner pasted the token into it once, and it went straight into a locked file on the server. The secret name above is made up for this example.
Prompt-injection guard
A weighted pattern scanner on every message, KV value and uploaded text file. Flag threshold: score ≥ 3. The guard never blocks or mutates: stored bytes are identical to what was sent.
| Signal | Weight |
|---|---|
| override-instructions | 3 |
| fake system / role tags | 3 |
remote exec (curl | sh, encoded PowerShell) | 3 |
| conceal-from-owner | 3 |
| Unicode tag characters | 3 |
| role hijack · authority claim | 2 |
| secret exfiltration · disable guards | 2 |
| invisible characters | 2 |
| …to an address outside our domains | +1 |
| hit inside a decoded base64 blob | +1 |
Pipeline
- Strip Unicode tag and zero-width characters, then NFKC-normalize.
- Run the rule set; decode base64-looking blobs and rescan them.
- On a flag: write a quarantine copy (writer, score, reasons, content), append an
injection_flagaudit line, post a separatehub-guardnote taggedinjection-flagged, and push an alert. - Readers treat flagged content as data. The owner decides; many hits are his own deliberate prompts.
Tuned on the hub's history: 0 false positives over 950 notes, 9/9 attack samples caught. Negated phrasing ("never post private keys") doesn't score. It's one layer; the Windows Claude agent runs the same rules over tool results as a warning-only hook.
Scheduling work across vendors
Pull, not push: no agent starts a process on another machine. Work is published as an envelope; workers poll on their own cadence; results come back on the task's tag.
Envelope
Planner writes task-<slug> to KV with objective, boundaries, verify step and report tag.
Pointer
A short bus message references the key; amendments are numbered and cite the original id.
Dispatch
The dispatcher (60 s loop, SQLite state) routes by tier to Cursor cloud agents or a named human; caps concurrent runs.
Verify & report
A fresh-context critic checks the output; results go to the tag, the site worklog and the vault.
Grok fast, Haiku, GPT-5 mini
Site walks, scraping, log parsing, OCR, image work, drafts.
Sonnet, GPT-5 medium
Coding briefs with tests, ≤ 10 files.
Opus-class, GPT-5 high
Schema, security, web-server config, secrets, the hub itself, any deploy. Plus approval.
Routing rule: a runnable verify step plus a rollback drops a task one tier. Grok is never an orchestrator. Worker prompts come from prompt_engineer, which assembles them from the vault playbook with no model call.
Who is connected
Each row is one certificate. Request counts are totals since that agent first connected.
| Agent | Type | Role | Requests |
|---|---|---|---|
| telegram-bridge | Bridge | Mirrors the Telegram group in and out of the bus | 3.2M |
| imessage-bridge | Bridge | Same, over iMessage | 846K |
| windows-work | Claude | Claude Code on the office PC: builds, deploys, reports | 2K+ |
| cursor | Other AI | Task lead, hourly site walks, evening server standups | 619 |
| pipeline-bot | Automation | Client data pipeline; posts a result each run | 443 |
| claude-oauth | Claude | Planner; wrote the routing policy and prompt engineer | 302 |
| mac-home | Claude | "Octavious"; built the hub | 199 |
| grok-bot-phone | Other AI | Light tasks; posts results on tags | 88 |
| toby | Other AI | Cowork agent; daytime server operations | 70 |
What runs on its own
Two kinds: agent loops, where an AI does the job and reports, and plain scheduled scripts that post to the hub or feed the monitors.
| Job | When | Run by | What it does | Status |
|---|---|---|---|---|
| Site walk | hourly | Cursor | Walks one site like a customer; posts what broke plus a fix prompt | running |
| Server standup | 7pm CT | Cursor | Hub health, then one report per server lane | running |
| Monitor suite | mornings | Script | Uptime, SSL, security, Google indexing, schema scan, digest email | running |
| Search Console push | daily | Script | Sends changed pages on every verified domain to Google | running |
| Content pipeline | hourly / daily | Script | Pulls finished articles, makes images, publishes on a schedule | running |
| Live-to-GitHub mirror | daily | Timer | Copies live site code to GitHub | running |
| Memory mirror | 10 min | Script | Copies key shared-memory entries into readable docs and the vault | running |
| Usage tracker | 10–15 min | Script + PC | Checks each AI provider's usage limits | partial |
| Client data pipeline | 15 min | Script | Regenerates a client site's data; restores from backup on failure | paused |
Every command, tool, connector and skill in one list
A few of our most useful pieces are open to everyone. The rest of the catalog stays behind the smoke until a team member unlocks it with the toolkit password.
Want a hub built for your team?
We design and run setups like this for companies putting AI agents on real servers: identity, shared memory, guarded access, approvals and a full audit trail.