← Back to dashboard

Remote Agents

Every other execution path in this dashboard dials out to its target. That works for a cloud API, and it works for a LAN the dashboard container happens to sit on. It cannot work for a private network the dashboard is not in — which is why the local Ansible runner says so outright when its Docker socket is missing:

On-prem Kubernetes targets run in a sibling container because only this host has a route to the cluster — a dashboard deployed in a cloud (ECS / ACI / Cloud Run) cannot run them.

A remote agent inverts the direction. It is a container you run inside the private network; it dials out to the dashboard, asks for work, does it, and reports back. No inbound firewall rule, no VPN, no port forward, no exposed management interface.

Philosophy

1. The dashboard is not trusted by the agent. This is the unusual part and everything else follows from it. Elsewhere in this codebase there is one trust domain — you run the dashboard, it holds your credentials, and SECURITY.md puts a malicious administrator out of scope. An agent breaks that, because it puts a dashboard-controlled process inside a network whose owner may not be the dashboard's operator. So the design assumes the dashboard may be compromised and makes that insufficient to execute anything meaningful on the LAN.

2. The security boundary is a file you own. policy.yaml is mounted read-only into the agent and no dashboard API can read, write or override it. It names what may be touched. It fails closed.

3. Nothing reusable crosses the wire. The agent authenticates with an Ed25519 signature over one method, one path, one body, valid for sixty seconds — not a bearer token. This is what makes it safe through a TLS-inspecting corporate proxy, where an Authorization header would be captured in the clear on every poll.

How the dashboard implements these

The agent's private key

The agent generates an Ed25519 keypair on first start and keeps the private half. Only the public half is ever sent, so a dump of the dashboard's database cannot impersonate an agent — there is nothing in it to replay.

Where it lives, exactly: $AGENT_STATE_DIR/identity.json, created mode 0600 inside a 0700 directory owned by uid 10001, written to a temporary file and renamed so it never exists world-readable even for an instant. It is not encrypted at rest: no passphrase, no TPM, no keyring. That is a deliberate limit rather than an oversight — a passphrase an unattended container must read at boot has to be stored next to the key it protects.

That reasoning still holds, and it is worth reading alongside Sealing a credential this host keeps, which does encrypt a hypervisor credential on this host. The two are not in tension: sealing is not offered as protection from root here either. It exists because connections.yaml is a file you copy into repos, tickets and backups, and this one never leaves its volume.

Who can read it:

What a stolen key actually grants, which is the part worth being precise about: it authenticates five POST /api/agent/* routes and nothing else. The holder can lease jobs an operator already queued for that agent, heartbeat them, push log lines, and complete them. They cannot create a job (that needs an admin session), learn anything about what else the dashboard manages, reach any user-facing route, or touch another agent's jobs — owned_job filters on agent_id and answers "not found" identically for "not yours", so the API is not even an oracle for their existence.

Once the agent brokers hypervisor connections it does hold target credentials, so be precise about what the holder inherits. Where the credential sits in connections.yaml, they have it outright — but then so does anyone who can read that file, and the stolen key is beside the point. Where the connection uses dashboard_secret, the key lets them request a credential, and that is a narrower and much noisier thing: only for a connection bound to that agent, only while a job it can lease is actually running, one audited event per release, and revocable by clearing the agent's public key. A leaked file is none of those.

That leaves three real consequences: denial of service against that one agent's queue; the ability to report false findings — data a human reads and acts on, so treat it as untrusted input, which is the other reason nothing is auto-registered; and, for dashboard_secret connections, credential requests that show up in the audit log and stop the moment the key is revoked.

Revocation does not depend on reaching the container. Revoke clears the stored public key, so the next poll fails verification whatever that container is still doing, and any job it held is failed immediately rather than at the next reconcile.

Running under Podman

Rootless Podman is a real improvement here and worth preferring — as long as it is recommended for the right reason. It changes who can steal the key. It does not change whether host root can.

What it actually buys:

• No daemon socket, so the docker group row above disappears entirely. In most shops that group is the larger of the two exposures.
• The state volume lives under ~/.local/share/containers/storage/volumes/…, owned by one unprivileged user rather than by root, with uid 10001 mapped through that user's subuid range.
• Every hardening flag in the emitted command works unchanged — --read-only, --cap-drop ALL, --security-opt no-new-privileges, --user, --tmpfs. Substitute podman run for docker run and paste it as-is.

Two things to know before you do:

• On SELinux hosts the bind mounts need a label suffix — :ro,Z — or the container cannot read policy.yaml. Podman's natural home is RHEL-family hosts, where SELinux is enforcing by default, so this is the common case rather than the exotic one. The emitted command already carries it, and Podman ignores Z where SELinux is absent; if you are pasting an older command, add it. That fails closed (Cannot read the policy file, exit 2), so it is a confusing five minutes rather than a hole — see Troubleshooting for how to recognise it, because the dashboard-side symptoms all look like networking.
• --restart unless-stopped does not survive a reboot on its own; rootless Podman needs systemctl --user enable podman-restart (and loginctl enable-linger) because there is no daemon to do it for you.

What neither runtime fixes is host root. If that is the threat you are defending against, the answer is not a different container runtime — it is a credential that cannot be copied, sealed to hardware, which is a future feature rather than a configuration. Until then the honest position is the blast-radius table above plus fast revocation.

Architecture

   private network                        │            dashboard
                                          │
  ┌───────────────┐   outbound HTTPS 443  │   ┌────────────┐   ┌──────────┐
  │ dashboard-    │ ────────────────────► │   │   Caddy    │──►│   app    │
  │ agent         │ ◄──────────────────── │   │ agents.… │   │          │
  └───────┬───────┘   signed job envelope │   └────────────┘   └────┬─────┘
          │                               │                         │
          │ unauthenticated probes        │                    ┌────▼─────┐
          ▼                               │                    │  jobs    │
  vCenter :443  Proxmox :8006            │                    │  table   │

The jobs table is the queue in both directions. An agent job is created queued with an agent_id; the agent leases it with the same atomic rowcount claim the local job runner uses, streams Live Output into job_logs, and completes it. Because it writes to the same tables, an agent's output appears in the existing Live Output pane and the existing Cancel button works — there is no separate UI for watching one.

Why the local runner and an agent never collide

Three independent guards, all pinned by tests/test_agent_lease_invariants.py:

• agent_service.AGENT_JOB_TYPES is disjoint from jobs_worker.HANDLED_TYPES, so the local runner's claim query never even sees an agent job;
• job_service.create_job(agent_id=…) forces status="queued", in the one funnel every job row passes through, so no call site can forget;
• the lease filters on agent_id in both the select and the claiming UPDATE.

Reconciling with "why one-shot runners"

config-management.md argues against long-lived runners. Read its wording precisely: no escape hatch for "give me a long-lived worker for performance reasons". An agent is not asking for persistence for performance. It asks for it for reachability — you cannot launch a one-shot container inside a network you cannot reach, so something must already be there.

The doctrine's actual invariants are preserved. Today the agent holds no target credentials at all (discovery does not authenticate). When agent-executed Ansible lands, the agent will spawn a one-shot sibling container per job, so the persistent process still never holds a credential and never executes operator-supplied code. The runner stays one-shot; only the thing that launches it moved.

Prerequisites

Running on Windows

A Windows desktop or server is a perfectly good agent host, and for some hypervisors it is the only sensible one — Hyper-V and VMware Workstation both live on Windows. There is no Windows-container build of the agent and you do not need one: Docker Desktop runs the Linux image in its Linux VM, the same way it runs everything else.

Almost nothing about the container changes. Every hardening flag is identical — --read-only, --cap-drop ALL, --security-opt no-new-privileges, --user 10001:10001, --tmpfs. What changes is the shell you paste into. Set the toggle on the Agents page to Windows (PowerShell) and the emitted command is already adapted:

• backtick line continuations instead of \. This is the one that bites: a trailing \ is not a syntax error in PowerShell, it is a literal argument, so docker run takes it as the image name and every following line executes as its own command. You get invalid reference format, then The term '-e' is not recognized, and nothing at all pointing at the paste having come apart.
• ${PWD} instead of $PWD, and Set-Content / Remove-Item instead of printf / rm.

Six things to know before you start:

• Docker Desktop must be in Linux-containers mode. In Windows-containers mode the pull fails with no matching manifest for windows/amd64, or — if the image is already local — image operating system "linux" cannot be used on this platform. Neither says "you are in the wrong mode". Right-click the tray whale → Switch to Linux containers.
• :ro,Z is inert here, and kept on purpose. It is the SELinux relabel, and Docker ignores z/Z on any host without SELinux — which includes Docker Desktop's VM, exactly as it does on Ubuntu or Debian. Keeping it means there is one mount string rather than two that can drift. It costs nothing; ignore it. The corollary matters more: every chcon and ls -lZ instruction in this document is Linux-only. If a mount is unreadable on Windows it is file sharing, not labelling — check Settings → Resources → File sharing. The agent applies that same qualification to its own error text: it recognises Docker Desktop's kernel and names file sharing instead of the SELinux label, so you should not see chcon advice on this host at all.
• File permissions work out in your favour. Docker Desktop presents bind mounts with permissive ownership because NTFS ACLs do not map onto POSIX mode bits, so uid 10001 can read policy.yaml with no umask and no chmod. The emitted PowerShell command has no umask equivalent for that reason, not by omission.
• Keep policy.yaml under your user profile. A UNC path (\\server\share) or a mapped network drive cannot be bind-mounted, and if Docker Desktop cannot reach a bind source on the Hyper-V backend it materialises an empty directory in the container instead of failing — so the agent reports it cannot read its policy file and the advice you get points at mount flags rather than at file sharing. Notepad is a fine editor for the file; CRLF parses.
• --restart unless-stopped only holds while the engine is running. Docker Desktop lives inside a logged-in user session, so enable Start Docker Desktop when you log in and know that logging out stops the agent. This is the Windows counterpart of the rootless-Podman caveat above. For an agent that must stay up unattended, use Windows Server with the WSL2 engine, or run it in a Linux VM on the same host.
• The source IP the dashboard records is Docker's NAT gateway, not the Windows host's LAN address. The Source IP column and the audit rows will show that. It is not a fault, and it does mean the column cannot identify which Windows host an agent is on.

What discovery can reach from a NAT'd container. Probes are ordinary unprivileged unicast TCP connects — no ICMP, no ARP, no broadcast or multicast — so NAT costs you nothing: anything routable from the Windows host is routable from the container, and being off the LAN's layer-2 segment does not matter. Every target is an address the agent computed arithmetically from a CIDR you wrote down.

The exception is the host the agent is running on. Inside the container 127.0.0.1 is the container, so allow_loopback cannot reach a service listening on the Windows machine's own loopback — see VMware Workstation Pro, where that is the whole question. Docker Desktop's host.docker.internal resolves to the host and is a legitimate fqdn: target, but it only helps if the service is listening on an address other than 127.0.0.1.

Workflow

1. Publish the agent endpoint

# Bash (WSL / Linux / macOS)
AGENT_HOSTNAME=agents.example.com docker compose -f docker-compose.hub.yml -f docker-compose.agent.yml up -d

# PowerShell (Windows) — the inline VAR=value prefix is a parse error here
$env:AGENT_HOSTNAME = 'agents.example.com'
docker compose -f docker-compose.hub.yml -f docker-compose.agent.yml up -d

This adds Caddy on a separate vhost that proxies only /api/agent/*. The UI, the login form, the OAuth callbacks and /setup stay on the internal address. The whole internet-facing surface becomes one machine-only, HTML-free, session-free prefix.

The base stack still publishes 8001; bind it to loopback or firewall it, or the plain-HTTP dashboard sits beside the TLS vhost and the split buys you nothing.

The overlay also sets two things you would otherwise have to reason about yourself, and both matter here:

• TRUSTED_PROXY_HOSTS — the literal IP of the Caddy container, which is why the overlay pins a subnet and gives the gateway a static address. Without it the app ignores the proxy's headers and every agent appears to come from the gateway, which also flattens the login throttle's per-address cap. It must be a literal: uvicorn 0.27 understands neither hostnames nor CIDR. Get it wrong and the app logs a warning naming the peer to add.
• PUBLIC_BASE_URL — the origin agents reach you on. The signing audience is derived from it, and an audience derived instead from an untrusted request would pin http://… into the config permanently, making every agent signature fail to verify with a 401 that looks exactly like a revoked agent.

Only two callers are allowed to pin that value: an admin minting an enrolment code, and a successful enrolment. An unauthenticated poll — including one arriving at the plain-HTTP 8001 with a forged Host — resolves an audience but never writes it down, so a stranger cannot pin one of their choosing and 401 your whole fleet. Set PUBLIC_BASE_URL regardless: it is what makes the pinned value correct rather than merely trusted, and it is the only way the value is right when the operator's browser and the agents reach the dashboard on different names.

If the hostname is internal

Caddy's automatic HTTPS means ACME, and ACME needs the name to resolve publicly with port 80 reachable. A lab name — agents.lab.internal, a split-horizon domain — can never satisfy that, so Caddy retries forever and serves nothing, and the agent (which refuses plain HTTP) cannot connect at all. This is the most likely reason a first run never gets started.

Set AGENT_TLS_INTERNAL=1 and Caddy issues from its own CA instead, immediately and offline:

# Bash (WSL / Linux / macOS)
AGENT_HOSTNAME=agents.lab.internal AGENT_TLS_INTERNAL=1 \
  docker compose -f docker-compose.hub.yml -f docker-compose.agent.yml up -d

# PowerShell (Windows)
$env:AGENT_HOSTNAME = 'agents.lab.internal'
$env:AGENT_TLS_INTERNAL = '1'
docker compose -f docker-compose.hub.yml -f docker-compose.agent.yml up -d

Then give the agents that CA — it is in no system trust store:

# Bash (WSL / Linux / macOS)
docker compose -f docker-compose.hub.yml -f docker-compose.agent.yml \
  cp agent-gateway:/data/caddy/pki/authorities/local/root.crt ./caddy-root.crt

# PowerShell (Windows)
docker compose -f docker-compose.hub.yml -f docker-compose.agent.yml `
  cp agent-gateway:/data/caddy/pki/authorities/local/root.crt ./caddy-root.crt

Copy it to the agent host, mount it, and set AGENT_CA_BUNDLE to the mounted path.

Prefer this over AGENT_INSECURE_TLS=1. The insecure switch also disables verification, which is the part most worth exercising, and it masks a wrong PUBLIC_BASE_URL or TRUSTED_PROXY_HOSTS until an agent starts 401ing for reasons that look like revocation. The root is generated on first start and lives in the caddy_data volume — it survives restarts but not docker compose down -v, and wiping it means re-issuing the root to every agent.

2. Write the policy

Copy examples/remote-agent/policy.example.yaml and edit the ranges:

targets:
  - cidr: 10.20.0.0/24
    ports: [443, 8006, 9440, 5985, 5986]
deny:
  - 169.254.0.0/16
job_types:
  - agent_discover
limits:
  max_hosts: 1024

Omitting ports allows any port in the range. Naming them is the difference between "may look for databases here" and "may reach anything here".

Mind the space after the dash. -cidr: 10.20.0.0/24 is valid YAML — for a key named -cidr, not for a list entry — so the whole list parses as a mapping and grants nothing. The agent refuses to start on that, and on any entry it cannot read as a target, and on a key written twice (YAML keeps only the last, which in this file silently drops a grant). Being loud is the only safe direction here: a target that is skipped rather than rejected leaves an agent running with an allow-list you believe you wrote.

Mount it :ro,Z, as the emitted command does. The Z is the SELinux relabel and it is not cosmetic: on Fedora, RHEL, CentOS, Rocky or Alma a bind mount keeps its host label, and the container may not read a file in your home directory however permissive its mode is — the agent then exits 2 on Permission denied against a mode 644 file. Docker and Podman ignore z/Z where SELinux is absent, so it is safe on every host.

3. Register and enrol

Agents → Register Agent. You get a single-use code, valid 15 minutes, and a copy-paste docker run built from the dashboard's own URL. On first start the agent generates an Ed25519 keypair, redeems the code, pins the dashboard's public key, and writes its identity to the state volume. The code is never needed again.

The command comes in two shell flavours, picked with the Linux / macOS and Windows (PowerShell) toggle above it; the container is identical either way, and on a Windows browser the PowerShell form is pre-selected. See Running on Windows for what differs on that host.

The private key never leaves the agent host. The dashboard only ever stores the public half, so a dashboard database dump does not let anyone impersonate an agent.

The default command passes the code as -e AGENT_ENROLLMENT_CODE, which is convenient and has one drawback worth knowing: an environment variable is baked into the container's config, so docker inspect still shows the code long after it has been spent. The modal offers a second form that mounts it as a file and sets AGENT_ENROLLMENT_CODE_FILE instead, leaving nothing durable in Docker's metadata; delete the file once the agent shows Online. To keep the code out of your shell history as well, create that file with an editor and paste only the docker run that follows it — on Windows that is the only way, because PowerShell writes its history to disk as you type.

If you do write it by hand, save it as plain ASCII or UTF-8. The emitted PowerShell command uses Set-Content -Encoding ascii -NoNewline for that reason, and the agent is belt-and-braces about it from its side: a UTF-8 BOM is stripped rather than sent as part of the code, and a UTF-16 file — what PowerShell 5.1's > and Out-File produce, and what Notepad calls "Unicode" — is refused with a message naming the encoding and the Set-Content flags that fix it, rather than the raw decode traceback it used to be. Either way the code is not spent, so the same one still works once the file is rewritten.

Because the container runs as uid 10001, that file has to be readable by it — a mode 0600 file owned by your host user is not, and the agent says so and exits rather than failing mysteriously. A single-use secret with a fifteen-minute lifetime in a directory you chose is an acceptable trade for that; a long-lived one would not be. (On Windows this cannot arise: Docker Desktop presents bind mounts with permissive ownership.)

The signing audience is pinned by that first code

The URL in the command you just copied is the signing audience: the value every agent signature is checked against from then on. The first enrolment code minted on an install pins it, permanently, and nothing later overwrites it — see §1 for which two callers may pin and why an unauthenticated poll must not.

Which means the first code decides it, and on a split-vhost deployment the default is wrong. Settings → Integrations → Remote Agents pre-fills Public base URL from the origin your own browser is on, and on the layout §1 recommends — UI on the internal address, /api/agent/* on its own vhost — that is the UI hostname. Pin that and the agent dials a vhost which does not serve /api/agent, gets a 404, and the console shows it enrolling with Last seen: never and no policy hash. Nothing distinguishes it from a revoked agent, and there is no /api/agent/enroll line in the dashboard log to contradict you, because no request ever arrived. Correcting Public base URL afterwards appears to work and changes nothing, because the pin wins.

So the panel shows the pinned audience read-only, beside — and clearly distinct from — the Public base URL field:

• Pinned to — the audience itself, and the number of enrolled agents signing against it.
• A warning when it is the same origin you are browsing (right for a single-hostname install, the failure above for a split vhost), when nothing is pinned yet and the next code is about to decide it, or when it is http://.
• Reset signing audience — clears the pin so the next enrolment code pins it again from Public base URL. This is the only mutation offered: there is no way to set the audience from an API, because a settable audience is the write-once rule with extra steps. Admin only, the same gate as minting a code.

It invalidates every enrolled agent. Each signs against the audience it was handed at enrolment, so all of them start getting 401s — which an agent reads as "revoked" and stops on. Every one needs a fresh enrolment code afterwards. The confirm dialog names the count.

Registering an agent is refused with a 409 while the pin contradicts Public base URL, rather than handing you a command built from the stale value. Nothing is created by the refusal — the check runs before the agent row exists, and before a re-issue clears the existing key — so a blocked re-issue leaves the running container working. Resolve it by setting Public base URL back to the pinned value, or by resetting the audience.

A divergence is not always a mistake, though, which is why the refusal is not a dead end. PUBLIC_BASE_URL is one value doing two jobs — the OAuth callback origin as well as the agent audience — and a split vhost is exactly the case where those want different hostnames. Confirming the prompt re-issues with ?acknowledge_audience=true, which mints against the pinned audience, labels the command with which URL it used, and logs and audits the exception. It does not move the pin: the override is permission to use the pinned audience, not to re-pin, because re-pinning would invalidate every enrolled agent as a side effect of registering one new one.

The same information is available to a script at GET /api/agent/audience, and the reset at DELETE /api/agent/audience; both are admin-only.

The BeyondTrust Gateway

agent_gateway lets an agent run a Gateway node on its own host — long-lived, and --privileged. Read that second word before enabling it: a Gateway carries protocol tunnels, which needs NET_ADMIN, NET_RAW, IPC_LOCK and /dev/net/tun, and Docker has no granular way to grant that set. The agent refuses to start one without the flag rather than start one that cannot work — without those capabilities a Gateway registers online and every tunnel silently times out, which reads as a firewall problem for days.

Four things must agree, the same shape Config Management uses:

1. the dashboard grants this agent the agent_gateway job type;
2. agent_gateway is in job_types: in your policy.yaml;
3. a gateway: block naming the image and setting privileged: true;
4. the deploy key, which the dashboard holds and the agent fetches per job, sealed.

The image comes from your file and never from a job — a job says only install or remove. Pull it yourself; the agent will not. Pin a tag rather than tracking latest under a registered Gateway.

It needs the Docker socket, which is root on the host — the same requirement and the same overlay as the other runners.

The dashboard-side driver for this today is the POV feature, which generates the whole policy for a broker VM it created; see pov-instance.md. The same feature uses agent_ansible to install a Password Safe Resource Broker on a Windows VM beside it — a .exe asset, which the Config-Management runner now installs with win_package.

4. Keep an eye on who is enrolled

The agents table answers "is this one mine?" without leaving the page:

Expand a row for the agent id, when it was registered and enrolled, and its policy hash — the sha256 of the policy.yaml on the agent host, self-reported on every poll. The dashboard cannot change it, so a value that moves means the file was edited on the host. That is the only signal you get that a compromised agent rewrote its own allow-list.

Revoke stops an agent immediately: its key is cleared, its running job is failed and its queued ones are cancelled, and no revoked agent can ever come back. The row stays, so the history of what it did stays with it.

Once revoked, Remove deletes the row. Its only purpose is to free the name — registration enforces uniqueness across every row, so a revoked lab-dc1 would otherwise squat that name forever. Job history survives the deletion; those jobs simply stop naming an agent, and the audit log records agents by name so it is untouched. Revoked rows are hidden by default; untick Hide revoked to see them.

5. Discover

Discover on an online agent, pick what to look for, optionally narrow the networks. The job page streams findings as they arrive.

Discovery looks for hypervisor management endpoints, and it is unauthenticated probing, always:

443 is shared by vSphere and XCP-ng, so the agent probes each host:port once and classifies whichever answered.

WinRM is the honest limit of the set. It identifies WinRM on Windows — and nearly every domain-joined Windows Server has WinRM enabled, the overwhelming majority of them not hypervisors. Those findings are marked possible only and rendered differently. There is a way to read the OS build, NetBIOS name and DNS domain anonymously (an NTLM type-1 negotiate token carries no credential, and the type-2 challenge answers with AV pairs) and the agent deliberately does not: it initiates an authentication exchange and lands in the Windows Security log as a logon event. A test bans the token by name.

No probe ever attempts a login. Authenticated probing of unknown hosts locks out service accounts and reads like credential spraying in a customer's SIEM.

Not probed, and not by omission: Proxmox Backup Server (8007) and oVirt/RHV answer on ports in this list and are returned as unidentified rather than mislabelled. VMware Workstation is absent because a desktop hypervisor exposes nothing on the network — a probe for it would be code that never returns anything.

Databases and Kubernetes clusters are no longer discovered by an agent. Password Safe already finds databases with managed credentials — it knows the platform, port and accounts, which a socket probe never could — so Import from Password Safe replaced that path. Kubernetes clusters are registered from a kubeconfig, which no credential-less probe could ever supply.

6. Add the connections

Findings are never auto-registered, and that is not a limitation — it is infeasible by construction. A connection needs a credential, and a probe by definition has none.

So you review the findings and click Add connection…, which prefills the Connections form with the host, port and a suggested name and runs under your own credentials and permission checks. Findings the dashboard already has a connection for are marked already_registered — computed server-side, because an agent should never be handed the inventory of everything else the dashboard manages.

One limitation worth knowing: a connection configured with an FQDN but discovered by IP reads as new, because the dashboard cannot resolve your private DNS — it is not on that network, which is the whole reason the agent exists. The prefill uses the IP the probe connected to, so a second scan matches.

Hypervisor connections

Once an agent is enrolled it can also broker hypervisor operations — inventory sync on a schedule, and power verbs on demand — for vCenter, Proxmox, Nutanix, XCP-ng, Hyper-V and bare ESXi endpoints the dashboard has no route to. Which of the two each product gets is its own table; Nutanix syncs but cannot be powered, and a page whose button has no agent verb says so rather than sending the nearest one.

The dashboard may hold the secret, never the target

A dashboard connection row bound to an agent stores the agent's name for it and never a host or a username. The job says "run inventory_sync on dc1-vcenter" and the agent resolves the endpoint from its own connections.yaml.

That asymmetry is the difference between this and a proxy, and it is not about secrecy — it is about aiming. A dashboard that could set host could redirect the agent's authenticated session at an endpoint of its choosing and harvest the credential on first use; one that could set username could spray a known password across accounts. So those stay yours, in your file, gated by your policy.yaml, which no dashboard API can reach.

The credential is the part that can move, and there is no single right answer:

The default is unchanged and stays available. The last row is the only configuration in which neither side holds a standing hypervisor credential.

The cost is two files joined by a string, and a typo in either yields unknown connection 'dc1-vcenter'. The connection form mitigates that by offering an agent picker rather than a free-text uuid.

The credential the dashboard holds

Set dashboard_secret: true on a connection in your connections.yaml and the agent stops reading a credential locally. Instead, for each job, it asks the dashboard.

Two things make that safe to do over the same channel the agent polls on:

• It is scoped to the job. The route is POST /api/agent/jobs/{job_id}/secret, and the connection it answers for is derived from the job row, not from anything the request says. The agent cannot name a different connection, so a stolen identity cannot enumerate what else the dashboard holds. The job must also be running — a cancelled job may still log and complete, but it gets no fresh credential.
• It is encrypted, not merely transported. The agent generates an X25519 keypair per fetch and sends the public half in the request body, which its Ed25519 signature already covers. The dashboard seals the credential to it (X25519 → HKDF-SHA256 → AES-256-GCM). So the guarantee in Behind a TLS-inspecting proxy still holds for a response body, not just a request: the inspecting proxy sees a ciphertext. The private half never touches disk and dies with the fetch, so unlike an enrolment-bound key there is nothing on the host to steal and use on traffic captured earlier.

The seal is bound to the agent, the job and the connection ref. That last one is the least obvious and the most important: without it a credential released for dc1-vcenter could be relabelled as the credential for a connection pointing somewhere else, which turns credential confusion into credential exfiltration.

The credential is held in memory for the job and scrubbed out of anything the agent sends back — Live Output and the job's error string both — because str(exc) from an arbitrary library can carry it, and that string is the only text a failed job renders. Nothing here claims to wipe it from memory: a Python string cannot be zeroed, and the bound is scope.

Requires an agent image of 2.1.0 or newer. The dashboard refuses to queue work for an older one rather than let it fail: a 2.0 agent does not know the key, so it would fall through to a password you had just deleted, send an empty one, and get back the hypervisor's own "wrong username or password" — the wrong diagnosis, and on the 30-minute sync schedule it retries until the service account locks out. Pull the image and restart; re-enrolment is not needed.

Sealing a credential this host keeps

Some sites will not move a credential to the dashboard — that being the point of an on-prem agent for them — but do not want it sitting in a YAML file as text. password_sealed: in connections.yaml, and client_secret_sealed: in passwordsafe.yaml, hold a value encrypted against a key in the agent's state volume.

Be precise about what this defends against, because The agent's private key argues the opposite for identity.json and both statements are true. Sealing is not protection from root on the agent host. The key is on the same machine, which is unavoidable for a process that has to restart unattended — a passphrase an unattended container must read at boot has to be stored next to the key it protects, and that has not changed.

What it protects is the file travelling. connections.yaml is a file you author, edit, version and copy: into a git repo, an Ansible role, a runbook, a support ticket, a screenshot, a config backup. identity.json does none of that — it is written by the agent into a 0700 volume and never leaves it. The key lives in that volume, so it goes into none of those copies, and a copy of the config is worthless anywhere else. That is the whole claim. If your threat model is a compromised agent host, use dashboard_secret or ps_managed_account instead; those are the rows in the table above that change what host compromise yields.

Seal a value with the agent image itself. Mount the same state volume the agent runs with:

docker run --rm -it -v dashboard_agent_state:/var/lib/dashboard-agent \
    chrweav/dashboard-agent:latest seal --host vcenter.lab.internal

docker run --rm -it -v dashboard_agent_state:/var/lib/dashboard-agent `
    chrweav/dashboard-agent:latest seal --api-url https://passwordsafe.corp.internal

It prompts for the value without echoing it, prints one line on stdout, and that line is what you paste in. Everything else it says goes to stderr, so seal … > value.txt gives you the token alone. A prompt rather than an argument is deliberate: on Windows it is the only way to keep the value out of the shell's on-disk history, the same reason the enrolment code has a file form.

Three things about it that are load-bearing rather than incidental:

• The address is part of the seal. --host must match the entry's host:, and changing the address means sealing again. connections.yaml is re-read per job, so an edit takes effect with no restart and without Docker access — and without the binding, somebody who can edit that file but cannot read the state volume could move a sealed vCenter password onto an entry pointing at a host of their own, with verify_ssl: false, and read the plaintext off the next sync. This is the same reason the dashboard is never allowed to set host. For --api-url only the host part is bound, so either accepted form of api_url works.
• Mount the volume, or the key is thrown away. Run seal without -v and it refuses outright rather than sealing against a key that dies with the container. If a value somehow was sealed elsewhere, the agent says which key sealed it and which key is present instead of "decryption failed".
• Delete the plaintext. A leftover password or password_file under a password_sealed is ignored and warned about on every job, because it means a credential is still here in the clear. Same rule as a leftover under dashboard_secret.

Requires an agent image of 2.2.0 or newer, and the dashboard cannot warn you — unlike dashboard_secret it never sees this file, so it has nothing to gate on. You cannot reach that state by accident either, because seal ships in the same image that reads the key. The backstop is that the agent now refuses a connection declaring no credential at all rather than sending an empty password, which is the shape an older image on a sealed-only entry would otherwise produce.

Four grants, all required

Nothing runs unless all four agree, and they belong to different people:

Withhold any one and nothing happens. The fourth is the only one the dashboard owns, and it owns it only because you said so in the third — a credential set on the Connections page for an agent-bound connection does nothing at all until your file opts that entry in. A refusal from the second or third arrives in Live Output naming the file and the line to add — the dashboard cannot fix it and does not pretend to.

The verbs

inventory_sync is read-only and runs on a schedule (default every 30 minutes; override per connection with options.sync_interval_minutes). power_on, power_off, power_reset, shutdown, reboot and restart are on-demand, issued by the existing power buttons on the hypervisor pages when the resolved connection is agent-bound.

A sync that comes back with no VMs at all is not applied to a connection that already has some. A zero-VM pass is how a deleted VM stops being listed — the sync removes every row it did not just write — and it is also exactly what an agent hands back when it could not read the host: on Hyper-V, an unloaded PowerShell module and a service account that cannot enumerate VMs both produce the same empty list. The agent therefore states which one it is, and an agent too old to say so never empties a cache. The rows stay, and the connection carries the reason on the Connections page (hover the red Error) instead of the page going quietly blank. The cost is the mirror image: a host you have genuinely emptied keeps showing its last VMs until you re-pull the agent — and, for Hyper-V or ESXi, the chrweav/hypervisor-runner image with it.

One inventory_sync is also queued automatically whenever a power job finishes, against the connection it acted on, so the page reflects a Start or a Stop without anyone pressing Sync Now. It is attributed to whoever pressed the button and appears on the Jobs page as Inventory sync: <connection> (page 1) — after power_off. Two things follow from where it sits: a power op that failed queues one too (an agent that lost the response to a call it did make is indistinguishable from one that never made it, and the VM may well have moved), and a burst of power ops queues one sync rather than one each — the last op to finish queues it, and it sees everything the burst moved. A graceful shutdown or reboot is the case where one press is not enough: the sync runs seconds later, while the guest is still on its way down, so it honestly records a VM that is still running.

Not every button on those pages has one of them behind it, and the ones that do not are refused rather than approximated. restart is why the mapping is per product rather than shared: each kind resolves it differently — Proxmox /status/shutdown (graceful), vSphere ?action=reset (a hard reset), XCP-ng VM.clean_reboot (a reboot), Hyper-V nothing at all. shutdown and reboot exist because that made every Shutdown button unmappable, and one Reboot button too:

Two of those are graceful in the strict sense — they ask software inside the guest. A vCenter Shutdown is not a power action at all but a call to /api/vcenter/vm/{vm}/guest/ power, and it answers 503 when VMware Tools is not running; the agent turns that into a message saying so, because "answered 503" on a plainly-running VM points nowhere. Hyper-V Shutdown is bare Stop-VM, which Microsoft documents as shutting down "through the guest operating system" — it carries neither -TurnOff (the power cut) nor -Force, which on Stop-VM means "regardless of any unsaved application data" and would quietly make it a different promise. Use Force Off when the guest cannot answer.

Hyper-V is the one product with no graceful reboot at all: Restart-VM is documented as a "hard" restart, "like powering the computer down, then back up again", so it is what the Restart button's power_reset runs and there is nothing left for reboot to be. The agent refuses that combination by name rather than letting the runner answer "unknown verb", which would read as an agent too old for the dashboard.

The mapping is one table, agent_hypervisor_meta.PAGE_OPS, keyed by kind — it was once a copy per router, and three identical copies is how shutdown came to hard -reset a vCenter VM. A refused button is greyed out on the page with the reason in its tooltip, and the endpoint answers 501 naming the substitution that would have been wrong and the buttons that do work. The refusal happens before a job row exists, so nothing appears on /jobs.

Adding a verb is a three-file change, and a partial one is worse than none. The dashboard normalizes an unrecognised verb to inventory_sync, so a verb granted here but missing from an agent would run a discovery scan and report success. The three are this allowlist, the agent's per-kind maps, and the sibling runner.

Version skew across them is safe in the direction it actually happens — the agent is deployed separately and lags. An old agent given a new verb refuses it out loud: it reads the verb raw and never normalizes, so policy.check_verb rejects it first, naming policy.yaml and the line to add. This is why shutdown and reboot are new verbs rather than a redefinition of restart: redefining a verb an old agent already implements, and an old policy.yaml already grants, would silently change what that agent does with a button — the exact failure the whole table exists to prevent.

The corollary is operational: after upgrading the dashboard, every deployed agent must be re-pulled before its Shutdown button works, and until then those agents refuse the verb in Live Output. Proxmox Shutdown is the one button this affects that worked before — it used to ride restart, which really is /status/shutdown on Proxmox alone. It was moved anyway, because leaving it would have kept "restart means shutdown here" alive as a per-kind special case, and that reading is what made Reboot unmappable in the first place.

snapshot creates a snapshot named dash-<job id>. The name is generated, never supplied — which is exactly why it was held back at first: a created thing needs a name, and a name is a free-form string. There is still no field in the protocol through which operator text could reach a hypervisor, and the job id makes the snapshot traceable back to the row that made it, which a typed name would not be.

Deliberately absent: deploy / clone / delete / console — they need sizes, networks and cloud-init, a payload shape indistinguishable from a config file, and a config file is one step from a script. Those stay dashboard-direct.

power_off and power_reset are separate verbs rather than one verb with a force flag, because a boolean on a destructive verb gets defaulted wrong exactly once.

What the agent can and cannot reach

esxi is a distinct connection kind from vsphere on the agent's side even though the dashboard has only vsphere: same product, different transport, and the agent is the only side that knows which one a given endpoint is. A job for vsphere is served by an esxi connection; the reverse is refused, because that direction would send SOAP work to a vCenter.

Four of six products over REST, and no new agent dependency for any of them. That is why the agent image still installs only requests, PyYAML and cryptography: those two audit tests are the security argument in executable form. The remaining two get their heavy dependencies in a container that lives for seconds, which keeps the long-lived supervisor inert — the property test_the_agent_imports_no_execution_machinery protects.

Where the credential comes from

Five sources. A remote source beats a local one — ps_managed_account or dashboard_secret, then password_sealed, then password_file, then an inline password — because an operator who has moved a connection off local storage should not silently keep authenticating with a stale literal left in the file underneath it. A leftover is warned about on every job rather than ignored quietly, since it means plaintext is still sitting on a host you meant to clear.

password_sealed is not a fourth authority; it is the same local credential, not written down in the clear. That is why it is ordered against the two plaintext forms rather than being exclusive with them, and why a sealed value found in password: — or in the file password_file points at — is refused rather than sent. Sent as a literal it would come back as a wrong password, which reads as the wrong problem entirely.

A connection that declares none of the five is refused, naming all five keys. It used to fall through to an empty password, which the endpoint answers as a wrong one, so a connection that simply had no credential looked like a connection with a bad one — and on the inventory sync schedule it retried until the service account locked out.

The two remote sources are mutually exclusive rather than ordered. They are different authorities — the agent asking Password Safe, versus the dashboard asking on its behalf — there is no stale-leftover story that makes preferring one kind, and choosing quietly would leave nobody able to say which credential a job actually used. Declare both and the agent refuses the job and names the file.

password_file — and client_secret_file in passwordsafe.yaml — are read with the same encoding rules as the enrolment code file: a UTF-8 BOM is stripped, and a UTF-16 file is refused with a message naming the encoding rather than a decode traceback out of a job. Write them with Set-Content -Encoding ascii -NoNewline on Windows.

With ps_managed_account, the agent holds no hypervisor credential at all — only a Password Safe OAuth client whose single power is to ask for one. Each job checks a credential out and checks it back in, so every use lands in Password Safe's audit trail and is subject to its policy and approval workflow. That gets the agent close to the end state this design was heading for: no standing hypervisor credential on the host.

It does not get all the way there, and the remaining gap is worth naming: the Password Safe OAuth client is itself a credential on this host, and its entitlements are usually broader than the single account it is being used for. dashboard_secret closes that gap by moving the asking to the dashboard, which already holds such a client for other features. Pick one — declaring both is a refusal, not a precedence.

Mount the client alongside the other two files — see passwordsafe.example.yaml. The dashboard never issues it and never sees it.

A check-in failure is swallowed deliberately: the credential has already been used and the request expires on its own duration, so failing a completed job over it would be the wrong trade. An account that requires human approval will not release, and the job fails rather than hanging — use auto-approve policies for accounts an unattended agent needs.

The sibling runner

Hyper-V and bare ESXi are the only two the agent cannot talk to itself: WinRM is SOAP with NTLM/Negotiate, and a standalone ESXi host serves only the SOAP API. Rather than put a real auth stack and pyVmomi into an image whose three-dependency restraint is the security argument, those two run in a one-shot container the agent creates, reads one line of JSON from, and deletes.

This needs the Docker socket, and the socket is root on the host. It is not mounted by the default deployment — docker-compose.yml still promises the agent launches nothing, and that stays true. Turning it on is a separate, deliberate act:

Withhold any one and nothing runs. Prefer the rootless Docker or Podman user socket; the example overlay uses the rootless path deliberately, so reaching for the root socket has to be a conscious edit.

What the agent does with it is deliberately narrow. Every field of the container spec is a constant or comes from your policy — the image, the network, and a HostConfig with no bind mounts, no capabilities, a read-only root filesystem and no-new-privileges. None of it is derived from anything the dashboard sends, because there is no field through which to ask; a test asserts that. The credential rides in the environment of the create call rather than argv, so it never appears in ps on the host. Containers are labelled and orphans from a crashed agent are swept at startup.

The agent will not pull the image for you. A pull is a network fetch of executable content, and that is your decision rather than a job's:

docker pull chrweav/hypervisor-runner:latest

VMware Workstation Pro

Workstation is the one hypervisor here that runs on somebody's desktop — nearly always a Windows desktop, so read Running on Windows alongside this. It was twice written off as unreachable because the dashboard drives it with vmrun against local VMX paths. That was true of vmrun and wrong overall: Workstation Pro ships vmrest, a REST daemon that is plain JSON over HTTP. An agent on that host reaches it with no extra dependency and no container.

On the Workstation host:

vmrest -C     # set the API credentials, once
vmrest        # run the daemon — 127.0.0.1:8697

Then add a workstation connection bound to that host's agent. It is agent-bound only: the dashboard has no transport for Workstation, so a connection without an agent is refused rather than created and left broken.

vmrest binds 127.0.0.1, and that address means something different inside a container. The agent denies loopback unconditionally — that deny is what stops a discovery sweep probing the agent's own container or a cloud metadata endpoint, and it is re-added even if an operator deletes it. But lifting the deny is only half the problem: inside the container 127.0.0.1 is the container, so the address has to be one that actually reaches the host as well as one the policy permits. Which of the two you need depends on how the agent is attached to the network:

• Docker Desktop on Windows or macOS — point the connection at host.docker.internal:8697 and add it as a target. Docker Desktop routes that name into the host's own network namespace, which is what makes a loopback-bound vmrest reachable at all. It is not a loopback address from the agent's side, so allow_loopback is not needed:

yaml targets: - fqdn: host.docker.internal ports: [8697]

The name is resolved once when the policy loads and pinned to the address it returned, so the allow-list is in IPs by the time a connection is checked.

• An agent sharing the host's network namespace (Linux, --network host) — here 127.0.0.1 really is the host's loopback, and the connection opts out of the deny explicitly in policy.yaml, next to its verbs:

yaml connections: - name: my-workstation verbs: [inventory_sync, power_on, power_off] allow_loopback: true

That exempts that connection, on the port its connections.yaml entry names, and nothing else. Discovery still refuses loopback however this is set.

power_on and power_off are not optional if you want the buttons. With only inventory_sync the VMs list correctly on the Workstation page and Start/Stop are refused by the agent — the refusal names this file and the verb to add, but it is the most common "I set it all up and the buttons still do not work" outcome. Grant them when you write the entry, not after.

If you can make vmrest listen on a routable address instead, do that and skip both exceptions — it is the only one of the three that needs nothing special from the container runtime. Note that vmrest documents a -p port option but no bind-address option, so on a stock install this usually is not available to you.

Inventory plus power on and off — and nothing more. vmrest's API is on/off/shutdown/suspend/pause/unpause with no reset, no reboot and no snapshot, so restart, power_reset and snapshot are refused with a message saying so. Mapping restart onto shutdown would quietly do something other than what was asked.

Synced VMs appear on the Workstation page alongside the ones this host scans locally, badged with the agent's name. They are tagged into workgroups the same way Proxmox and Nutanix VMs are, and an untagged VM is admin-only — which is what stops an agent widening what a non-admin can see.

Large inventories

A sync returns one page plus an opaque cursor; the dashboard applies it and enqueues the next. The chain is capped at 40 pages (10 000 VMs), which is what stops a misbehaving agent making the dashboard enqueue work forever. Every page of one sync shares a batch_id, so N job rows roll up as one run on /jobs.

The cap that forces this is MAX_RESULT_BYTES (256 KB), and raising it is not an option — it is the only bound on an agent's write path into the database.

Agent-executed Config Management

Discovery finds hosts and hypervisor brokering powers them on. This applies a playbook to them — to the guests, and to on-premises databases, over SSH or WinRM, from an Ansible process that runs inside your network.

Why this is the feature that makes on-prem hypervisor support worth having. Most on-prem hypervisors have no usable Terraform provider, so the dashboard cannot build their VMs. All of those VMs speak Ansible. And a cloud-hosted dashboard cannot run Ansible against them at all by itself: the "Local Docker" runner is a sibling container on the dashboard's host, which has no Docker socket on ECS/ACI/Container Apps and no route to your LAN in any case. Moving that container onto the agent's host is the whole change.

What the dashboard sends, and what it does not

This is the part worth reading closely, because a playbook is executable content and the philosophy above says the dashboard is not trusted.

The signed job envelope carries four scalars: the kind of run, the transport, and the target address and port. That is all. No playbook, no filename, no variable name, no credential — services/agent_ansible_meta.py is a closed allowlist and a test asserts none of its fields can name a thing to fetch or run.

Everything else arrives sealed, from POST /api/agent/jobs/{id}/ansible-bundle, encrypted to an X25519 key the agent generated for that one fetch — the same mechanism as the credential the dashboard holds, and for the same reason: a signature proves the dashboard sent it, which is a weaker claim than the seal's "sent for this agent, this job and this endpoint". The AAD binds the endpoint, so a bundle released for one host cannot be relabelled as another's.

The agent writes the Ansible inventory itself. This is not tidiness. An inventory is a place to write ansible_connection: local, which turns "configure that VM over SSH" into "execute this playbook inside the runner container" — on your network, with the runner image's kubectl and helm on PATH. ansible_python_interpreter and ansible_ssh_executable are the same hole in different clothes, and -e outranks every inventory variable in Ansible's precedence order, so filtering only the inventory would be worth nothing. So the bundle carries typed credential fields — one key per meaning — the agent renders the inventory from the four verified scalars, and any ansible_* extra var is refused by name. The address in that inventory is the resolved IP, pinned, because the policy check happens here and the connection happens seconds later in another container.

Four grants, all required

ansible.targets is a separate list from targets, and that is the point. The top-level list grants a port probe; this one grants a playbook running as root. An operator who widened a discovery sweep has not agreed to the second, so there is no fallback: an empty ansible.targets means nothing can be configured, however wide targets is. Name the ports — 22 for Linux, 5985/5986 for Windows, the database port for a database.

Both images come from policy.yaml and never from a job. A job names only the kind:

docker pull chrweav/ansible-winrm:latest     # VM targets — SSH and WinRM
docker pull chrweav/ansible-cloud:latest     # database targets — the localhost play

The agent will not pull either for you, deliberately — a pull is a network fetch of executable content.

Before a VM appears as a target: it needs an address

Get-VM and /api/vcenter/vm describe the VM, not the guest inside it, so a synced row has no address until something asks for one. Three things must all be true:

1. the guest is powered on;
2. guest tools are installed in it — Integration Services, VMware Tools, or qemu-guest-agent;
3. sync_guest_details: true on that connection in your connections.yaml.

The third is off by default because reading each guest's addresses costs an extra call per VM on every sync, and an estate whose guests are not config targets should not pay it. Set it, Sync Now, and the guests become individually selectable. Until then the row is listed but disabled, with that reason on hover — which is the answer to "why is my VM not in the list".

Supported on hyperv, vsphere and proxmox. Nutanix and XCP-ng report no guest address yet and say so on the row rather than appearing broken. Hyper-V needs a re-pulled chrweav/hypervisor-runner as well as the agent, because that is the image doing the asking.

On-premises databases

Register the database with cloud = local and bind it to an agent. The run is the same hosts: localhost play the cloud databases use — community.postgresql / mysql / general reaching out to the endpoint — except the controller is a container on the agent's host instead of an in-cloud task. Its admin credential is checked out of Password Safe just-in-time by the dashboard and sealed into the bundle, so nothing durable sits on the agent.

What the run looks like

A one-shot container, created per job and deleted after it: --read-only, --cap-drop ALL, no-new-privileges, no bind mounts, 1 GiB of memory and a 128 MB /tmp. Every field of that spec is a constant — a test asserts none of it comes from the job.

The playbook, the inventory, the SSH key and the vars go in through the Docker archive API rather than the environment. That is a deliberate departure from the cloud runners' PLAYBOOK_B64 contract, and the reason is unforgiving: execve caps a single environment string at 128 KB on a 4 KB-page host and 2 MB on a 64 KB-page arm64 one, so an env-delivered playbook works on the machine it was developed on and fails on a customer's with argument list too long. Files land mode 0600 and appear in no docker inspect output.

They are extracted into /opt/job, which is an anonymous volume — the one addition to that otherwise mount-free spec, and not optional. The Engine refuses an archive extract with 400 container rootfs is marked read-only on any container whose HostConfig sets ReadonlyRootfs, unless the destination resolves into a mount; it decides that from HostConfig alone, so extracting before the container starts does not get around it. The volume is anonymous, so it still names no path on your host, and it is removed with the container. A tmpfs will not do in its place: it accepts the extract and is then mounted empty over the top of it at start, which loses the run's files with no error at all.

Output streams into the existing Live Output pane, and the existing Cancel button works: a watcher sends SIGTERM to the container, then SIGKILL after ten seconds. There is also a wall-clock ceiling, ansible.max_runtime_minutes (default 30).

Ansible's own exit codes are reported in words rather than as a number, because "exit 4" sends people to a search engine: 2 is "one or more hosts failed a task", 4 is "the target was unreachable". A non-zero exit fails the job — it is never reported as a completed run, which for config management would be the worst possible outcome.

Requires an agent image of 2.3.0 or newer

The dashboard refuses to queue for an older one. An older agent has no agent_ansible entry in its closed HANDLERS dict, so it would refuse the job into Live Output — where the message reads as a policy.yaml problem and sends you to edit the wrong file. Refusing at enqueue puts the remedy where it is legible and leaves no job row behind.

Best practices

• Run in audit mode first. AGENT_MODE=audit logs every job it would run, in full, and executes nothing. Two weeks of that, diffed against the policy, is usually what gets an agent approved by a security team.
• Ship the agent's stdout to your SIEM. The dashboard is not the authoritative audit record for an agent — if it were, a compromised dashboard could delete the evidence of its own compromise. The container log is the copy it cannot reach.
• Prefer rootless Podman on hosts where more than one person is in the docker group — see Running under Podman for what it does and does not buy you.
• Name ports in the policy, not just networks.
• One agent per site, named for the site.
• Watch the policy hash on the Agents page. It only changes when the file does.
• Re-enrol rather than reuse. Issuing a new code clears the stored public key, so the container being replaced stops being able to lease work immediately.

Rate limits

The agent vhost is the only surface this dashboard deliberately publishes to a network it does not own, so it is the only one with a throttle of its own (services/agent_guard.py). Nothing else in the app is rate limited — the SlowAPI limiter in main.py is inert on purpose, because a blanket per-address cap would break a UI that fires many calls per page load.

Three caps, keyed differently because only one of these routes has a principal to key on:

Set agent_throttle_enabled false to disable, or any cap to 0. The per-agent cap keys on the agent id rather than the address for the same reason the login throttle keys on the username: an address is only as trustworthy as TRUSTED_PROXY_HOSTS, while an agent id is proven by a signature the caller cannot forge.

240/minute is about three times what a busy agent does. While a scan runs it reports every two seconds — roughly 60 requests a minute — and idle polling at the default interval is 12. If you lower this, lower it toward 60, not toward the poll interval.

Over a cap the answer is 429 with Retry-After, never 401, and the agent waits the interval it was given with jitter so a fleet throttled together does not come back in lockstep. Only failed enrolments count, so enrolling a fleet back to back trips nothing.

Neither cap is brute-force protection — an enrolment code is 256 bits and every other route needs a valid signature. They bound denial of service and table growth, which is what an unauthenticated internet-facing endpoint actually needs.

Behind a TLS-inspecting proxy

The property this section describes is the reason the agent signs requests instead of carrying a bearer token, and it extends to credential responses: a credential fetched with dashboard_secret is sealed to a per-fetch key inside the TLS session, so an inspecting proxy that reads the whole body reads a ciphertext. That was the objection to holding credentials centrally at all, and it is why this is encrypted at the application layer rather than trusted to TLS.

Mount the inspection CA and point AGENT_CA_BUNDLE at it — docker-compose.corp-ca.yml is the same pattern for the dashboard. HTTP_PROXY / HTTPS_PROXY / NO_PROXY are honoured automatically.

They apply to the dashboard connection only. The discovery probes are raw sockets and ignore proxy variables entirely, so NO_PROXY is not needed to keep a scan off the corporate proxy — a scan was never going through it. What NO_PROXY is for here is the narrower case of the dashboard itself being on a network the proxy should not be used for.

A bad AGENT_CA_BUNDLE path is worth knowing about because it fails badly: requests raises a bare OSError("Could not find a suitable TLS CA certificate bundle, invalid path: …"), which is not a RequestException and so is not caught — the agent dies with a traceback and exit 1 rather than a readable message.

Worth saying plainly: an inspecting proxy sees the full content of every request. That is exactly why the agent's credential is a per-request signature rather than a bearer token — the proxy captures nothing it could replay. This turns the corporate proxy from an objection into a demonstration.

Troubleshooting

Where this is heading

Discovery was the first slice, chosen because no credential crosses the wire at all. Hypervisor brokering followed it and is described above. Next:

• ~~Agent-executed Ansible against private targets~~ — shipped; see Agent-executed Config Management. One detail of the plan did not survive contact and is worth recording: it was to reuse the cloud runners' PLAYBOOK_B64 env contract byte for byte, and that is not safe. execve caps a single environment string at 128 KB on a 4 KB-page host and 2 MB on a 64 KB-page arm64 one, so an env-delivered playbook works on the machine it was written on and fails on a customer's. The files go in through the Docker archive API instead, which also keeps them out of docker inspect. The just-in-time secret fetch landed as planned, and grew to carry the playbook as well — because executable content must be sealed, not merely signed.
• ~~Password Safe JIT checkout by the agent~~ — shipped, and then superseded for the case it was aimed at. See Where the credential comes from for the agent-side checkout, and the credential the dashboard holds for the variant that leaves no credential on the on-prem host, not even a Password Safe client.
• Nutanix power verbs and snapshots. Both are full spec PUTs carrying a metadata version rather than simple actions, so getting one wrong writes to the VM instead of failing. Worth doing carefully rather than quickly.
• Nutanix and XCP-ng guest addresses. Both sync VMs but report no guest IP, so their guests cannot be Config-Management targets yet — the rows say so rather than failing. The data is there (Prism's nic_list, XAPI's VM_guest_metrics); it is a producer change in each sync, gated behind the same per-connection sync_guest_details flag.
• On-premises Kubernetes clusters as agent targets. cloud="local" clusters have the identical problem an on-prem database had, and the fix is the same shape — an agent_id on the cluster row and the existing run_kind enum grown a third member.
• Retiring POWERSHELL_EXECUTION_MODE=ssh, now that a co-located agent does the same job by polling outward instead of the dashboard holding an inbound SSH key to a Windows desktop.
• ~~VMware Workstation over a co-located agent~~ — shipped; see VMware Workstation Pro.
• The rest of what was deferred. Not network-reachable, so this is a new deployment shape rather than one more connection kind — the agent would have to run on the desktop host and speak to vmrest on localhost. It would replace the existing POWERSHELL_EXECUTION_MODE=ssh dev escape hatch, which does the same job today with an inbound SSH key.

config.py's singleton hypervisor keys (one proxmox_host, one vsphere_host) were the blocker for all of this. They are now a one-time seed for the hypervisor_connections table rather than the source of truth; the old Settings panels are read-only and say so.