← Back to dashboard

Design: SPIRE for agents, Dex for people, and nothing existing removed

Audience: contributor · Profile: both · Read this when: you are about to give the remote agent a SPIFFE identity, put Dex in front of the dashboard's SSO or a cluster's API server, or you are wondering why the Ed25519 agent path was not retired when SPIRE arrived.

Partly built. What exists: - the two compose overlays: docker-compose.spire.yml for the dashboard and examples/remote-agent/docker-compose.spire.yml for the agent, - SPIRE attestation for agents: POST /api/agent/attest, the Migrate to SPIRE action and banner, and central credential storage for attested agents (agent 2.6.0 and later), - the Dex lab plays: examples/playbooks/dex/ and examples/playbooks/k3s/k3s-dex-auth.yml, and the on-prem Dex kubeconfig for cloud="local" clusters, - the shared AuthenticationConfiguration handling that lets the SPIFFE and Dex authenticators sit on one k3s API server.

Phase 2 lists each of these with its details, then what is still to build (Dex as the dashboard's SSO provider, a Dex overlay for the dashboard host, and the rest). Managed clusters on Dex are paused. None of it has run against a live host. The dashboard's own SPIFFE identity, which came after this note, has its own: dashboard-workload-identity.md.

The problem

There are two problems, and they are different enough that one tool should not solve both.

1. The remote agent keeps a long-lived private key on disk. - runners/agent/agent.py generates an Ed25519 key at enrolment and writes it, unencrypted, to /var/lib/dashboard-agent/identity.json. - docs/remote-agents/agent-host.md is honest about this: whoever can read the volume is the agent until someone re-issues it.
2. People reach Kubernetes six different ways. - EKS, AKS, GKE and OKE each have their own token exec plugin and their own user model. - k3s and other on-prem clusters have whatever the operator set up. - The dashboard's own SSO is a separate OIDC client again. - There is no single place where "this person, in this group" means the same thing everywhere.

The split - SPIRE is for workloads. It attests a machine or process and issues short-lived SVIDs. The agent is a workload. - Dex is for people. It federates upstream human IdPs (Entra, Okta, LDAP, GitHub) into one OIDC issuer that every cluster and the dashboard can trust.

Dex was looked at for the agent first and rejected for it. See Rejected.

The rule: additive

Nothing here removes or changes a path that works today, with one deliberate exception: how people get a kubeconfig for an on-prem cluster (see On-prem: the one path that changes).

Three reasons the legacy agent path is permanent rather than deprecated: - The proxy constraint below. Some sites can never reach a SPIRE server. - Agents already in the field must keep working after a dashboard upgrade, with no action from whoever runs them. - Rollback. If a SPIRE migration goes wrong, the fix is to re-issue an agte_ code, which is the procedure operators already know.

The dashboard recommends SPIRE with a banner. It never forces it. See The banner.

Part 1: the agent attests through SPIRE

Two facts that bound what this buys

1. "No private key on disk" depends on the node attestor. It is not a given.

The SPIRE agent has to prove which node it is on before it gets an SVID. How it does that decides whether a key is stored anywhere:

• With join_token the SPIRE agent still stores a key. That is better than today: the key lasts an hour instead of indefinitely, rotates, and can be revoked centrally. But it is not zero, and the docs must not say it is.
• join_token + memory is not an option. The token is single-use, so the first restart would leave the agent unable to attest.
• k8s_psat needs the SPIRE server to call that cluster's TokenReview API. That works for a cluster the dashboard already manages. It does not work for an on-prem cluster behind a firewall, which is exactly where remote agents live. For those, the agent is a join_token host, or a TPM host.

2. SPIRE agent ↔ server traffic is mTLS gRPC. TLS-inspecting proxies break it.

agent_signing.py explains why the agent signs requests instead of using mTLS: an inspecting proxy terminates TLS and cannot forward a client certificate. The SPIRE agent's connection to the server is mTLS gRPC on port 8081, so it hits the same wall.

• A site behind an inspecting proxy needs a bypass rule for the dashboard's SPIRE host and port.
• Without one, the agent cannot attest, and the site stays on Ed25519.
• That is the main reason the legacy path is permanent rather than a migration with an end date.

sealing.key is a separate key and is out of scope. - It is the agent's host-local AES key for password_sealed: / client_secret_sealed: values (agent.py:438). - It is not an authentication credential, and SPIRE does not replace it. - It stays in the agent_state volume in both modes. The agent overlay keeps that volume for this reason.

The flow

spire-agent (sidecar)                dashboard-agent (uid 10001)               dashboard
──────────────────────               ───────────────────────────               ─────────
node attestation ──── mTLS gRPC :8081 ───────────────────────────────────────▶ spire-server
                     ◀── agent SVID (memory or disk) ──

                     ◀── Workload API (unix socket, attested by uid 10001) ──
                         FetchJWTSVID(aud = <dashboard>/api/agent/attest)
                                     generate Ed25519 keypair IN MEMORY
                                     POST /api/agent/attest ──── HTTPS ───────▶ verify SVID
                                       body: {svid, public_key}                  (spiffe_assertion:
                                       signed with the NEW key                    sig, aud, lifetime
                                                                                  cap, single use)
                                                                                 map SPIFFE ID →
                                                                                   RemoteAgent row
                                                                                 bind public_key
                                     ◀── {agent_id, dashboard_public_key, audience}
                                     every later request: the EXISTING signed path
                                       (X-Agent-Id / -Timestamp / -Nonce / -Signature)

• One token crosses the proxy per agent start. It is audience-bound to the attest route and lives at most default_jwt_svid_ttl (5 minutes in the lab config). spiffe_assertion records its hash, so it is single-use: a copy lifted from a proxy log after the agent has used it is refused.
• Signing after that is unchanged. signed_agent, agent_service.authenticate and the nonce table are not modified.
• Request signing is not what puts a key on disk. Persisting the key across restarts is. Keeping signing means the proxy-safety argument in agent_signing.py still holds without change.
• Signing in the other direction is unchanged too. Job envelopes are still signed with the dashboard's own key, and the agent learns that public key from the attest response, as it does at enrolment today.
• On restart the agent attests again. It writes nothing to identity.json.
• Workload attestor: unix, selector uid:10001. The two containers share a PID namespace (pid: "service:spire-agent"), so the SPIRE agent can see the caller's uid.
• The alternative, the docker workload attestor, needs /var/run/docker.sock mounted into the sidecar. That is root on the host, and the agent's whole hardening story (docs/remote-agents/agent-host.md) is that nothing on this host has it.

How each side picks a mode

Agent side - AGENT_SPIFFE_JWT_FILE set → attest through SPIRE at every start, key in memory, identity.json neither read nor written. - Otherwise → today's behavior exactly: use identity.json if present, else redeem AGENT_ENROLLMENT_CODE. - A new agent image without the sidecar is indistinguishable from the old one.

Where the token comes from (as built). Not the Workload API directly: that is gRPC, and the agent image deliberately carries four Python packages. A spiffe-helper sidecar runs as the agent's uid (10001), so the unix:uid:10001 entry matches it, and keeps a JWT-SVID for <DASHBOARD_URL>/api/agent/attest in a memory-only (tmpfs) volume that only uid 10001 can read. The agent reads that file. It needs no socket and no shared PID namespace; the helper has both. A token that a previous start already presented is refused as a replay, so the agent waits for the helper's next rotation (half the token's lifetime) instead of failing.

Server side - RemoteAgent gets two new columns: - auth_mode (ed25519 | spiffe, default ed25519), - nullable spiffe_id (unique). - The migration (the additive ALTER TABLE list in database.py) needs no backfill: every existing row is ed25519 by default, which is what it already is. - /api/agent/attest accepts only rows whose spiffe_id matches the SVID's sub. An ed25519 row cannot be taken over by someone who happens to hold an SVID.

Migrating one agent

1. An admin clicks Migrate to SPIRE on the agent's row. This needs the SPIRE server enabled (spire_server_enabled).
2. The dashboard creates the SPIRE entries through the server's local admin socket: - a node entry for the chosen attestor (or a join token), - a workload entry spiffe://<td>/agent/<agent-id> with parent = that node and selector unix:uid:10001.

It then sets spiffe_id on the row. auth_mode stays ed25519 for now. 3. The operator adds the sidecar overlay on the agent host and restarts. 4. On the first successful attest, the server sets auth_mode = spiffe and nulls the old public_key in the same transaction. - This mirrors what re-issue does today. The legacy key stops working only once the new path has proven itself. - Agent id, job history and policy hash carry over. 5. Rollback: re-issue an agte_ code. Re-issue already clears the key. It now also clears spiffe_id and sets auth_mode = ed25519, and deletes the SPIRE entries.

The banner

• Shown on the Agents list and on the agent detail view, for each agent with auth_mode = ed25519, only when spire_server_enabled is on. A dashboard that has not opted in to SPIRE never shows it.
• Text depends on agent_version, which every poll already reports and which api/agent.py already gates on through _major_version:
• below the first SPIRE-capable major: "Update the agent image, then migrate it to SPIRE to stop storing a long-lived key."
• otherwise: "This agent can move to SPIRE.", with the Migrate button.
• Dismissible per agent. "Not now" is a legitimate answer for a site behind an inspecting proxy, and a banner that cannot be dismissed trains people to ignore banners.

The dashboard-hosted SPIRE server

• Packaged as an opt-in compose overlay, docker-compose.spire.yml. It reuses the lab's hardening and image-digest pins from examples/playbooks/spire/spire-docker-server.yml, and tests/test_spire_overlay.py keeps the two from drifting.
• It stays on docker-compose, as the roadmap requires (docs/editions/roadmap.md, "Feasibility flag").
• Port 8081 is published directly, not through Caddy.
• Caddy terminates TLS, and SPIRE's agent-to-server channel is mTLS that has to reach the server untouched.
• Passing it through would need Caddy's layer-4 plugin, a custom build for one port.
• The admin API stays private. It is a unix socket on a named volume that only the dashboard containers mount. Nothing about administration listens on the network.
• CA key storage
• disk KeyManager in community.
• The hosted topology should use aws_kms / azure_key_vault / gcp_kms, which is the same move the roadmap already made for the JWT root key.
• Gated by spire_server_enabled. It is off by default. While it is off:
• no attest route answers,
• no banner shows,
• no Migrate button exists.

What the server is used for next (the dashboard as a workload in its own trust domain, federated cloud credentials, and Workload Lab demos that no longer need the lab's server) is in The dashboard as a SPIFFE workload.

Part 2: Dex for people

What changes, and what doesn't

• Dashboard SSO: Settings → Identity → SSO provider
• direct (default) is today's behavior.
• dex fills oidc_issuer / oidc_client_id / oidc_client_secret from the Dex settings. The same oidc_service code path runs either way.
• Dex is a different issuer URL, not a different integration.
• Clusters: how people authenticate depends on where the cluster runs.

• Managed: paused. The design below was optional and never the default; it is now not being built. Kept for the record: Not only on AKS, where the cluster side is still preview. One default for all four is easier to reason about than a per-cloud exception that changes when a preview ends. A cluster would have flipped to dex only once its API server trusted Dex (the Terraform wiring below).
• On-prem: Dex is the only way. A self-built cluster has no cloud identity to be "native" to; the alternative is handing people copies of the admin client certificate, which cannot be revoked per person and leaves no per-person audit trail. So kubeconfigs the dashboard generates for people on a cloud="local" cluster always use the oidc-login exec plugin against Dex, and the setting is not shown for those clusters.
• The on-prem admin kubeconfig is not removed. It is how the dashboard registers and operates the cluster, and it is the break-glass path if Dex is down. It stays where it is stored today (the encrypted config store or an external vault) and is never offered to a person.
• Consequence for building an on-prem cluster: k3s-dex-auth.yml becomes a required step after k3s-server-init.yml, not an extra. Until a cluster's API server trusts Dex, the dashboard has no way to give people access to it, and says so on the cluster's row rather than falling back to the admin kubeconfig.
• dex generates kubeconfigs that use an oidc-login exec plugin against Dex; native is unchanged from today.

Managed clusters: paused

Dex for managed clusters is paused, not scheduled. Each managed cloud already gives people its own short-lived, per-person identity: the EKS, GKE and OKE exec plugins, and Entra on AKS. Dex in front of those would add one more issuer to run and make public, and a Terraform change per cluster, to replace something that already works per person. The standardization it buys is real but small next to that cost.

So for EKS, AKS, GKE and OKE: - native is the only mode. k8s_human_auth_mode and its per-cluster override are not being built. - The dex_issuer_url Terraform wiring for the four managed-cluster modules is not being built. - The managed rows of Every cluster type can trust Dex stay as reference, not as a plan. Re-check them before reviving this: the GKE row in particular should be verified against current GKE guidance (Workforce Identity Federation) rather than taken as written.

On-prem is unaffected. Dex stays the only way people reach a cloud="local" cluster (built in #995), because there is no cloud identity to be native to.

On-prem: the one path that changes

Today a registered cloud="local" cluster shows the API tunnel button, and its download (GET /api/k8s/clusters/{id}/api-tunnel-kubeconfig → k8s_service.build_api_tunnel_kubeconfig) returns the stored kubeconfig, repointed at the tunnel. - For EKS, GKE and AKS that file is token-free: its users entry is the cloud's exec plugin, so each person authenticates as themselves. - For k3s the stored kubeconfig is the admin one from k3s-kubeconfig.yml, with the client certificate and key inline. So the same download hands cluster-admin credentials to anyone with k8s:read. That credential cannot be revoked per person, and the audit trail cannot tell people apart.

"Dex is the only way on-prem" closes that. In Phase 2, for cloud="local": - the route returns a kubeconfig whose users entry is the Dex oidc-login exec block (cluster CA kept, no client certificate, no key); - if the cluster's API server does not trust Dex yet, the route refuses with a message naming k3s-dex-auth.yml. It never falls back to the stored file.

The stored admin kubeconfig keeps doing what it does for the dashboard itself (registration, the transient runners, break-glass). It just stops being downloadable by people. This is the only behavior in this note that is removed rather than added, and it is removed because it hands out a shared admin credential. - Not affected: the transient-runner kubeconfig (k8s_service._runner_kubeconfig). It swaps in a server-minted token for the dashboard's own runs and has nothing to do with people. - Groups: Dex passes the upstream IdP's groups through in the groups claim. The cluster maps them to RBAC with a prefix (dex: in the lab play), so a Dex group can never collide with a built-in system: group.

Every cluster type can trust Dex

Standardizing on Dex is therefore always possible: mandatory on-prem, and technically available on every managed cloud. The managed half is paused.

On every managed row this is paused (see Managed clusters: paused); the rows are kept as reference. On the k3s row it is required and built.

• Each Terraform change takes a dex_issuer_url variable. Empty means no change, so every existing cluster plans with an empty diff.
• This corrects an earlier claim. The header of k3s-spiffe-auth.yml said managed clusters cannot be pointed at an external issuer. That is true for that play's workload case: arbitrary --authentication-config, an issuer on a private address, a SPIFFE CA. It is not true for a public OIDC issuer used by people.

Dex must be public

Managed control planes fetch the issuer's discovery document and JWKS from the internet.

• A Dex instance that only resolves inside a lab can serve k3s and nothing else.
• To serve EKS, GKE, OKE or AKS, Dex needs a public DNS name and a certificate from a public CA.
• The k3s play also accepts a private CA, through dex_ca_pem, for the lab case.

Dex and SPIRE on one k3s API server

k3s takes one --authentication-config file. That file holds a list of JWT authenticators, so both can live there, as long as neither play overwrites the other's entry. The rules:

• Each play owns its entries by username prefix.
• k3s-spiffe-auth.yml owns entries whose claimMappings.username.prefix is its username_prefix (spiffe:).
• k3s-dex-auth.yml owns dex:.
• Each play reads the current file and replaces only its own entry, in place (appending it the first time). Re-running either play never removes or reorders the other's entry. In place matters: if each play moved its entry to the end, running them alternately would rewrite the file, and restart k3s, every time.
• The prefix is the ownership key, not the issuer URL. Kubernetes rejects unknown fields in this file, so there is nowhere to put an ownership marker.
• Keying by prefix also means a play re-run with a new issuer replaces its old entry instead of leaving a stale second one.
• Both plays point the API server at the file through the same drop-in. The file and drop-in names keep their historical spiffe- spelling, so an existing SPIFFE-linked node is upgraded in place and never ends up with two authentication-config flags.
• Teardown. k3s-spiffe-unlink.yml and dex-remove.yml each remove only their own entry. The drop-in goes only when the list is empty, and k3s restarts only when the file actually changed.
• Bug fixed along the way. The unlink play used to target paths the link play never wrote (90-spiffe-auth.yaml, /etc/rancher/k3s/spiffe-auth.yaml), so unlinking left the flag in place.

Rejected

• Dex for the agent.
• Dex has no way to attest a machine. An agent would authenticate to Dex with client_credentials, swapping an Ed25519 key for a client secret, which is still a secret on disk.
• The resulting token is a bearer token on every poll, which is exactly what agent_signing.py refuses to put through an inspecting proxy.
• Dex's RFC 8693 token exchange only moves the problem to whatever upstream token the agent presents. SPIRE is that upstream, minus the extra hop.
• X509-SVID mTLS from the agent to the dashboard. Same proxy problem as any mTLS. The SVID is used as a JWT to bind a key, never as a client certificate on the dashboard connection.
• The docker workload attestor. It needs the Docker socket, which is root on the host. A shared PID namespace plus unix:uid gets the same answer without it.
• Forced migration with a deadline. Agents behind inspecting proxies could never meet it. A deadline would turn a security improvement into an outage for exactly the customers the agent exists for.
• Running SPIRE behind Caddy. See The dashboard-hosted SPIRE server.

Phase 2: what is specified here and not built

Built since: the on-prem Dex kubeconfig. For cloud="local" clusters the API-tunnel download is a kubectl oidc-login kubeconfig against Dex (k8s_service._onprem_dex_tunnel_kubeconfig), gated on the Dex settings (dex_issuer_url, dex_k8s_client_id, dex_ca_pem) and the per-cluster Trusts Dex flag (POST /api/k8s/clusters/{id}/dex-trust). Without either it refuses with the missing step. The stored admin kubeconfig is never returned. See docs/kubernetes.md.

Also built: SPIRE attestation for agents. - POST /api/agent/attest (agent_service.attest, verification by spiffe_assertion.verify + consume), off unless Settings → Remote agents → Let agents attest through SPIRE (spire_attest_enabled) is on, and checking the SVID audience against the pinned audience only. - RemoteAgent.spiffe_id / auth_mode, bound with POST /api/agents/{id}/spiffe-id (agents:write; "" unbinds and cuts off an attested key). Re-issuing an enrolment code clears both: that is the rollback. - The agent's AGENT_SPIFFE_JWT_FILE mode (agent 2.6.0) and the spiffe-helper layout in examples/remote-agent/docker-compose.spire.yml.

Also built: the one-click Migrate to SPIRE action and the banner (services/dashboard_spire.py, POST /api/agents/{id}/migrate-spire). - It drives the dashboard's SPIRE server with docker exec <spire_server_container> spire-server … -output json, the way the SPIRE lab drives its Docker-mode server. The Docker socket and CLI were already in the app image, so there is no gRPC dependency, and the compose overlay's admin-socket mount is not needed for it. - On a managed container runtime there is no Docker socket, so the same CLI calls run another way. The app runs the spire-server binary it carries against a sidecar's admin socket (SPIRE_SERVER_SOCKET; examples/cloud-spire). The worker, a separate deployment with no volume in common, relays each call through the database to an app process (services/spire_relay). One argv and one set of result checks serve all three transports. See On a managed container runtime. - One click: a one-use join token for spiffe://<td>/node/<agent-id>, the workload entry spiffe://<td>/agent/<agent-id> selecting unix:uid:10001 under it (reused if it exists), the trust domain registered from bundle show -format spiffe, and the binding. The modal shows the token once, the .env lines and the trust bundle for spire/bootstrap.crt. - The trust domain row is owned (created_by="dashboard-spire") and re-synced daily by the existing SPIRE refresh loop, because SPIRE rotates JWT keys within ca_ttl. A row a SPIRE lab or an admin registered under the same name is never overwritten. - The banner lists active Ed25519 agents when attestation is on, says "update the image first" below agent 2.6.0, and is dismissible per agent. - Not automated: cloud node attestation (aws_iid, azure_imds, gcp_iit), whose node SPIFFE ID is derived from the instance and is unknown until it attests. Those entries are still created by hand.

Also built: central credential storage as the recommended model for attested agents. - The existing per-job release routes (/api/agent/jobs/{id}/secret, /gateway-key, /ansible-bundle) were already scoped to the job, sealed per fetch and audited. What held the docs back from recommending them was that an Ed25519 agent's request ability is a file on the host. Attestation removes that file, so the docs now recommend dashboard_secret (ideally a ps_account:// reference) for attested agents and treat host-side storage as the fallback. - Settings → Remote agents → Release dashboard-held credentials only to SPIRE-attested agents (dashboard_secrets_require_spire, off by default). When it is on, all three routes refuse an agent whose auth_mode is not spiffe with 403 and a remedy. The check runs after job ownership, so it adds no oracle. The banner says when it applies.

Still to build:

• Settings: SSO provider direct | dex.
• A Dex compose overlay for the dashboard host, for sites that want Dex beside the dashboard rather than on a cluster.