← Back to dashboard

SPIFFE and SPIRE

Audience: operator · Profile: both · Read this when: you are running the dashboard's own SPIRE server (on Docker Compose or a managed container runtime), deciding which SPIFFE feature you need, or a SPIFFE feature fails and you need to know whether the server is the cause.

SPIFFE is a standard for naming a workload (spiffe://<trust domain>/<path>) and handing it short-lived, verifiable credentials: SVIDs, as an X.509 certificate or a JWT. SPIRE is the reference implementation. A SPIRE server holds a trust domain's keys. A SPIRE agent proves which machine it runs on (node attestation) and hands SVIDs to the processes on it (workload attestation).

The dashboard uses SPIFFE in two quite different ways:

• It runs a SPIRE server of its own, for its trust domain. Remote agents attest to it, the dashboard mints its own cloud tokens from it, and service accounts and demo cells can be issued identities in it. One per install, and this page is its home.
• It builds and governs other people's SPIRE servers. The Workload Lab stands up SPIRE trust domains for demos, and the Password Safe SPIFFE SVID plugin governs them. Those servers are covered on their own page.

Where SPIFFE shows up

The designs behind these are SPIRE for agents, Dex for people, The dashboard as a SPIFFE workload and A workload reaching Kubernetes with a short-lived token.

Every one of these is off by default, and nothing changes for an install that does not turn one on. Ed25519 agent enrolment and stored cloud keys stay supported.

The dashboard's trust domain

You pick the name once, as SPIRE_TRUST_DOMAIN (for example dashboard.example.com). Renaming it later creates a new trust domain: every agent would have to be migrated again and every cloud trust policy rewritten.

The dashboard owns these paths in it, and refuses to bind any other client to the reserved ones:

The dashboard registers its own trust domain under Users → SPIFFE trust domains (created by dashboard-spire). It re-reads the bundle once a day, which is ahead of SPIRE's own key rotation. It also re-reads it at once when an assertion names a key it does not hold yet, at most once a minute. It never overwrites a registration of the same name made by hand or by a SPIRE lab, and says so.

How the dashboard drives it

Always with the server's own CLI and its -output json: no gRPC client, no new dependency. How that CLI reaches the server is chosen per process, because the app and the worker can differ:

SPIRE_SERVER_TRANSPORT=docker|socket|relay overrides the choice. The same argument list and the same result checks serve all three, so a feature cannot tell which one answered.

The relay. A process with neither socket writes each call to the spire_admin_calls table. An app process with the socket claims the row (the same UPDATE … WHERE status='pending' claim the job queue uses), runs it and writes the result back. The requester then reads the result and deletes the row. The database is the one thing every deployment already shares on every platform, so this needs no port, no route between the two services, and no credential of its own:

• Encrypted. A join token or JWT-SVID in transit is encrypted with the same key as app_config.
• Allow-listed. The serving side runs only the subcommands the dashboard uses.
• Fast, and fails fast. A call takes about half a second. If no app process with a socket has checked in within the last 20 seconds, the call fails at once with a message that says so, rather than after a timeout.

Running it

On Docker Compose

Layer the overlay over the base compose file. SPIRE_JWT_ISSUER is needed only for the dashboard's own identity:

SPIRE_TRUST_DOMAIN=dashboard.example.com SPIRE_JWT_ISSUER=https://agents.example.com/spiffe docker compose -f docker-compose.yml -f docker-compose.spire.yml up -d

$env:SPIRE_TRUST_DOMAIN = 'dashboard.example.com'
$env:SPIRE_JWT_ISSUER   = 'https://agents.example.com/spiffe'
docker compose -f docker-compose.yml -f docker-compose.spire.yml up -d

docker-compose.spire.yml runs the server as vmdash-spire-server, keeps its datastore and CA keys on the spire_data volume, and publishes 8081 and 8082. Back that volume up: losing it re-keys the trust domain.

On a managed container runtime

Azure Container Apps, GCP Cloud Run, AWS ECS/Fargate and a Kubernetes pod all run the same shape. docker-compose.spire.yml cannot be lifted across as it is, for three reasons that have nothing to do with SPIRE:

• There is no Docker socket, so the dashboard cannot docker exec into a server.
• No volume crosses two deployments. The app and the worker are separate services (Cloud Hosting → The shape), and a pod's volume is that pod's only.
• Nothing on disk survives a restart. A server keeping its CA keys and datastore on container disk re-keys the trust domain on every revision, and every attested agent is stranded with nothing in the logs saying why.

So on these platforms the server is a sidecar of the app, built from examples/cloud-spire, and each of those three gets an answer:

 app: one pod / task / replica                   worker: its own deployment
 ┌─────────────────────────────────────────┐     ┌─────────────────────────────┐
 │ gateway ──▶ app ──┐                     │     │ worker                      │
 │                   │ spire-server CLI    │     │ no socket, no Docker        │
 │                   ▼                     │     └──────────────┬──────────────┘
 │      /run/spire-server   (pod volume)   │                    │ each call
 │                   ▲                     │                    ▼
 │ spire ────────────┘ admin socket        │ ◀─ claims, runs, ─ spire_admin_calls
 │   :8081 agents   :8082 federation       │    answers         (dashboard's DB)
 └────────────────────┬────────────────────┘
                      ▼
        PostgreSQL `spire` database: the server's datastore

The worker relays through the database, as described under How the dashboard drives it. There is nothing to configure on it.

The upstream CA

Generate it once, offline, and keep it the way you keep JWT_SECRET_KEY. The private key is the trust domain. Lose it, and every agent has to be migrated again. Leak it, and whoever has it can mint identities in your trust domain.

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -days 3650 -subj "/O=vm-dashboard/CN=dashboard SPIRE upstream CA" -addext "basicConstraints=critical,CA:TRUE" -addext "keyUsage=critical,keyCertSign,cRLSign" -keyout spire-ca.key -out spire-ca.crt

Store both files as platform secrets (SPIRE_UPSTREAM_CA_CERT, SPIRE_UPSTREAM_CA_KEY). The sidecar refuses to start without them rather than fall back to keys that die with the container. The bundle that Migrate to SPIRE hands an agent host as bootstrap.crt is this certificate, so it stays valid across every restart and every replica.

SPIRE logs UpstreamAuthority plugin does not support JWT-SVIDs at start. That is expected: JWT signing keys stay the server's own, and a restart starts signing with a new one at once. Two things keep that from breaking anything:

• The previous keys stay in the bundle in PostgreSQL until they expire.
• An agent token signed with a key the dashboard has not stored yet makes it re-read its own bundle on the spot, at most once a minute.

The dashboard's /spiffe/keys endpoint always reads the live bundle.

The datastore

Create a database for the server on the PostgreSQL the dashboard already uses. Keep it out of the dashboard's own database: SPIRE manages its own schema.

CREATE DATABASE spire;

SPIRE_DATASTORE_URL is a secret: postgresql://<user>:<password>@<host>:5432/spire?sslmode=require. Several replicas may share one datastore and one upstream CA. That is SPIRE's own HA shape, and each replica's JWT key is published in the shared bundle.

The sidecar holds at most 5 connections to it (max_open_conns), per replica. Count them against the server's limit next to the dashboard's own pools: a small tier such as Azure's B1ms allows 50 in all.

The sidecar

docker build -t <registry>/dash-spire:1 examples/cloud-spire
docker push <registry>/dash-spire:1

It mounts a pod-local volume at /run/spire-server, and so does the app container. Probe it with httpGet on 8088: /live for liveness, /ready for readiness. Never publish 8088. The sidecar starts as root only to take ownership of that volume and write the CA files 0600. It then runs as SPIRE_RUN_UID.

The app container

Two additions, both additive:

• SPIRE_SERVER_SOCKET=/run/spire-server/api.sock, which switches the app from docker exec to the sidecar's socket;
• the same volume mounted at /run/spire-server.

The app image must be from this release or later, because that is the first one to carry the spire-server CLI at /usr/local/bin/spire-server. The worker needs no change at all: it has neither socket, so it relays, and it keeps its own token files.

Publishing 8081 to agents

The agent's connection is mTLS gRPC and must reach the server untouched: TCP passthrough, never an HTTP load balancer or a TLS-inspecting proxy. Port 8082, for federation with a Workload Lab, has the same requirement. Restrict both to the networks that need them.

The Migrate to SPIRE modal hands the agent host whatever that setting resolves to, as SPIRE_SERVER_ADDRESS and SPIRE_SERVER_PORT.

Protecting the keys

Whoever holds the trust domain's signing keys can mint every identity in it: agents, the dashboard's own cloud tokens, service-account SVIDs. Where those keys live depends on how the server runs:

On a host with a cloud identity of its own, the server's keys can live in that cloud's KMS. examples/spire-server/server.conf carries a commented block for each:

Comment out the disk line, uncomment one block, fill it in, and restart the spire-server container. Check the field names against SPIRE 1.15's server_keymanager_* plugin docs first: the examples are starting points.

Switching creates a new CA. The old keys stay in keys.json, and the server will not read them through a KMS plugin. Every agent attested to the old CA has to attest again, and the published JWT keys change. Each cloud re-fetches /spiffe/keys on its own schedule, so expect federated calls to fail for a while after the switch. Switch before you migrate agents, or plan the re-attestation. A self-hosted Docker host with no cloud identity has nothing to authenticate to a KMS with, so it stays on disk.

The app administers this server, so anyone who compromises the app can mint tokens for any identity in it while they are present. That is still better than a stored key, which can be copied and used offline for ever. It is not "the app holds nothing".

Settings and environment

The switches that use the server are on the feature pages: - Let agents attest through SPIRE (spire_attest_enabled); - Give the dashboard its own SPIFFE identity (dashboard_spiffe_identity_enabled). - On a POV instance, AWS for the POV cloud provider with no stored key, which uses that identity.

These two are about the server itself, both on Settings → Integrations → Remote Agents:

Environment, on the dashboard's containers (all optional):

On the compose overlay: SPIRE_TRUST_DOMAIN (required), SPIRE_BIND_PORT (8081), SPIRE_FEDERATION_PORT (8082) and SPIRE_JWT_ISSUER. On the sidecar: see The sidecar.

Troubleshooting

Every SPIFFE feature reports a server problem in the same words, so start here when one says the server "is not running".