← Back to dashboard

Config Management

This document explains how the dashboard does config management — the philosophy that drives the design, the best practices the codebase encodes, and how the on-premises and cloud paths fit together.

If your dashboard runs in a cloud (ECS / ACI / Container Apps), the "Local Docker" runner below cannot serve your on-premises targets: it is a sibling container on the dashboard's host, which has neither a Docker socket nor a route to your LAN. Those targets are reached by a remote agent instead, which runs the same one-shot container inside your network. Everything else in this document — the asset types, the secret handling, the drift tracking — is unchanged.

The companion docs:

• Infrastructure as Code — how the dashboard stands the infra up in the first place
• Image Management — what's on the VMs before config-management runs against them
• Secrets Management — where credentials live
• Storage Management — where playbooks and assets live

Philosophy

Config management lives or dies on three principles. We try to bake all three into the dashboard rather than leave them as user discipline.

1. Declarative, not procedural. You describe the state you want; the runner figures out whether work is needed. A playbook that says "nginx should be installed and running" is right whether nginx is already there or not. A shell script that says "apt install nginx && systemctl start nginx" is wrong on the second run, on a yum system, on a host that's already serving traffic.

2. Version-controlled assets. Playbooks and scripts are code. They go in source control, get reviewed, get rolled back when wrong. The dashboard's storage layer is the runtime distribution channel — your git history is still authoritative. Enable versioning on the underlying bucket (S3, Azure Blob, GCS all support it) so the runtime store is also recoverable.

3. Separation of what from where. The asset describes the desired state. The inventory says which hosts to apply it to. Mixing the two ("install nginx on web-01") couples the two and turns a 20-host fleet into 20 unique playbooks.

How the dashboard implements these

The execution paths

The dashboard surfaces three distinct execution paths, all reaching the same Config Management page. The first two SSH/WinRM to a host; the third runs a localhost play that reaches out to a managed service.

On-premises hypervisors

If you've enabled any of the on-prem hypervisor integrations (Proxmox VE, vSphere/ESXi, Hyper-V, Nutanix AHV, XCP-ng, VMware Workstation), the dashboard auto-builds an Ansible inventory from each integration's configured host list. Targets appear in the run-asset dropdown as group keys (e.g. proxmox, vsphere).

Behind the scenes (see services/ansible_local_service.py):

• build_inventory() returns a JSON inventory grouped by hypervisor type, with hostvars wired for the right Ansible connection plugin (ansible_connection=ssh or winrm depending on the hypervisor).
• get_configured_targets() powers the UI dropdown — only enabled + configured hypervisors appear, no empty groups.
• The Local Docker runner (default) executes playbooks against this inventory directly from the dashboard host.

Two limits of that path are worth knowing before you rely on it, and a remote agent is the answer to both:

• It targets the hypervisor hosts, not their guests. build_inventory() emits one Ansible host per connection — the vCenter, the Proxmox node — so configuring the VMs running on them is a different thing entirely. Guests synced by an agent are individually selectable once the sync reports an address.
• Agent-bound connections are excluded from it deliberately. They hold no host and no credential on this side, so an inventory entry for one could never connect. Their guests are reached by the agent instead.

This is where contributors with on-prem labs help most — see CONTRIBUTING.md → Where the community can help most.

Cloud providers (AWS / Azure / GCP)

Cloud VMs you've deployed via the dashboard appear in the same target dropdown, prefixed with aws:, azure:, or gcp:. Picking one tells the runner three things:

• Where the target is — the bare IP or hostname (the prefix is stripped before the runner sees it).
• Whose SSH key opens the door — fetched from the matching cloud's secret store at run time. AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager. The fetch is JSON-aware: it handles either a raw PEM string or a {public_key, private_key} envelope.
• Which user to log in as — the per-cloud SSH user (ansible_aws_user, ansible_azure_user, ansible_gcp_user) drives the default; the run form lets the operator override per job. Stock-image conventions (ec2-user / azureuser / gcp-user) are pre-populated.

Cloud runs can use any of the four runners — the choice mostly affects where the Ansible process executes, not the playbook semantics. See the runner section below.

Kubernetes clusters & databases (localhost plays)

Registered or provisioned Kubernetes clusters and databases appear in the same target dropdown, under their own groups. A database imported from Password Safe is an ordinary registered row and behaves identically as a target. These are not SSH targets — Ansible's kubernetes.core and community.postgresql/mysql/general modules run on the controller (hosts: localhost, connection: local) and reach out to the API server (via a kubeconfig) or the DB endpoint (via login vars). So the model is inverted from the VM paths, and three things follow:

• Connection material is injected automatically. Pick a cluster and the dashboard token-preps its stored kubeconfig (swapping the cloud exec-auth block for a short-lived bearer token) and hands it to the runner via K8S_AUTH_KUBECONFIG/KUBECONFIG. Pick a database and it resolves the admin credential server-side — the provisioning job's for a database it built, a just-in-time Password Safe managed-account checkout for a registered one — and injects db_login_host/_port/_user/_password (+ db_name) as scrubbed extra-vars. The operator never sees or types either. No SSH user or key field is shown.
• Cloud-hosted targets always run on a remote in-cloud runner. Those endpoints are private-only, so the run executes in-cloud, in-subnet (ECS / ACI / Cloud Run — the same infra the VM cloud runners use), never the local sibling-Docker path. This is mandatory: the dashboard host can't reach an RFC1918 endpoint, and its egress traverses the corporate TLS-inspecting proxy. Running in-cloud keeps the Ansible→endpoint data path entirely within the cloud. (ansible_runner_<cloud> selects the backend; local is rejected for these kinds.) An on-premises target (cloud = local) inverts it — a cluster registered from a kubeconfig, or a registered on-prem database, sits on your LAN, which no cloud task can reach, so those runs execute in a sibling container on the dashboard host instead.
• A different runner image. k8s/DB runs use chrweav/ansible-cloud (kubernetes.core + the DB collections + client libs + the helm CLI), selected via ansible_cloud_image — never the winrm VM image, which lacks those collections.

Runs are dispatched by the durable job worker (ansible_cloud_run job type) rather than an in-process background task, since they launch a cloud task that can outlive a request worker's recycle.

Scope note: the stored kubeconfig is cluster-admin and the database credential is the admin/master login — a localhost play has full rights. Treat these playbooks accordingly. Starters live in examples/playbooks/k8s/ and examples/playbooks/database/.

Bulk runs from the inventory

The Config Management page runs one asset against one target. To apply a playbook across a fleet, use the Inventory page (/inventory): filter to what you want, tick the rows, and a run panel appears. Each selected resource becomes its own job, all tagged with a shared batch_id — so one host failing doesn't roll back the others, and each job keeps its own log and output scrubbing.

Runs are queued work, not request-side work: the endpoint writes a job row and the job runner claims it. So a batch survives a dashboard restart mid-flight, and its jobs spread across WORKER_REPLICAS (default 3) instead of executing one at a time. Worth knowing before firing a large batch — three playbooks run against three hosts at once, and raising WORKER_REPLICAS raises that concurrency.

Queueing a batch lands you on /jobs?batch_id=…, filtered to just that run, with a rollup across the whole batch — N total · N running · N failed — rather than the one page of rows the table happens to show. The URL is shareable, and any job that belongs to a batch carries a batch badge linking back to its siblings, so you can still find a run after the toast is gone.

One kind per run. Selecting a VM locks the checkboxes on Kubernetes clusters and databases, and vice versa. This isn't a UI convenience — the kinds are not interchangeable at any level. A VM run SSHes to a host; k8s and database runs are localhost plays that reach out over a kubeconfig or DB login. Different request fields, a different runner, and a playbook written for one is meaningless against another. A mixed selection could only ever produce a pile of failed jobs, so it is refused rather than attempted.

Rows that can't be a target at all are disabled, with the reason on hover:

Those reasons come from the server, computed by the same rule the endpoint enforces, so the page can never offer a checkbox the API would reject.

Two limits worth knowing. A batch is capped at 50 targets — each one is a job, so a mis-clicked "select all" against a large estate would otherwise fan out unbounded work. And while selection problems refuse the whole request before any job exists, a per-target failure at dispatch does not: several checks depend on the target's cloud, so a mixed-cloud VM batch can be valid for one host and not another. Those targets come back in the response's failed list and are named in the toast; the rest still run.

Secrets and managed accounts in a bulk run

The inventory panel covers the common case — asset, SSH user, extra vars. For a run that needs a Secrets-Management secret or a Password Safe managed account, use Continue on the Config Management page →. It carries the selection over and the full run form applies to it: named secret vars, become password, SSH key, and the managed-account picker.

A managed account is matched by name on each host. A ManagedAccountRef normally pins system_id + account_id, and both belong to one managed system — reusing one across a fleet would check out a single machine's credential and connect to every host with it. So a bulk run sends the account name instead, and each job resolves it against the host it is actually configuring, then checks out that host's own credential. The account list you pick from is read from one target as a sample; a host that doesn't have an account by that name fails only its own job, with a message naming the host and the account.

This works for domain accounts too — the Password Safe lookup already falls back to domain-linked accounts — and it matches the {user};{suffix} form that cloud-native onboarding registers (the AWS Systems Manager plugin appends a scope suffix), so picking svc-ansible matches svc-ansible;local.

Connection credentials are refused for Kubernetes and database batches. Those run a localhost play that reaches out over a kubeconfig or DB login — there is no SSH connection to authenticate, and the run path silently ignores managed_account, managed_become, secret_ssh_key_source and secret_become_source. A single run can absorb that quietly; a batch would leave you believing a credential had been applied to fifty clusters, so /run-bulk rejects the combination with a 400. Named secret_vars are honored on those targets and stay available.

Asset types

The auto-wrap path is a convenience for one-off operations, not a substitute for proper playbook authoring. If you find yourself writing the same .sh script three times with different targets, that's a signal to write a real .yml playbook with vars and when clauses.

Need a starting point? Ready-to-adapt Linux, Windows, Kubernetes, and database playbooks live in examples/playbooks/.

Runners

Where the Ansible process actually runs. Picked in Settings → Ansible → Runner.

The cloud runners exist because connecting from a dashboard sitting on a corporate LAN to a deeply-private cloud subnet is often impossible without a VPN. Running the playbook inside the cloud avoids that network problem at the cost of one Fargate task / ACI / Cloud Run invocation per run.

The runner choice constrains the storage choice. The runner has to fetch the asset before executing it, which means the runner needs network reachability to the storage backend. Two combinations don't work:

Most teams pair Local Docker + Local Filesystem (or a cloud bucket) for on-prem labs, and one of the cloud runners + a cloud bucket for cloud fleets.

Why one-shot runners (the security argument)

Every runner — Local Docker, ECS Fargate, ACI, Cloud Run Jobs — is ephemeral by design. A new container is spawned per run; it executes the playbook; it exits and is destroyed. Nothing persists between runs.

This is deliberate, and it matters. Long-lived runners are a known weakness in CI/CD and config-management estates: they accumulate secrets in environment variables, cached SSH keys in ~/.ssh, remembered hosts in known_hosts, leftover state from previous runs. A single compromise of a long-lived runner can yield credentials that have been touched by every job that ever ran on it. The community edition's design refuses to be that target:

• Secrets are fetched just-in-time. The cloud secret store is consulted at the start of each run; the SSH private key lives in the side-car container's tmpfs for the duration of the playbook; the container is destroyed when the playbook finishes. There's no place for the key to leak to.
• No process or filesystem outlives the run. The Ansible binary, the playbook bytes, the asset cache, the inventory JSON — all of it goes away with the container. The next run starts from zero.
• No shared user namespace between runs. Two operators running back-to-back jobs against the same target cannot accidentally read each other's variables, output, or temp files. Each run is its own Linux process tree.
• The dashboard process itself never holds the secret long. The playbook bytes pass through the dashboard's memory once on the way to the runner; nothing is cached on disk on the dashboard host.

The compliance angle: regulations like SOC 2 CC6.1, NIST SP 800-53 AC-6 (least privilege) and SC-39 (process isolation), and CIS Controls 4.1 / 4.7 all point at "minimise persistent privileged surface". An ephemeral runner satisfies them by construction — there's nothing persistent to harden, audit, or rotate. Auditors generally accept "the runner has a 90-second lifespan and zero state at rest" with less friction than "here's our hardening baseline for the long-running worker fleet."

Compared to common alternatives:

This isn't unique to the dashboard at the technical level — the underlying primitives (Fargate / ACI / Cloud Run Jobs / docker run --rm) have been around for years. What's notable is the design decision to only offer ephemeral runners. There's no escape hatch in the codebase for "give me a long-lived worker for performance reasons." You pay a one-second startup penalty per run; you never have to defend a fleet of long-lived runners to a security review.

The one long-lived process, and why it doesn't contradict this

Remote agents run a persistent container inside a customer's private network. That looks like the exact thing this section argues against, so it is worth being precise about why it isn't.

Read the sentence above again: no escape hatch for a long-lived worker for performance reasons. An agent is not asking for persistence to save a second of startup. It asks for it for reachability — you cannot launch a one-shot container inside a network you cannot reach, so something has to already be there. That is a different justification, and the four invariants this section actually names all survive it:

So the runner is still one-shot. What moved is the thing that launches it — from the dashboard's Docker socket to one inside the network the target actually lives on. The supervisor that does the launching is small, holds no credentials, and executes nothing the dashboard sends it: the job payload is a closed allowlist of scalars and network addresses, and its handler table is a closed dict rather than a dispatch on a string from the wire.

Secret scanning (advisory)

Uploaded assets are scanned for hard-coded secrets and you're warned — it's advisory: the upload always succeeds, the finding is a heads-up. The point is to catch an AWS key or a plaintext password before it's stored in the asset backend and shipped to a target.

• When — on upload, at both /api/storage/upload and /api/config-mgmt/upload.
• What it catches — AWS access keys, private-key blocks, GitHub / Google / Slack tokens, and generic password / secret / token / api_key assignments. Matched values are redacted in the finding.
• What it ignores (so it doesn't cry wolf) — templated values ({{ … }}, $VAR, ${VAR}), placeholders (changeme, <your-password>), Ansible-Vault-encrypted files, and binary assets (.rpm / .deb).
• Config — secret_scan_enabled (default on; it only warns). Set false to disable.

The right fix when it fires: move the value into a vault reference (see Secrets Management) or Ansible Vault, and reference it from the playbook rather than hard-coding it.

Config-drift visibility

The Ansible stream remembers each successful apply, so you can tell when a target has drifted out of "known-good." It's passive — it records a fingerprint on a successful run; it never touches a target to check (no --check reconciler).

• What's recorded — on each successful run, a per-(target, playbook) row with a content fingerprint of the applied asset and the timestamp (config_drift_tracking_enabled, default on).
• Two signals —
• Unverified — the last apply is older than config_drift_stale_days (default 14): "host X unverified since 2026-06-20."
• Changed — the playbook's current content in storage no longer matches what was applied: the target is running an older version than what's on disk now.
• Where it shows — GET /api/config-mgmt/drift returns the per-target signals; the dashboard's Needs attention panel rolls up "N config targets need attention." Re-applying the playbook clears the signal.

Fingerprints are one-way hashes — the inputs/secret values themselves are never stored.

Best practices

Stage your changes. Build a target group with one or two test hosts in it before applying to the whole fleet. The inventory is just JSON — add a test group in your hypervisor hostvars module to make this trivial.

Keep secrets out of playbooks. Anything sensitive belongs in Secrets Management, not embedded in YAML or shell scripts. Reference secrets via Ansible's lookup('env', ...), the ansible-vault integration, or fetched-at-runtime variables that the runner reads from the cloud secret store.

Use a secret without seeing it. The run form's Use a secret panel injects a Secrets-Management secret (as a named var, become password, or SSH key) — or a BeyondTrust Password Safe managed account checked out just-in-time — straight into the run. The operator never sees the value; it's scrubbed from job output and the use is audited. Requires the secrets:use permission. See Using a Secrets-Management secret in a run.

Tag your runs. The Extra Vars field on the run form accepts JSON — include a run_id or a deployment ticket number so when something breaks at midnight you can grep the logs back to the playbook invocation.

Read the job logs. Every run lands in the dashboard's job tracker (/jobs). Cloud-runner runs include a CloudWatch / Azure Monitor / Cloud Logging log link. Local Docker runs include the full Ansible output inline. Drift, rollback decisions, and post-mortems live there.

Don't modify assets in place. Re-upload as a new file with a date or version stamp (hardening-base-2026-04-12.yml). The storage backend preserves both. When you're sure the new one works, delete the old via /storage.

Idempotency tests. Run the same playbook twice in a row. The second run should report 0 changed (or as close to it as your real workload allows). If the second run still does work, your "declarative" playbook is hiding procedural logic and will surprise you on partial failures.

Where this is heading on SaaS

A few things the community edition does not try to do. They're SaaS priorities — see docs/saas-comparison.md for the hosted-edition philosophy.

• AI-assisted playbook generation. "Install fail2ban with custom jail rules for SSH on these eight Debian hosts" → generated YAML the operator reviews, edits, and runs. The community edition's auto-wrap is a baseline; the SaaS edition's helper service is the up-leveled version, with the dashboard's asset schema and the active inventory in context so the generated playbook is opinionated about your environment, not generic.
• Tenant-scoped asset libraries. Multi-tenancy means each organisation gets its own storage namespace, its own inventory, and its own credentials — without per-instance dashboard deployment. The community edition keeps everything single-tenant by design.
• Drift-aware runs. SaaS keeps a per-target hash of the last successfully-applied asset and surfaces "the live state of host X hasn't been verified since 2026-04-12" without manual checks. Community runs leave this kind of telemetry on the table; the job log is the audit trail.

The ephemeral-runner property in Why one-shot runners above carries forward to SaaS — every run still gets its own single-purpose container and is destroyed after execution. The hosted edition adds tenant-scoped network isolation on top, so a run in tenant A can't reach tenant B's targets even by mistake.

You can be productive on community indefinitely with the practices in this doc. Move to SaaS when the AI assistance, tenant separation, or managed audit trail are worth more than self-hosting flexibility.

Troubleshooting

Run fails with "no active storage backend." Set up storage on /storage first; the Ansible feature flag depends on it. See storage-management.md.

Local Docker runner fails with "permission denied" mounting /var/run/docker.sock. The dashboard container needs Docker-out-of-Docker access to spawn the side-car runner. Check docker-compose.yml includes /var/run/docker.sock:/var/run/docker.sock:ro and that the container user can read it.

Cloud runner says "AccessDenied" fetching the asset from S3. The IAM role / service principal / GCP service account the dashboard configured for VM deploys also needs read access to the storage bucket. Add the s3:GetObject / Storage Blob Data Reader / storage.objects.get permission for the storage bucket's resource path.

Hyper-V / Windows targets connection refused on PowerShell runs. The target's hostvars need ansible_connection=winrm plus the WinRM auth fields (ansible_winrm_transport, credentials). Verify by running docker compose exec app cat /tmp/inventory.json after a failed run and checking the target's hostvars block.

Run log shows the right SSH user but Permission denied (publickey). The cloud secret store probably has a stale key, or the VM's authorized_keys doesn't include it. Confirm the secret's public_key matches what's actually on the VM via cat ~/.ssh/authorized_keys.