← Back to dashboard

Storage Management

This document explains how the dashboard stores playbooks, scripts, and other large assets that don't fit in the encrypted credentials database — and how to choose, configure, and migrate between cloud object stores.

The companion to Secrets Management: secrets are small, sensitive, and live in a per-key encrypted store. Storage holds bigger, mostly-non-sensitive payloads (playbooks, shell scripts, package files, image artefacts) that need to be readable by Ansible runners across hosts and by cloud VM-import APIs. For the philosophy and best-practice side of running playbooks against your fleet, see Config Management. For the IaC layer that stood the targets up in the first place, see Infrastructure as Code. For the image build → promote lifecycle that produces the binaries the IaC layer deploys, see Image Management.

Philosophy

The dashboard talks to one active object-storage backend at a time. You pick from three providers, configure as many as you like, and choose one to be active. Backends not currently active stay reachable through the migration UI so you can copy assets between them without downtime.

All four are interchangeable from the dashboard's perspective. Switching backends does not move data; the Migrate panel does that explicitly, and only deletes from the source if you ask it to (today: never — see "Migration semantics" below).

The active backend holds two kinds of data: user assets (playbooks, packages, Packer manifests, the image-registry hub) and Terraform remote state for cloud deploys (see What counts as an asset). That second kind is why switching backends is guarded — if live deployments have state on the current backend, the dashboard won't let you flip the active backend out from under it without migrating that state too (see "Migration semantics").

Why storage is its own page

Storage was originally configured inside the Ansible feature panel. It lives on its own page because:

• Future features may use it. Image manifests, log archives, and capture artifacts all fit the same backend abstraction. Pinning the config to "Ansible" would force every new feature to either re-implement storage or pretend to be Ansible.
• It's a deployment-level concern. Picking S3 vs. GCS is an organisation-policy decision, not a per-feature one. Surfacing it alongside /secrets (which is the same kind of decision for credentials) keeps the mental model clean.
• The Ansible feature flag depends on it. The Settings → Integrations toggle for Ansible is greyed out until storage is configured and active, with a link to /storage in the tooltip.

What counts as an asset

Files outside this set are rejected at upload. Assets are stored under a configurable key prefix (default config-mgmt/), so multiple deployments can share a bucket if the prefix differs.

The table above governs the ASSET UPLOAD path only — files a user uploads for the Ansible runner to execute. It is not a restriction on what the dashboard itself can store. In particular, Cloud Functions are unaffected by it: their Python handlers are not uploaded at all. They ship inside the dashboard image, the packager builds the deployable zip in memory, and it goes to a dedicated per-cloud bucket (function_package_s3_bucket / function_package_gcs_bucket / a container on storage_azure_account) rather than to the active storage backend. See Cloud Functions → package stores.

Beyond user-uploaded assets, the active backend also holds Terraform remote state. Cloud VM, cloud-database, and Kubernetes-cluster deploys write their state to the same active backend (keyed per job under terraform-state/, with backend-native locking), so the backend you pick here is load-bearing for infrastructure teardown, not just playbooks. The BeyondTrust PRA tunnel state is the one exception — it's scrubbed of credentials and kept in the database instead. See Infrastructure as Code → State.

Uploading

Two equivalent paths to put an asset in storage:

1. /storage page — drag-and-drop or pick a file in the Upload card, click Upload. Goes straight to the active backend. Available to any logged-in user.
2. /config-mgmt page — same upload form, plus inline run controls for executing the asset against your hypervisor inventory or cloud instances. Available to any logged-in user.

Either path uses the same POST /api/storage/upload endpoint behind the scenes. The /config-mgmt-side POST /api/config-mgmt/upload endpoint also still works (delegates to the same service).

Configuring storage

Open /storage (admin only). The page has four sections:

1. Backend — pick the active backend with a radio button. Each backend's configuration card shows below; fill in the fields for the one(s) you want to use. A backend appears as configured when its primary identifier (bucket / storage-account / bucket name) is set, regardless of whether it's the active one.
2. Image-registry hub — pick the backend that holds the canonical VHD/raw artefact for every registered image. Leave on "Same as active backend" for single-backend installs; only change it if you want the image hub on a different cloud than your day-to-day asset uploads. See Image-registry hub below.
3. Stored assets — once a backend is active, the list shows what's in it. Use Config Management's upload form (/config-mgmt) to add playbooks; this page is read-mostly except for delete.
4. Migrate — covered below.

Required cloud credentials

Each backend reuses cloud credentials configured elsewhere in the dashboard:

If the cloud you want to use for storage isn't already configured for VM deploys, set those creds first in /setup or the matching Settings panel.

S3

Azure Blob Storage

Google Cloud Storage

Local Filesystem / UNC

Constraint: local backend only works with the local Ansible runner

The Local backend is only selectable when Settings → Ansible → Runner is set to Local Docker (default). Cloud Ansible runners (AWS ECS, Azure ACI, GCP Cloud Run) live in cloud-only VPCs/VNets and have no network path back to a corporate file server. If you tried to use a UNC path from a Fargate task, the SMB connection would fail at TCP 445 and the run would error before the playbook ran.

The dashboard enforces this in two places:

• Frontend: the radio button for the Local backend is disabled with an inline note when the runner isn't local.
• Backend: PATCH /api/storage/config returns a 400 if you try to set storage_active_backend=local while ansible_runner != local.

Concrete fit: if your contributors test on-prem hypervisor targets (Proxmox VE, vSphere/ESXi, Nutanix AHV, XCP-ng, Hyper-V) with playbooks hosted on a corporate share, the Local backend is the right choice.

After filling in fields, click Test connection to probe the backend — it lists the bucket/container as a quick reachability check. Save with Save configuration; activation flips the moment the save succeeds.

Image-registry hub

The hub backend is the single storage backend that holds the canonical VHD/raw artefact for every registered image, regardless of which cloud built it. It's the source the cross-cloud promote flow reads from when it kicks off a per-target runner — see Image Management for the full lifecycle and runners/promote/README.md for the runner internals.

Configuration. On /storage, the Image-registry hub picker has four options:

• Same as active backend (default) — storage_hub_backend is unset; storage_service.hub_backend() resolves to whatever active_backend() returns. Single-backend installs need nothing more.
• S3 / Azure Blob / GCS — pin the hub to that backend explicitly. Useful when your day-to-day asset uploads live in one cloud but you want the image hub in another (e.g. day-to-day in S3, image hub in GCS for cost reasons).

Local / SMB backends can't be the hub. The promote runners need an HTTPS-reachable URL for the source artefact, which the local filesystem doesn't offer. The page's picker doesn't list local; posting it via the API returns a 400 with a pointer to this section.

What the hub does. After every successful Packer build the dashboard exports a portable VHD via the cloud's native export API into same-cloud storage. If that same-cloud storage is the hub (e.g. build cloud = AWS, hub = S3), no extra hop. If it isn't (build cloud = AWS, hub = Azure Blob), the dashboard runs storage_service.copy(build_backend, build_key, hub, hub_key) to move the VHD into the hub and deletes the build-side staging copy. Into an Azure Blob hub this copy is fully server-side (the staging object is presigned and Azure pulls each block itself — no bytes through the dashboard container); into an S3/GCS/OCI hub from a different build cloud it stages through the container's ephemeral disk, which multi-GB VHDs can overflow — pick an Azure Blob hub or the build cloud's own backend if you build cross-cloud.

The same export-and-land-on-hub path runs when an operator clicks Export VHD on a cloud-native image in the per-cloud Images tab (AWS Private AMIs / Azure Managed Images / GCP Custom Images). This is the recovery path for builds whose auto-export was skipped (e.g. the storage prerequisite was missing at build time). See Image Management → Manual export.

What it doesn't do. The hub is not where the promote runner uploads to. Each target cloud has its own staging container the runner writes into (promote_runner_aws_staging_bucket, promote_runner_azure_staging_container, promote_runner_gcp_staging_bucket) so the cloud's import API reads from local storage. The hub stays the read-only source-of-truth.

Promote-runner config. The promote_runner_* keys (image override, ECS/ACI/Cloud Run/Container Instances plumbing, target-side staging, IAM role ARNs) are edited in Settings → Remote Worker → Image-promote runner, one sub-card per target cloud. PATCH /api/storage/config also accepts them — the promote runner shares the hub backend's lifecycle, so the storage API round-trips them for scripted setups. The runner README has the full table.

Migration semantics

The Migrate panel copies every asset from a source backend to a target backend. Operating principles:

• Source is preserved. Today the dashboard never deletes from source during migration. Verify the target is healthy, then delete from the source manually (use each backend's native console). This is intentional — first cutovers are when things go wrong, and rolling back is much cheaper if the data is still where it started.
• Skip-by-default for collisions. Files already present in the target are skipped. Tick Overwrite existing to replace them.
• Active backend doesn't change automatically. After the migration completes successfully, switch the active backend in the Backend section above and click Save configuration — that's a separate, explicit step. The dashboard reads from whichever backend is active at the moment a request lands; there's no warmup or cache.
• Per-asset errors don't abort the run. A file that fails to copy appears in the Failed list with its error; the rest of the migration continues. You can re-run with the same source/target to retry just the failed ones (already-copied files are skipped automatically).
• Terraform state migrates separately, and switching is guarded. The asset Migrate panel copies assets. Terraform remote state (under the terraform-state/ prefix) is handled on the active-backend switch itself: if live deployments still have state on the current backend, the dashboard blocks the switch rather than stranding it, and asks you to migrate the state. Confirming copies every terraform-state/* object to the new backend (storage_service.migrate_terraform_state), then flips the active backend. So the safe cutover is: migrate assets → switch the active backend and confirm the state migration → verify → delete from the old backend by hand. Losing that state orphans the resources it tracks (see Infrastructure as Code → State).

The migrate result block on the page summarises three lists: Copied, Skipped, Failed. Save the page or screenshot before navigating away if you need a record.

API reference

Storage credentials live in the encrypted DB exactly the same way as other config values; nothing on this page reads or writes .jwt_secret_key.

Cost

Idle cost is roughly the cost of the underlying object store, which is near-zero for the playbook/script asset profile (KB-MB files, low PUT/GET volume).

Network egress during runs (the runner downloads the playbook bytes once per job) is the same per-byte rate as any cross-AZ traffic in the respective cloud.

Backup and lifecycle

The dashboard does not manage backup or lifecycle for storage backends. Use each provider's native primitives:

• S3 — versioning, lifecycle policies (e.g. Glacier transition).
• Azure Blob — soft delete, snapshot, lifecycle management.
• GCS — object versioning, lifecycle rules.

Recommended baseline for any production-style deployment: enable versioning so an accidental overwrite or migration can be reverted. This matters doubly for the terraform-state/ prefix — versioning there is your recovery path if a state object is corrupted or deleted, since losing it orphans the resources that state tracks.

Troubleshooting

"No active storage backend" error on the Config Management page. You have a backend configured but didn't activate it. Open /storage, pick the radio button for the backend you intended, and Save.

Test connection fails with a 403 / AccessDenied. The cloud credentials configured in Setup → AWS/Azure/GCP don't include read+write access to the bucket. For S3 this typically means s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject on the bucket and arn:aws:s3:::bucket/prefix/*. For Azure Blob, the service principal needs Storage Blob Data Contributor on the storage account or the specific container.

Migrate finishes, but the new uploads still go to the old backend. Switching the active backend is a separate step. After migration, change the radio button in Backend and click Save configuration.

An asset shows up in GET /api/storage/list/{backend} but not in the active list view. Different backends maintain different prefixes. The active backend's prefix may be excluding it. Either reuse the same prefix everywhere, or include the prefix when uploading.

The Ansible feature flag toggle is greyed out. Working as intended — the dashboard requires storage to be configured and active before Ansible can be enabled. Open /storage, pick a backend, save, then come back to Settings → Integrations.