BeyondTrust/POV

← Back to dashboard

Skytap

Audience: operator · Profile: pov · Read this when: you are pointing a POV instance at Skytap and want to know what it will and will not do with the account.

The first lab platform for POV environments. A POV is a Skytap template instantiated whole; the dashboard creates those environments and wires their VMs into that POV's PRA, Password Safe and Entitle tenants.

Available on a POV instance only — see POV profile. On an estate instance the integration is masked off and Settings refuses to enable it.

What the dashboard does with an environment. Create it from a template, power it on and off, and destroy it. Enrol an agent inside it — see The control node VM and the template contract. Install a BeyondTrust Gateway on that control node, registered into the POV's own PRA tenant (the POV Gateway), and a Password Safe Resource Broker on a Windows VM beside it (the Resource Broker). Then, per VM: one PRA jump item through that Gateway, Password Safe onboarding through that Resource Broker, and Entitle registration for the Linux guests (wiring the VMs). Publish a customer-facing share link for the whole environment (the share link) — Skytap calls it a publish set, and the dashboard always gives it a generated password and an expiry. And reap the environment on an auto-delete timer (the auto-delete timer), which warns on a ladder rather than once because an evaluation outlives whoever set it up.


Prerequisites

Account A Skytap account whose user can see the templates you want to build POVs from
Credential An API security token from the Skytap account page — not your account password
Network Outbound HTTPS from the dashboard to cloud.skytap.com
Network, for template builds only Outbound TCP to an arbitrary high port at a Skytap NAT address. Only the automatic runner install needs this — see building a template. An egress rule that allows 443 to the API host and nothing else will block it, and the builder says so rather than reporting a generic connection failure
Network from the guest A route to a container runtime, unless the base template already carries one: either outbound HTTPS from the control node VM to download.docker.com, or the guest's own distro repository carrying podman + podman-docker, which is the fallback and is usually already reachable. This is the guest's route out, not the dashboard's — the only prerequisite on this list that nothing on this side can test for you. A build that cannot reach either still bakes, with the reason in the Runner column, and the POV's first Install control node press tries again

Getting the token

In Skytap, open your Account page. If an API security token is listed there, that is what you use; Skytap authenticates with HTTP Basic as username:api_token.

This is the single most common setup mistake, so it is worth stating plainly: the password field is not your password. A wrong value produces a 401, and the dashboard's error message says so rather than reporting a generic failure.

Configuring it

Settings → Integrations → Skytap.

Field Key Notes
API URL skytap_base_url https://cloud.skytap.com unless your account is on another region's endpoint
Username skytap_username The account's login, usually an email address
API security token skytap_api_token From the account page. Stored encrypted; the panel shows "stored — leave blank to keep" once set
Project ID skytap_project_id Optional. Templates and environments are listed from this project, and new environments are created inside it. Blank lists everything the token can see

The token is encrypted at rest with the same Fernet key as every other secret, and — like any secret in this dashboard — it can instead be a reference into an external vault.

Verifying the connection

The Skytap panel has a Test connection button. It reads the saved values, not what is currently in the form, so save first.

It issues one page of GET /v2/templates — the same read the POV page makes — scoped to the Project ID when one is set. That single call answers three questions at once: whether the token authenticates, what this account can actually see, and whether the Project ID is real.

It distinguishes five outcomes:

It says Meaning
Connected. Templates are visible Everything the POV page needs
not configured One of the three connection fields is empty
Skytap rejected the credentials Wrong username or token. Note it is an API security token, not your account password
exposes no templates The token works, but the account — or the project — shows nothing. There is nothing to build a POV from, so this is reported as a failure, not a warning
no project N (404) The credentials work; the Project ID is stale or wrong
the host could not be reached DNS, firewall or proxy — not a credential problem

An account that authenticates but exposes no templates is deliberately red. A POV cannot be created from an empty catalogue, so reporting green for a connection that cannot do the one thing it is for would be the exact false positive a check like this exists to prevent.

What the dashboard does with the API

Three of Skytap's behaviours are easy to get wrong once and then wrong everywhere, so all three are handled in one place (services/skytap_client.py) rather than at each call site.

423 Locked is normal. It is not an error — it is Skytap saying "this resource is busy, or the account is being rate-limited", and it carries a Retry-After. A plain 429 is retried identically. Environments also expose a rate_limited boolean, which the POV page surfaces as a badge. The client retries, honouring Retry-After, bounded; only after that does it report a failure, and the message says it is a rate limit rather than a fault.

The retry budget is per client. Provisioning gets the generous default, because a job that has already waited minutes should keep waiting; Test connection asks for one retry, so a rate-limited account answers the button instead of hanging behind a spinner.

Every read carries keep_idle=true. Without it, reading an environment resets its idle timer. A dashboard that polls environments would hold every one of them awake and quietly defeat suspend_on_idle — the single biggest lever on Skytap spend. The only symptom would be the invoice, which is exactly why it is not left to the caller.

The dashboard's view of an environment is refreshed on a timer. A POV's runstate used to be written only when this dashboard changed it — at provision, or at an explicit Start or Suspend. Nothing asked Skytap again. So suspend_on_idle, the biggest lever on spend, would suspend an environment and the POV page would go on saying running indefinitely; worse, it gates the Start button on that value, so the environment you most needed to wake was the one whose Start button was hidden. A reconcile sweep (services/pov_reconcile) now reads every managed environment back every ten minutes — one paginated collection read — and the POV page shows when each row was last confirmed rather than implying it is live. Each pass is a pov_env_reconcile job; completed ones are hidden on Job History unless you tick Show routine sweeps, and are pruned on the same retention as the other sweeps.

Collections paginate by count/offset. A single GET returns a first page that looks exactly like a complete answer, so listings are walked to the end.

Listings are project-scoped when a Project ID is set — they read /v2/projects/{id}/templates and /v2/projects/{id}/configurations instead of the account-wide collections. A project id the token cannot see answers 404, and the dashboard turns that into a message naming the remedy rather than a bare error. That is why the sub-resource paths were chosen over a filter parameter: a filter an API does not implement is silently ignored, and an unscoped list looks exactly like a correct one.

The lifecycle

Action What happens
Reconcile Every ten minutes, and on Re-check platform: read every managed environment back, update its runstate, rate-limit flag and idle timer, and flag one that has vanished. Never destroys anything
Build a template Instantiate a base template into a scratch environment, check it against the template contract, install the metadata runner on its control node VM, save the environment back as a new template, then reap the scratch environment. See building a template
Create Instantiate a template, set the idle timer, power on, wait for it to settle, read the VMs back, then enrol the control node's agent
Start / Suspend A runstate change, then a poll until it settles
Install control node (Rebuild control node once its agent is online) Re-issue the enrolment code and re-write the bootstrap. The remedy for every way the first attempt can fail
Gateway Start a BeyondTrust Gateway container on the control node VM, registered into this POV's PRA tenant
Resource Broker Run the staged Password Safe installer on a Windows VM, over WinRM from the control node
Wire One PRA jump item per VM through this POV's Gateway, one Password Safe managed system through its Resource Broker, and one Entitle integration per Linux guest
Share Publish every VM as one password-protected, time-limited publish_set; re-sharing revokes the previous link first
Auto-delete Reap the whole environment when its timer expires, warning on a ladder as the deadline approaches
Destroy Revoke the share link, reap the PAM artifacts, revoke the control node's agent, then delete the configuration and everything Skytap keeps inside it

Three orderings are load-bearing, and each is wrong in a way that leaves a resource nobody can reclaim:

The platform id is persisted before anything else can fail. An environment that exists in Skytap and not in this database is the one failure mode nothing can clean up automatically. So the create call is followed immediately by a commit, and every later step is written to be re-runnable around it.

A failure after creation keeps the id. If the power-on fails, the row stays failed with its Skytap id and a message telling you to press Destroy. Failing without the id is how an orphan is made.

A failed destroy does not mark the row destroyed. Marking it would hide an environment that is still running and still billing. The row stays visible, keeps its id, and Destroy can be re-run once the cause is cleared.

Three smaller rules worth knowing:


The control node VM

Everything the dashboard does inside a POV after it is running goes through one VM in the environment: the control node. It runs the remote agent, and the agent dials out — because a POV lives on a lab platform's private network and this dashboard has no route into it. The Gateway install, the Resource Broker install and the per-VM wire-up all wait on this one thing, so it is worth setting up properly once.

Which VM? Leave Control node VM name blank and the POV works it out: the one Linux VM in the template. A control node is a Docker host by construction, so on the usual shape — one Linux guest beside some Windows ones — its OS identifies it and its name does not have to. That matters because templates come from wherever the SE got them, and a guest called BtPocLin01 was never going to match a name this dashboard guessed.

The full order it resolves in, and why each step exists:

1. The name you typed, if you typed one Matched exactly and case-insensitively, and refused if nothing matches rather than falling through to a guess. A name you wrote down is an instruction: "contains broker" also matches a customer VM called password-broker, and the cost of that mistake is an agent installed on a machine nobody expected
2. The VM that is already the control node An enrolled control node is not guessed away from. The agent's identity lives on that guest, so re-resolving onto a different VM leaves a container polling from a machine nothing points at. Below the typed name, deliberately — otherwise name VM would be silently ignored on the POVs that most need it
3. The only Linux VM Uniqueness, never position. "The first Linux VM" is not a decision a position in a list can make, but "the only one" is a fact about the template. VMs already named as the Entitle agent host or the Resource Broker host are excluded — those are deliberately different machines
4. A VM called broker, if several Linux guests remain The old convention, kept as a tie-break
5. Otherwise it refuses, naming the candidates So you can pick one. The Control node column reads none with the reason beside it

A blank OS is not Linux. Skytap does not report a guest OS as a field — the dashboard infers it from whatever text the platform hands back, and it answers "unknown" rather than guess wrong, because a confident wrong answer sends a Windows VM down an SSH path. A guest whose OS reads — on the POV's VMs tab is therefore skipped by step 3, by the wire-up, and by every other POV feature. Set it yourself in the OS column on that tab; it is stored separately from the platform's own answer, so it survives every later refresh. That one control is the remedy for most "this VM is being ignored" questions, not just for the control node.

You can also correct the name after the fact: name VM beside the Control node column sets it on a POV that already exists, which is what answers a step-5 refusal.

A POV that also runs a Resource Broker needs a second special VM — a Windows Server 2019 or 2022 x64 guest with WinRM enabled, on the same automatic network. See the Resource Broker; the rest of this section is about the Linux control node.

That VM needs three things:

Docker, enabled at boot The agent is a container, and so is the Gateway it later runs beside itself. Enabled matters as much as installed: a POV boots the template, so a daemon that is merely running when the template was baked is a control node with no runtime on every POV. The build installs Docker CE if the base template carries no runtime, enables it, and leaves an existing one alone — Podman works if docker resolves to it, though what the builder bakes is Docker, because the agent drives the Engine API over the socket rather than a CLI
An automatic network Skytap's metadata service answers only on VMs attached to one. On a manual network the VM gets no metadata at all, which looks exactly like a missing runner
The metadata runner Below. Skytap hands user_data to the guest and nothing executes it

What the dashboard generates

A policy.yaml granting one /32 per VM in this POV, on ports 22, 443, 3389, 5985 and 5986 — not the environment's subnet. The subnet is what the platform hands out and it is bigger than the POV; on a shared lab network it can contain somebody else's environment entirely. 169.254.0.0/16 is denied, because the agent has no business at the metadata service the bootstrap itself came from.

Then a /bin/sh script that writes that policy, writes a single-use enrolment code, and starts the agent container. Two lines in it are load-bearing:

SPIRE on the control node

When Let agents attest through SPIRE is on (Settings → Integrations → Remote Agents) and this dashboard's SPIRE server answers, the control node attests through SPIRE, so its signing key is held only in memory. Nothing to configure per POV: each control node install (Install control node or Rebuild control node) does what Migrate to SPIRE does for a hand-installed agent:

On the guest, the script starts spire-agent and spiffe-helper, the same two sidecars as the agent overlay, with the same images and the same locked-down settings. It then waits up to two minutes for an SVID:

The guest The agent starts with
gets an SVID AGENT_SPIFFE_JWT_FILE; the enrolment code is not mounted, and is deleted
gets none: the SPIRE port is blocked, a proxy inspects TLS, or an image cannot be pulled its enrolment code, exactly as before. The script prints a warning and spire-agent's last log lines, and removes the sidecars

The fallback is the design. A customer's network that stops SPIRE is the usual case on site, and a control node on an Ed25519 key is worth more than one that never comes up.

The control node install job's last line, and an SPIRE or Ed25519 label beside the control node status on the POV list, say which way it went. If you expected SPIRE and got Ed25519, the control node VM could not reach SPIRE server address for agents over plain TCP. Teardown removes the control node's SPIRE entries along with the agent.

Port 8081 is the default. On an instance that shares its environment with another dashboard, as the reference POV instance does on port 8091, set SPIRE server address for agents to host:port first: a blank one sends control nodes to the other dashboard's server.

Before it mints anything, the control node install checks that address: it opens TLS to it and requires the certificate to chain to this dashboard's own trust bundle and to name spiffe://<trust domain>/spire/server. If something else answers, such as another dashboard's server, SPIRE is not offered and the job says which address to fix, instead of the guest waiting two minutes and falling back. If nothing answers from the dashboard, it is offered anyway, because that proves nothing about the guest's own network.

Fitting Skytap's 16,384-character limit

Skytap refuses a longer user_data with a 422 ("Contents is too long"). About half of the script is comments written for whoever reads it on the guest, and the policy grows with every VM in the POV. So the dashboard fits it before injecting, changing as little as it can:

  1. sent as written when it fits;
  2. else with its comment lines removed. Heredocs (the policy, SPIRE's configs, the trust bundle) are kept byte for byte;
  3. else gzip-compressed and base64-encoded, between the same two markers, with a short decoder that writes the script to a temporary file and runs it. base64 and gzip are in coreutils and BusyBox alike.

The control node install job says when it had to do 2 or 3. A POV large enough that even 3 does not fit is refused with both sizes named. Cloud POVs use the same 16 KB limit, which is EC2's.

The ordering that matters

The payload is injected after the power-on, never before. An enrolment code lives fifteen minutes and a first boot is not bounded — a Windows template pulling updates can eat all of it. Injecting first hands the guest a code that expired while it was starting, and the symptom is an agent stuck at enrolling with no request ever reaching the dashboard.

Once the agent enrols, the dashboard clears user_data. It is readable by anyone who can read the environment in Skytap, and clearing it also stops a reboot re-running a bootstrap whose code is spent.

Building a template

POV → Templates (/pov/templates), admin only.

A POV is a template instantiated whole, so until this page existed the whole feature was downstream of a catalogue that could only be authored by hand in Skytap's own console. That matters most where the existing catalogue was built for an on-premises approach: a template carrying a full product stack inside the environment is the wrong shape for the SaaS-first POV this dashboard runs, where PRA, Password Safe and Entitle are tenants reached from outside and the environment needs only the customer-like VMs plus a control node.

A template is immutable — there is no edit a template call — so authoring is a bake:

clone a base template → power on → check the contract → install the runner → shut the
environment down → save as a template → reap the scratch environment

The shutdown is a stage, not a courtesy. Skytap will not bake a multi-VM environment whose VMs are running: it answers 409 {"error":"The machine was busy. Try again later."}, and later never arrives — it fails the same way for as long as they are up. The job stops the environment gracefully and waits; a guest that ignores the shutdown gets halted, which is Skytap's documented force-off and still settles on stopped.

The scratch environment is created, used and reaped inside one job. Its id is committed the moment Skytap returns it, before anything else can fail, for the reason the whole lifecycle section gives: an environment that exists on the platform and not in this database is the one failure nothing can clean up — and a scratch one bills until somebody notices.

What the build does about the runner and Docker

This is the point of the page. The template contract below requires the control node VM to carry a metadata runner, because Skytap hands user_data to the guest and nothing executes it. Before this, that runner existed only as an example in this document for somebody to copy into an image by hand — and it is the single most common way a POV fails.

And a runner with no container runtime is the same failure one line later. The payload ends in docker run; everything above it is a mkdir, a heredoc and two || trues. A control node VM with a perfect runner and no Docker therefore reads the payload, dies on that line, and re-reads it every twenty seconds forever — while the POV page says enrolling and names nothing, exactly as if no runner existed. Base templates that carry neither are common: a stock AlmaLinux 8 guest has no docker, no podman, and no socket.

The build installs both. It publishes SSH on the control node VM, reads the login from Skytap's own stored credentials, installs the runner and its systemd unit, then a container runtime if — and only if — docker does not already resolve, and revokes the published service in a finally. If the VM carries more than one credential it tries each in turn — the first the guest accepts wins, and the Runner detail names it — so a box holding a stale login beside a good one builds rather than refusing.

And since 2026-09-16 the injected bootstrap installs a runtime too, using the same block, so a POV made from a template that baked without one repairs itself on the first Install control node press instead of sitting at enrolling. That does not make the bake redundant: an install at bake time lands on every POV made from the template, before anyone is waiting fourteen minutes for it.

Docker CE first, the distro's Podman second. Docker CE comes from download.docker.com and is what the agent's use of the Engine API is tested against, so it leads. But a lab guest routinely reaches its own distro mirror and no external CDN at all — a Skytap AlmaLinux 8 control node resolves appstream and baseos perfectly and cannot reach Docker's — so every step of that install is non-fatal, and podman + podman-docker from the guest's own repository is the fallback. Podman serves the same Engine API on the same socket path, so the agent cannot tell the difference. Podman on a lab VM is a supported outcome here, not a degraded one.

podman-docker is not optional garnish. Podman alone gives no docker command and no /var/run/docker.sock, and the agent speaks the Engine API over that path directly — it never runs the CLI. Nor does installing the package start anything: the install enables and starts podman.socket itself, because a guest where the shim answers and nothing is listening enrols, goes green, and then fails every Gateway and Config-Management job.

Both verbs matter, and they fail differently. systemctl enable podman.socket writes a symlink and starts nothing, so an enabled-only guest has no API until it is rebooted; starting without enabling is the mirror image, and is the trap docker-ce sets on the RHEL family. Where socket activation produces no listening socket the install falls back to podman.service — the API service running persistently, which is what an operator reaches for by hand.

The boot gate asks Podman the same question. It used to skip a podman-docker guest entirely, because systemctl cat docker.service is false there and there is no docker unit to enable — so a template could bake with a socket that answered only because the install had just started it, and every POV from it came up with a working docker command and nothing listening. Either podman.socket or podman.service being enabled satisfies it.

Three more things are worth knowing before you rely on it:

Published services are otherwise deliberately unused, because a published address changes per environment and per power cycle. A build is the one case where that does not matter: the address is used once and revoked in the same job.

The contract report

Verify checks a template — including one somebody else authored — against the contract without building anything. Each check is reported on its own, never collapsed into one badge, because "no control node" and "the control node is on a manual network" have different fixes:

Check fail means
Control node No control node could be resolved, so a POV from this template has nowhere to run its agent: either the name you typed matches nothing, or nothing distinguishes the template's Linux guests. The report names the VMs it did find. Note a template read has no POV, so the per-VM OS override cannot apply here — a template whose only Linux guest reports no OS fails this check while the POV built from it resolves fine once you set the OS on its VMs tab
Control node network The control node is on a manual network. The metadata service answers only on automatic networks, so the guest would receive no bootstrap at all — which looks exactly like a missing runner
Resource Broker host Never fails. No Windows guest is a warning: a PRA-and-Entitle POV does not need one
Workload VMs Never fails. A control-node-only template is a warning — a POV built from it has nothing to demonstrate

Only a fail stops a build. A warning is a statement about what a template is for, not a defect in it.

Whether the runner is actually installed cannot be read from the platform at all — it is a file inside the guest. Neither can whether Docker is: command -v docker is not a thing an API answers. So a template carrying neither still reports pass on all four checks above, which is correct — those checks are about what the platform can see — and is also the reason a build installs both rather than trusting a green report. A Verify says what it cannot know rather than guessing.

The scratch environment bills

Three guards, because a running environment costs money whether or not anyone is watching:

  1. suspend_on_idle is set on it before the power-on. This is the only one that survives the worker being killed mid-build, which is the failure nobody notices.
  2. The job reaps it after a successful bake, unless you asked to keep it. It deliberately does not reap on failure — a build that broke is the one whose environment you may want to look at.
  3. Discard on any build row reaps it by hand. A failed build keeps its environment id precisely so this can work — and a failed reap does not mark the row discarded, because that would hide an environment that is still running.

The template contract

Skytap's bootstrap_injection mechanism is metadata: the platform stores the payload and the guest fetches it. There is no cloud-init datasource and nothing runs it for you, so the control node VM in your template must carry a small runner.

The builder writes this for you. Building a template generates the runner from the same marker constants the dashboard writes into the payload, installs it, and installs Docker beside it when the guest has no runtime. The shape below is what it generates, and what you need if you are baking a template by hand — POV → Templates will also just hand you the script to paste. That script is plain ASCII on purpose: the console you may have to type it into has no clipboard.

The runner has to do four things. Anything that does them is fine; this is the shape:

#!/bin/sh
# /usr/local/sbin/dashboard-bootstrap-runner - run at boot and keep running.
set -eu
MARKDIR=/var/lib/dashboard-bootstrap
mkdir -p "$MARKDIR"

while :; do
  body=$(curl -fsS http://169.254.169.254/skytap/vms/self/user_data 2>/dev/null || true)
  case "$body" in
    *BEGIN-DASHBOARD-AGENT-BOOTSTRAP*END-DASHBOARD-AGENT-BOOTSTRAP*)
      sum=$(printf '%s' "$body" | sha256sum | cut -d' ' -f1)
      if [ "$sum" != "$(cat "$MARKDIR/last" 2>/dev/null || true)" ]; then
        printf '%s' "$body" > /run/dashboard-bootstrap.sh
        sh /run/dashboard-bootstrap.sh && printf '%s' "$sum" > "$MARKDIR/last"
      fi
      ;;
  esac
  sleep 20
done
  1. Poll, do not read once. The payload arrives after the VM is up, because of the ordering above. A runner that reads user_data once at boot finds it empty and stops.
  2. Require both markers. BEGIN-DASHBOARD-AGENT-BOOTSTRAP and END-DASHBOARD-AGENT-BOOTSTRAP must both be present before anything runs. A truncated metadata read would otherwise execute half the script — and the half at the top is the half that deletes the running agent and its state volume.
  3. Key the "already ran" marker on the payload's hash, not on a flag. A reboot with unchanged user_data must not re-run; a re-injection — a new enrolment code — must. A boolean gets exactly one of those right.
  4. Run it as root. It writes to /etc/dashboard-agent and calls docker.

Install it as a service that starts at boot — a systemd unit with Restart=always is the usual form — and bake it into the template. It is idempotent and it costs one HTTP call to a link-local address every twenty seconds.

If your account's metadata service does not serve http://169.254.169.254/skytap/vms/self/user_data, read http://169.254.169.254/skytap instead and pull user_data out of the JSON document. The payload is what matters; the path is Skytap's.

The calls that are v1

Most of what the dashboard sends is the v2 API. Which calls have a v2 form is decided per verb, not per path, and the paths that look most settled are the ones that are not: /v2/configurations/{id} serves GET and PUT and not DELETE, and /v2/templates/{id} serves only GET. Every one of these was learned from a live account answering 404 {"error":"Not Found"} — never from the documentation, which reads as though both objects support the full set.

Operation Call Why not v2
Create an environment from a template POST /configurations.json — template_id, optionally project_id and name no POST on the v2 collection
Create a template from an environment POST /templates.json — configuration_id, name no POST on the v2 collection
Copy VMs into an existing environment PUT /configurations/{id}.json — template_id, vm_ids[] the v2 PUT answers 200 and does nothing — see below
Delete an environment DELETE /configurations/{id}.json DELETE /v2/configurations/{id} 404s for every id, including a running one
Delete a template DELETE /templates/{id}.json /v2/templates/{id} serves only GET
Set a template's description PUT /templates/{id}.json — description same; the v2 PUT 404s

Everything nested under those paths is v2 and works: …/vms/{id}/user_data.json, …/publish_sets, …/vms/{id}/interfaces/{id}/services, …/vms/{id}/credentials. So is PUT /v2/configurations/{id} for a name, a runstate or an idle timer.

A 404 on one of these means the endpoint, not the id — and that is the trap worth remembering, because the message reads exactly like "the id you sent does not exist". The first live template build was spent inspecting a base template id that was perfectly good.

A 404 on a DELETE is not proof the thing is gone

The delete rows above cost six environments. delete_environment sent the v2 path, Skytap 404ed it, and the adapter read that 404 as "somebody already deleted it" — so every POV destroy reported a clean success, marked the row destroyed, wrote "The infrastructure is gone" on the archive page and deleted the VM rows that were the only record of what had been inside. The environments went on running, and on billing, invisible to every sweep: pov_reconcile only asks about rows that are not destroyed.

The deletes are still idempotent — a teardown that fails because the thing is already gone leaves a row nobody can ever clean up — but the evidence for "already gone" is now a direct read of the object, never the status code of the delete (skytap_service._confirm_gone). A 404 on the delete with a readable object raises, which puts the POV row in failed with a re-runnable Destroy rather than in destroyed with an orphan. It is the same refusal to guess that makes pov_reconcile confirm a missing environment with a direct read before flagging it, and it is worth applying to any delete added here later.

A read that fails with something else — a 423, a 500, a network blip — re-raises rather than being guessed either way.

The merge, which is quieter still

The VM copy in the table above is the one with no failure signal at all. It is v1 and a PUT, at a path the v2 API also serves.

Skytap's v2 reference says plainly that "VMs are created indirectly, either by creating an environment from a template or by merging a template into an existing environment", and documents no endpoint for the second half. The v1 merge is that endpoint. It also accepts merge_configuration in place of template_id, to merge from another environment; the dashboard does not use that form.

PUT /v2/configurations/{id} exists, is what update_environment uses for the name and the idle timer, and answers a template_id with 200 and the environment unchanged. No error, no VMs, nothing to investigate — strictly worse than the 404 the creates give, because a 404 at least says something happened. skytap_service.add_vms is the only caller of the v1 path and tests/test_pov_add_vms.py pins it against the code, not the docstring.

Two behaviours worth knowing:

If you add VMs in Skytap's own UI instead

The dashboard will not notice. pov_reconcile is deliberately one collection read per pass — that is the property that lets it run over every POV cheaply — so it refreshes a runstate, a rate-limit flag and an idle timer, and never a VM list. Adding a per-environment VM read to it would turn one call into one per POV.

pov_environment_vms is refreshed by three things: a provision, a power action, and the add-VMs job. Rebuild control node (or Install control node) does it too, via run_env_broker, which is the shortest way to pick up a guest added outside the dashboard — and it is what you would press for a new guest anyway, since a Config-Management grant is written at enrolment.

What is deliberately not used

The Terraform provider. skytap/skytap last released v0.15.1 in November 2022, and its own documentation says it "doesn't enumerate the resources contained within that template, including VMs and networks" — which is the one thing a POV needs, because a POV is a template instantiated whole. The dashboard talks to the REST API directly, as the Portainer and Rancher integrations do. That also keeps the feature clear of the provider pre-cache coupling in the image build.

Published services for PAM wiring. Skytap can NAT a guest port to a public ip:port, and the dashboard displays those, but the wire-up does not use them: a published address changes per environment and per power cycle. POV wiring reaches VMs on their private IPs through a Gateway inside the environment.

Capabilities

The registry (services/lab_platforms.py) records what each platform can do, so a feature a platform lacks degrades visibly instead of failing late:

Capability Skytap
Templates yes — /v2/templates
Runstate yes — running / suspended / stopped / halted
Idle suspend yes — suspend_on_idle, per environment, in seconds
Bootstrap injection metadata — per-VM user_data, read by the guest at http://169.254.169.254/skytap. Used by the control node VM
Share link yes — publish sets, with a password and an expiry. The expiry is two fields, expiration_date + expiration_date_tz; sending the date alone is a 400 that names the tz
Stored credentials yes — …/vms/{id}/credentials. Used by the Resource Broker install, which is why the dashboard stores no Windows password for a POV
Verify yes — one page of /v2/templates, surfaced as Test connection in Settings
Project scoping yes — /v2/projects/{id}/templates and /v2/projects/{id}/configurations
Template authoring yes — POST /templates.json with a configuration_id. There is no edit a template call on any lab platform, so authoring is always instantiate → change → bake. Used by building a template
Published services yes — …/interfaces/{id}/services, a guest port NAT-ed to a public ip:port. Used only by a template build, for the length of one build
Add VMs to a live environment yes — PUT /configurations/{id}.json with template_id + vm_ids[]. The only platform that can: a cloud POV's VM set is whatever its template service created. See the quietest v1 trap

The two project paths are not yet confirmed against a live account. Three things are assumed: that they return the same object shape as the account-wide collections, that they honour count/offset pagination, and that a project id the token cannot see answers 404 rather than an empty list. They are flagged the same way in the code, above the one place the paths are built. Nothing changes for an install that leaves the Project ID blank, and Test connection is the quickest way to check one that does not.

bootstrap_injection is one intent with more than one mechanism. Skytap hands data to the guest and the guest fetches it; another platform might run a script on the guest instead. The distinction matters because Skytap's metadata service works only on VMs attached to automatic networks, and nothing executes user_data for you — there is no cloud-init datasource, so the guest must fetch it. That is what the template contract above specifies, and it is why the control node's agent install refuses outright on a platform whose mechanism is anything other than metadata rather than failing somewhere inside a job.

Troubleshooting

Symptom Cause Fix
"Skytap rejected the credentials … uses an API token, not your account password" The account password was pasted into the token field Use the API security token from the Skytap account page
The POV page says Skytap is not configured No URL, username or token stored Settings → Integrations → Skytap
Settings refuses to enable it with a 409 This is an estate instance Skytap is POV-only; see POV profile
No POV nav link at all pov_environments_enabled is off, or the profile is demo The flag must be ON and the profile must be pov — GET /api/features shows install_profile
"Skytap is still busy after N retries" The account is genuinely rate-limited Expected under heavy concurrent use; retry shortly. Running or suspending many VMs at once makes it more likely
An environment shows a rate-limited badge Skytap set rate_limited on it Operations against it will be slow until it clears
A POV reads running on the page but is suspended in Skytap The reconcile sweep has not run since it changed The row says how old the reading is — confirmed 8m ago. Press Re-check platform for the answer now. Both power buttons show while a reading is stale, so the one you need is never hidden
A POV is badged gone from the platform Skytap answered 404 for it Somebody deleted the environment outside this dashboard. The row is kept deliberately — it holds this POV's tenant references and reaping manifest — so use Destroy to close it out, which is idempotent on a 404
A POV is badged gone but definitely exists The Project ID changed under it, and the direct read also failed The sweep confirms with a direct read before flagging, so this needs both to fail. Check the Project ID, then press Re-check platform — the flag clears by itself once the environment is visible again
Environments list is empty but the account has some The token's user cannot see them, or they are outside the configured project Check the user's access in Skytap, or clear the Project ID to widen the scope. Test connection says which of the two it is
Test connection says the account exposes no templates The token authenticates but sees nothing Check the user's access in Skytap. If a Project ID is set, clear it first to find out whether the project is the constraint
"Skytap has no project N (404)" on the POV page or in Test connection The Project ID is stale, wrong, or belongs to an account this token cannot see Correct or clear it in Settings → Integrations → Skytap. Blank lists everything the token can see
Test connection says the host could not be reached DNS, a firewall or an outbound proxy Not a credential problem. Check outbound HTTPS to the API URL from wherever the dashboard runs
VM counts show — The collection read did not include the VM array Expected. Open the environment for the measured count — a dash means "not measured", never zero
"publishing the share link failed … Expiration date tz is invalid" Skytap did not accept the expiry's timezone. A publish set's expiry is two fields — expiration_date and expiration_date_tz — and the date alone is a 400 naming the tz Fixed: the dashboard now sends both, the date in UTC and the zone as UTC. If it recurs, the message names the zone it sent — change SHARE_EXPIRY_TZ in services/skytap_service.py to one the account accepts
The Control node column reads none and the row names other VMs The control node could not be resolved. Three different causes, and the message says which If it says no VM reports a Linux OS: set the OS in the VMs tab's OS column for the guest that runs Docker. If it says several VMs could be the control node: press name VM beside the Control node column and name one. If it says no VM is named X: you typed a name this template does not use — clear it to auto-detect, or correct it
The control node resolved to the wrong guest The template has one Linux VM and it is not the Docker host Press name VM and name the right one. A typed name always wins over auto-detect
The Control node column reads none on a POV that used to work A second Linux VM was added, so auto-detect became ambiguous This refuses rather than silently moving the control node. Name the original with name VM. An already enrolled control node is never moved by this
The Control node column reads enrolling and never changes Nothing executed the payload, or it executed and died The control node VM has no metadata runner (template contract), has no docker, is on a manual network, or cannot reach the agent endpoint. Fix it and press Install control node to re-issue
"no agent enrolled within 14 minutes" Same causes as above Read the whole message first. When this dashboard built the template, the failure names it: "built that template (…) and recorded 'docker: MISSING' on its control node" means the runtime is the cause and the bootstrap's own install could not fix it either. Otherwise: the bootstrap is still on the VM — user_data is only cleared on a successful enrolment, so being able to read it proves the agent never enrolled. On the guest, journalctl -u dashboard-bootstrap-runner -n1 -o cat names the line that failed and docker logs dashboard-agent, if it ever started, names the rest
"nothing is listening at /var/run/docker.sock" The runtime is installed and its API is not running Almost always Podman with podman.socket not started — the docker command answers while nothing serves the Engine API, and the agent uses only the API. systemctl enable --now podman.socket on the control node VM, then press Install control node. The install does this itself; seeing this means it did not take
"/var/run/docker.sock … is a DIRECTORY … NOT EMPTY" A docker run -v created one on a host whose daemon was down, and something has since been written into it The install clears an empty one itself — rmdir, never rm -rf — so seeing this means there is content in it. Look at what is inside, remove it, make sure the daemon is up, and press Install control node again
"this dashboard does not know its own public URL" No pinned audience and no public_base_url An agent inside a customer network needs an address. Set it in Settings → Integrations → Remote Agents
"the agent endpoint is http://…" The audience is plaintext The agent refuses to sign over plaintext, so the control node would never enrol. Terminate TLS and correct Public base URL
A build fails with "does not satisfy the template contract" The base template has no control node VM, or its control node is on a manual network Read the contract report on the build row. Press Discard to reap the scratch environment, fix the base template, and build again
The Runner column reads failed and names a firewall The SSH install could not reach the published port It dials a NAT-ed high port, not cloud.skytap.com. Either open that egress or clear Install the metadata runner and Docker and paste the install script onto the control node VM by hand — the template itself is fine
The Runner column reads failed with "refused all N stored credentials" Every login stored against the control node VM was rejected by its sshd The detail names the usernames it tried. Correct the credential on that VM in Skytap and build again — SSH answered, so this is the login and not the route
The Runner column reads failed with "the install exited …" A login worked but the script did not finish It installs as root, so check that the credential you left is an administrator with sudo. The detail carries the state read back off the VM afterwards, so it says which half landed. The template still bakes; paste the script in by hand
The Runner detail says docker at boot: disabled The daemon is up but its unit is not enabled The build refuses this rather than baking it: a template is booted for every POV, so a daemon that is only running now gives each one a control node with no runtime. systemctl enable docker on the control node VM and build again — dnf install docker-ce leaves it disabled on the RHEL family
The Runner detail says docker: MISSING The runner landed and neither runtime did No route from the guest to download.docker.com and no podman in its own repository (see prerequisites) — the detail names what it tried. The template is still usable: the POV's first Install control node press runs the same install again, and a POV whose control node still will not enrol now says this in the job's own error. Install a runtime by hand and re-bake to fix it for every POV; the runner on the VM is already correct
The Runner column reads skipped You cleared the checkbox, or no control node VM was resolved Not a failure. Paste the install script from POV → Templates onto the control node VM
A build row shows a build env that is still running The build failed, or you asked to keep it It is billing. Press Discard to reap it. A failed build keeps the id on purpose, so this always works
Discard says the environment could not be deleted Skytap refused the delete The row stays visible with its id rather than being marked discarded — marking it would hide a running environment. Clear the cause and press Discard again
A POV built from a new template still reads enrolling The runner is present but the control node cannot reach the agent endpoint The runner is not the only requirement — see the control node VM. Check Docker and the automatic network, then press Install control node
The control node was online and 401s after a re-run Its state volume survived a control node rebuild The generated bootstrap removes it. A hand-run docker run that skipped that line leaves the old identity in place — remove the dashboard_agent_state volume and press Rebuild control node