← Back to dashboard

The Gateway and the Resource Broker

Audience: operator · Profile: pov · Read this when: a POV's guests need a route in and a credential rotated, and you are installing the two things that do it.

Part of A POV Instance. The outbound-dialling Gateway that brokers sessions in, and the Resource Broker that lets Password Safe reach the guests.

The POV Gateway

The point at which a POV becomes reachable. POV page → the Gateway column → edit.

A POV's VMs live on the lab platform's private network, and every PRA jump item a later release creates has to name a Gateway that can see them. This installs one: a BeyondTrust Gateway container, on the control node VM from the control node section, registered into the PRA tenant this POV is wired into.

The dashboard creates the Gateway in PRA for you. It uses the tenant's API account (the client id and secret on the tenant) to create a clustered Linux Gateway, reads the Docker deploy key PRA returns for it, and installs it on the control node. The sequence is:

1. On the POV, press edit in the Gateway column. The name defaults to POV-<pov name>; change it if you want.
2. Press Create & install. The dashboard creates the Gateway in the customer's PRA, stores its deploy key, and queues the install. The control node's agent starts the container and the job reports what happened on the VM.
3. Press check to ask PRA whether a node of it is connected.

The tenant's API account needs Configuration API access with permission to manage Gateways (PRA Management > API Configuration). Without it the create answers 403, and the refusal says so.

A Gateway that already exists under that name is refused, never adopted. Its deploy key would join this POV's control node to a Gateway somebody else made, inside the customer's appliance, and a name proves nothing about who owns it. Pick another name, or use the manual path below if it really is this POV's Gateway.

Using a Gateway you created by hand. Under use an existing Gateway (paste key), set the name you gave it in PRA, paste its deploy key and press Install. This is the same path the cloud gateway hosts use, where the key sits in aws_ecs_docker_deploy_key and similar settings.

The row shows created here when the dashboard made the Gateway. That is the only kind it ever deletes. See Teardown.

Where the deploy key goes, and where it does not

It is stored encrypted with the same Fernet key as every other secret here, and it reaches the control node VM over the agent's sealed per-job channel — fetched once the job is running, bound to a key that exists only for that fetch.

It is deliberately not in the job's metadata, not in the signed job envelope, and above all not in the lab platform's user_data. That last one is the important distinction from the enrolment code, which does ride that channel: an enrolment code is single-use and lives fifteen minutes, whereas a PRA deploy key is neither. Every node registered with it joins the same Gateway — that is exactly what makes the cloud gateway hosts a cluster — so anyone with read access to the environment seeing it once is enough.

What the agent will and will not do

The Gateway image comes from the control node's policy.yaml, never from the job. A job says only install or remove; there is no image field, no container name and no privileged flag anywhere in the protocol. That is the same rule the Config-Management and hypervisor runners follow, and examples/remote-agent/policy.example.yaml documents the gateway: block for an operator's own agent.

On a POV that file is generated by the dashboard, so read it accordingly: it is not the trust boundary it is on a customer-owned agent, because the dashboard also created the VM, the bootstrap and the agent. Keeping the same shape is still worth it — one code path on the agent rather than two — but nobody should read a generated gateway: block as the customer having agreed to it.

privileged: true is in that block and is not decoration. A Gateway carries protocol tunnels, which needs NET_ADMIN, NET_RAW, IPC_LOCK and /dev/net/tun. Docker has no granular way to grant that set, so it is the all-or-nothing flag — and the agent refuses to start a Gateway without it rather than start one that cannot work. Without those capabilities a Gateway registers online and every tunnel silently times out, which reads as a firewall problem for a long time.

The images, and who fetches them

On a customer-owned agent host the agent will not pull an image for you — a pull is a network fetch of executable content, so policy.yaml grants and the operator provisions. That rule is unchanged, and the remote-agent guide still describes it.

A POV control node is the exception, because every clause of that argument inverts. The dashboard built the VM, wrote the policy.yaml and chose the image names, so leaving the fetch to a person who was never told produced agent_gateway failing with "is not present on this host" on a machine nobody had ever logged into. The bootstrap therefore pulls them — the same script that mounts the Docker socket, under the same argument, and without the agent doing the pulling.

The agent's own image, plus the ones the generated policy names:

chrweav/dashboard-agent:latest is on that list for a different reason, and it is the one that used to be missing. docker run fetches an image only when it is absent locally, so a control node VM resolved that tag exactly once — at first bootstrap — and every rebuild afterwards started the copy cached in the guest's image store on the day the VM was built. A control node is long-lived, so an agent fix could not reach a POV that already had an agent: it merged, it was tagged, :latest moved, you pressed Rebuild control node to pick it up, and the guest ran the same old binary and failed the same old way. Pressing Rebuild control node now refreshes the agent as well as its runners.

The pull runs before the agent is replaced, so a slow registry costs a rebuild nothing, and it is non-fatal: a POV with an enrolled agent and one missing image refuses one job and names it, which beats a bootstrap that died at a registry hiccup and left no agent at all. If a pull did fail you get the old refusal, which is now accurate — docker pull it on the control node and run the job again.

This needs outbound HTTPS from the control node VM to Docker Hub, alongside the runtime install's own need for either download.docker.com or the guest's distro repository.

The bootstrap installs a container runtime

Since 2026-09-16 the injected payload installs one when the guest has none, using the same block a template bake runs — Docker CE first, podman + podman-docker from the guest's own repository as the fallback. A POV built from a template that baked without a runtime therefore repairs itself on the first Install control node press, instead of sitting at enrolling and saying nothing.

It is idempotent and skips a guest that already resolves docker, so a healthy control node and every cloud POV pay nothing for it. It runs before the image pulls (which need a runtime) and before the docker rm -f, so a slow or failed install costs a rebuild nothing.

Podman is a supported runtime here. It serves the same Engine API on the same socket, and the agent only ever speaks that API — it never runs the docker command. Which is why the install enables and starts podman.socket and then checks that something is actually listening: the shim answering while nothing serves the API is a control node that enrols, goes green, and fails every Gateway and Config-Management job.

Enabling alone leaves the guest with no API until its next reboot; starting alone leaves it with none after that reboot. Where socket activation yields no listening socket the install falls back to podman.service, the API service running persistently.

Telling the dashboard what a guest runs

The control node VM is found by its OS, not its name — the only Linux VM in the template, unless you name one (the order it resolves in). That rests on the OS column being populated, and Skytap does not report a guest OS as a field: the dashboard infers it from whatever text comes back and answers "unknown" rather than guess wrong.

So a guest whose OS column reads — on the POV's VMs tab is invisible to the control node's auto-detection, and to the wire-up, guest steps, functional accounts and the Password Safe half besides. Set it in that column. It is stored separately from the platform's own answer — the platform's stays visible as the blank option — so a later refresh cannot overwrite it, and clearing it hands the question back. Only linux and windows: a third value would just be a new way of saying unknown.

How big the control node VM has to be

One vCPU is enough. It did not used to be: the Config-Management runner asked for a flat 2.0 CPUs, and Docker refuses a create whose CPU limit exceeds the host's CPU count — 400 Range of CPUs is from 0.01 to 1.00, as there are only 1 CPUs available. So a one-vCPU control node could not run Config Management at all, which is the size most lab templates give it. The runner now clamps its ceiling to what the host reports.

Memory is the real floor: the runner caps a run at 1 GiB and the controller alone is ~150-200 MB once its dependencies import, so give the control node at least 2 GB of RAM.

Why "is it there?" is the wrong question

A Gateway is a cluster, not a host. Re-installing on a rebuilt control node VM adds a node and PRA parks the dead one, so the name is present either way. The check therefore asks whether a node is connected, and reports the node count next to it — if that number is climbing across reinstalls, old nodes need retiring in the appliance.

connected is tri-state. Which field an appliance reports varies by version, and one this dashboard does not recognise reads as unknown, never as disconnected: telling an operator their Gateway is down because we did not know a key name is worse than saying nothing.

Upgrading a POV that predates this

A control node enrolled before this release has a policy.yaml with no agent_gateway grant and no gateway: block, and an agent image below 2.4.0, the build that first carried the agent_gateway handler. Both are fixed by pressing Rebuild control node on the POV, which pulls the current agent image, re-issues the enrolment and rewrites the policy. The install refuses with that remedy rather than queueing a job the control node would decline.

Check it landed on the Agents page: the control node's reported version is the build that is actually running, and it moves on every agent change — a rebuild that leaves it unchanged did not fetch anything.

Teardown

Destroying a POV queues the Gateway removal first, then revokes the control node's agent, then deletes the environment. The removal is tidiness, not reaping — the environment delete takes the container with it either way — so a POV whose control node is already gone still destroys cleanly, with a line in the job log saying the PRA-side node will linger.

The stored deploy key is cleared synchronously, because that is local state and leaving a customer's credential in this database after their POV is gone is the part that matters.

If the dashboard created the Gateway (created here), the destroy then deletes it from PRA as well. This runs after the POV's jump items are removed, because PRA deletes every Asset a Gateway owns along with it. A Gateway you created by hand and pasted the key for is never deleted; retire it in the appliance yourself.

The Resource Broker

The Password Safe half, and the second Windows machine in a POV. POV page → the Resource Broker column → edit.

A Password Safe Resource Broker gives the tenant reach into the POV's private network, the way the Gateway does for PRA. It is a Windows program — Server 2019 or 2022 x64 — so it does not live on the control node VM, which is a Linux Docker host. A POV therefore has two special VMs: the Linux control node running the agent and the Gateway, and a Windows host running the Resource Broker.

The dashboard cannot download the installer. The package is generated per Password Safe tenant and comes from that tenant, so there is no image to name and no URL to fetch. The customer stages it:

1. Download the installer from the Password Safe tenant, along with its installer key.
2. Upload it on Config Management → assets. It is a .exe.
3. On the POV, name the Windows VM, pick the staged installer, and set the resource zone and the installer key.
4. Press Install.

Two parameters, not one

This is the part that catches people. A silent install needs INSTALLKEY and ZONE, and without the zone the installer prompts — which in an unattended run is not an error but a process that sits there until the run's timeout kills it, with an install log that ends mid-dialog. So the dashboard refuses to queue a run with either missing, and says which.

The key is a credential and the zone is not, so they travel differently: the key is stored encrypted and resolved when the agent fetches its sealed bundle, while the zone is a plain value on the POV. Neither reaches the command line of the runner process — argv is readable by every process on that host.

RESTART is deliberately never set. During a silent install it restarts the machine automatically, which would drop the WinRM session mid-play and make Ansible report the failure of a step that had in fact succeeded — the worst kind of wrong answer, because retrying reinstalls a working broker.

There is no login field, on purpose

The Windows credential comes from the lab platform's own stored credentials — Skytap records a login against each VM, and the dashboard reads it per run. So no Windows password for a customer's environment is stored here at all, and a template whose password changed is picked up on the next run with nothing to update.

Skytap stores that as free text — whatever somebody typed in the box — so the dashboard parses it, and refuses rather than guesses. administrator / Passw0rd, administrator:Passw0rd and username: … password: … are understood; a sentence or a multi-line note is refused, naming the VM. A bare space is not treated as a separator, because a password containing one would split in the wrong place.

Several credentials on one VM

This is the normal state, not a fault: a lab guest carries the superuser its template promised plus whatever else somebody left in the box. The dashboard picks between them by the guest's own operating system:

Two tie-breaks sit under that rule:

• Privilege first. The OS superuser wins even when it is domain-qualified, because a Resource Broker installed as a service account fails in ways that do not say wrong account.
• Local second. Between two logins the OS rule cannot separate, the unqualified one wins. A DOMAIN\user login needs a domain controller up and reachable at the moment of the run — in a lab whose boot order is not guaranteed, that failure arrives at WinRM as an authentication error and reads as a bad password. When a domain-qualified login is chosen anyway, the job log says so.

Position in the list is never used. That is still a guess, and it is still refused.

When the rule cannot decide

Two logins of equal standing — two local service accounts, say, or root and administrator on a guest whose OS nobody has established — are refused, and the refusal names the usernames it could not choose between. A parsed username is not a credential; the password and the raw text it came from stay out of every message and every log line.

Answer it once in the Login column on the POV's VMs tab. Type the account name as the platform records it, or just the bare name — administrator selects a stored CORP\administrator. Leave it blank, which is the normal state, to let the OS rule decide. An account the platform does not hold is refused when you type it rather than at the next run, and a name you set survives a VM re-read.

That column stores a username and nothing else. The password is still read live off the lab platform at the moment of every run and is still stored nowhere here.

A template build is the one caller that does not need any of this: it opens the SSH connection itself, so it takes the whole list — best first, by the same rule — and tries each until one is accepted. The build row records which one won. If the guest answers and refuses them all, that fails immediately rather than on the readiness ladder: a rejected password does not become right in fifteen seconds.

If a credential box cannot be parsed at all, fix the box — a wrong username comes back from WinRM as an authentication failure, which reads as a bad password and sends you to reset one that was fine.

What it grants, and why the scope is narrow

Installing the broker runs a playbook, as root, on a host in the customer's environment. That is a much larger grant than the discovery sweep the control node's agent already has, so it gets its own target list in the generated policy.yaml — scoped to the named Resource Broker host alone, or to the POV's Windows guests before one has been named, on 5985/5986 and nothing else. The Linux control node's own address is deliberately not in it.

A POV whose control node was installed before this release has no agent_ansible grant at all. Press Rebuild control node to rewrite its policy; the install refuses with that remedy rather than queueing a job the control node would decline.

How long it takes

Longer than it looks. The bootstrapper is a bundle: it installs the VC++ 2010 and 2015-2019 redistributables, .NET Framework 4.7.2 and the .NET Core hosting bundle before the broker itself. Minutes, and often a fresh download on a clean template.

Teardown

Destroying a POV clears the stored installer key and forgets the broker id. It does not uninstall anything — the environment delete takes the VM with it, and an uninstall run would need the very WinRM session the teardown is about to make unreachable.

The broker's registration in the Password Safe tenant is a customer-side object this dashboard never created, so the job log says it is still listed there rather than pretending to have reaped it.

Failure modes

The Gateway install refuses with "its policy.yaml predates the Gateway grant". The control node was enrolled before this release. Press Rebuild control node — see upgrading.

A job fails with an error that a merged fix was supposed to have removed — the canonical one being Config Management refused with "could not create the Ansible runner (400): Range of CPUs is from 0.01 to 1.00". The control node is running an older agent than the release notes describe. Compare the version on the Agents page against the one in the build you expect, and press Rebuild control node; before this release no amount of rebuilding would have changed it, because nothing ever re-pulled the agent image.

The Gateway registers online and every tunnel times out. privileged is missing from the control node's gateway: block, so the container has no NET_ADMIN//dev/net/tun. A control node rebuild rewrites the policy with it.

The Resource Broker install hangs and the job times out. Almost always a missing ZONE — the installer prompts for it and nothing can answer. The dashboard refuses to queue without one, so this means the zone was set to something the installer did not accept; check the install log on the target.

"no stored credential this dashboard can use". The lab platform's credential box for that VM is empty, or holds something the parser will not guess at. See there is no login field.

The Gateway check says disconnected but the container is running. Look at the node count. A rebuilt control node VM adds a node and PRA keeps the dead one; the Gateway is a cluster, and old nodes are retired in the appliance, not here.