← Back to dashboard

Notifications

Audience: operator · Profile: both · Read this when: you want to hear about expiring resources and failed jobs without opening the dashboard.

Part of Operations.

Outbound only. The dashboard POSTs to endpoints you configure. It never opens an inbound webhook, and nothing here listens for anything. (A per-tenant inbound endpoint is reserved for the hosted edition — see SaaS Roadmap — and is a different feature entirely.)

The problem it solves: everything the dashboard knows is on a page you have to open. That was tolerable until the auto-delete timer started destroying resources on a schedule, whose only warning was a banner in a browser nobody had open. Three more signals had the same shape — a cloud budget, a stale secret, a drifted host — each with an evaluator and no way to tell anybody.

Off by default, and dry-run by default even once on.

Architecture

 EMIT — INSERT only. No network call, ever, at any of these sites.
   job_service.set_failed / reconcile_stale_jobs   ──┐
   expiry_reaper  (warn, reaped)                    ──┼──► notification_deliveries
   notify_scanner (cost, secrets, drift)            ──┘        one row per endpoint
                                                                       │
 DELIVER — worker process, alongside the job runner                    ▼
   notification_service.drain_loop()  ── claim → POST → sent | retry | failed

Two properties carry the design:

A notification can never fail the thing it reports on. Emit sites only write a row. The HTTP call happens later, in a different process, in a different transaction. A dead webhook cannot slow down, block, or fail a deploy — and this is enforced statically by tests/test_notify_wiring.py, which fails the build if job_service so much as imports an HTTP client.

UNIQUE(dedupe_key) is the coordination. The app runs gunicorn -w 2 and the worker at replicas: 3 — five processes that all reach the same tick. An in-process dedupe set would be worthless across them; a database constraint is not. That is also why no advisory lock is taken.

Delivery is at-least-once. A worker killed after the POST but before the sent write re-sends when the row is reclaimed 10 minutes later. A duplicate alert beats silence.

Events

Selection is the Events to send CSV. notify_min_severity is a second filter on top.

Not wired on purpose: job.completed and job.cancelled. set_completed fires for every expiry_sweep every 30 minutes, and a 30-VM bulk deploy would be 30 messages.

Endpoints

Each endpoint is a URL plus a format. Transport is always HTTP POST; the format picks the body, because Slack and Teams reject anything that isn't their own shape.

Add them under Settings → Notifications → Endpoints. URLs and signing secrets are Fernet-encrypted at rest with the same key as every other stored secret, and are never returned by the API — the UI shows a scheme+host hint.

slack

An Incoming Webhook URL from the Slack app that owns the channel. Body is {"text": …, "blocks": […]}.

Slack reports invalid_payload / no_service as HTTP 200 with an error body. The dashboard treats any 200 whose body isn't ok as a failure, so a misconfigured webhook shows up in the delivery log instead of silently doing nothing.

teams

A Power Automate Workflows webhook. Create a flow from the "Post to a channel when a webhook request is received" template and paste its URL (it will be on logic.azure.com).

Three things that will otherwise cost you an afternoon:

• The old connector webhooks are gone. Office 365 Connectors inside Teams were permanently switched off in May 2026, not deprecated. A URL on outlook.office.com will not work and cannot be made to work — recreate it as a Workflow.
• Workflows answers 202 Accepted, not 200. The dashboard accepts any 2xx. If you are writing your own receiver, don't accept only 200.
• The card needs the envelope. The body is {"type":"message","attachments":[{"contentType":"application/vnd.microsoft.card.adaptive","content":{…}}]}. Posting a bare Adaptive Card is the single most common reason a Workflows webhook "succeeds" and renders an empty post. If your Flow reads the card explicitly, the path is triggerBody()?['attachments']?[0]?['content'].

Messages post as the Flow bot; the name and icon can't be customised. That's a Microsoft limitation, not a setting we've omitted.

custom — and how to send email

A stable, versioned JSON envelope. There is no SMTP client in this codebase, and adding one would mean a blocking I/O path, TLS modes and its own CA handling for one more delivery target. Instead: point a custom endpoint at a Power Automate Flow, a Zapier hook, or anything else that can send mail — and get a mailbox, a channel, and a ticket queue out of one endpoint.

{
  "version": 1,
  "event": "resource.expiring",
  "severity": "warning",
  "subject": "[WARNING] web-01 auto-deletes in about 23h",
  "body": "…",
  "resource": {"id": "job:3f21…", "kind": "vm", "name": "web-01",
               "cloud": "aws", "region": "us-east-1", "workgroup": "lab"},
  "fields": {"Expires": "2026-07-30T14:00:00", "State": "active"},
  "url": "https://dashboard.example.com/inventory",
  "occurred_at": "2026-07-29T15:04:05Z"
}

version is the contract. Receivers can rely on these keys.

Headers:

The signature is HMAC-SHA256(secret, "<timestamp>." + raw_body) over the exact bytes posted (compact separators, sorted keys). Verify against the raw request body, not a re-serialisation of the parsed JSON. The timestamp gives you replay protection.

The signing secret may be an aws_sm:// / azure_kv:// / gcp_sm:// / bt_safe:// reference; it is resolved at send time.

Rolling it out

1. Add an endpoint and press Test. This ignores dry-run and sends immediately, returning the verbatim error — which is the whole point of the button.
2. Enable the feature (notifications_enabled), leave Dry run on. Set the Dashboard URL.
3. Watch Recent deliveries for a day. Everything appears with status dry-run, rendered exactly as it would be sent.
4. Turn Dry run off.

Settings

Failure modes

Everything fails with CERTIFICATE_VERIFY_FAILED, but terraform and boto3 work. You're behind a TLS-inspecting proxy. httpx verifies against certifi and ignores SSL_CERT_FILE / REQUESTS_CA_BUNDLE — which this image sets and docker-compose.corp-ca.yml mounts over. The dashboard resolves the bundle explicitly from those variables, so the fix is to make sure your CA actually reaches the container: use the corp-CA overlay, or --corp-ca at onboard time. There is deliberately no "disable TLS verification" switch.

A Teams post arrives blank. The Flow isn't reading the Adaptive Card from the attachment. See the envelope note above.

Everything fails right after a JWT_SECRET_KEY rotation. Endpoint URLs and secrets are encrypted with a key derived from it. After a rotation they can't be decrypted, and the delivery log will say so. Re-enter the URL and secret on each endpoint. (Every stored secret in the dashboard has this property — it isn't specific to notifications.)

A resource was deleted and never warned. Check /api/notifications/deliveries?status=failed. expiry_warned_at is stamped only after the outbox accepts the row, so a warning that was never queued — feature off, no endpoints, storm-suppressed — is re-offered next sweep. But a warning that was queued and then failed to deliver has burned the latch for that resource.

Duplicate messages. Expected, occasionally: delivery is at-least-once. If it's constant, look for a worker being killed mid-send (rows reclaimed from sending).

Nothing at all is queued. In order: is the feature on; is there an enabled endpoint; is the event type in the CSV; is its severity at or above the floor; is the queue over its ceiling.

Where things live

Deliberately not built: per-user subscriptions, digests, on-call routing, and owner-address lookup. Routing is one function (recipients_for semantics live in emit), so those are additive when someone actually needs them.