Health and monitoring

There is no unified monitoring stack. What exists is a set of probes and a handful of records in RTDB and Firestore. Knowing which signal answers which question is the whole skill.

Probes

Component	Probe	Expected
Customer API	GET /health	{status:"ok", appEnv, production}
Admin API	GET /health	Same shape, plus version on current builds
Static frontends	GET /healthz	200
Multimedia	GET /health on :8000	200
Worker	No HTTP	Liveness via vpsStatus/heartbeat
Gateway	Per-domain TLS and path checks	200 on a known path

Version drift looks like a contract change

A deployed build that returns only {"status":"ok"} predates the current controller. On 2026-10-22, production returned three fields while the test API returned one, even though the branch served the full payload. Re-probe before assuming the environment is broken.

The worker heartbeat

vpsStatus/heartbeat is the only liveness signal for a machine that has no HTTP server:

Field	Meaning
instance, env, zone, machineType	Identity; env is the labels.env value
bootedAt, lastSeenAt	Freshness
state	Worker state
queueAtBoot	Depth when the job started
jobsDone, jobsFailed	Counters for this VM
currentVideo	Optional in-flight job

A heartbeat is not a running VM

A heartbeat that names an instance that no longer exists is normal. It ages out at VPS_HEARTBEAT_STALE_MS (default 60 s). The Cloud console's instance list is the truth about existence; the heartbeat is the truth about work.

Signals, by question

"Is the platform up?"

1. /health on both APIs and /healthz on each frontend.
2. /health on multimedia.
3. One catalogue call from a browser on the public origin.

"Is work moving?"

1. RTDB queue/transcode depth.
2. RTDB queue/resolutionEvents depth.
3. vpsStatus/heartbeat.lastSeenAt and state.
4. vpsStatus/idleSince — a worker that has been idle while a queue is non-empty is stuck.

A queue that stays at the same non-zero depth for more than a minute, on a worker that is not processing, means nothing is claiming it.

"Did the upload succeed?"

The admin notification feed, not the wizard:

• video.live — the title is published.
• video.failed — the reason is in the message.

If neither appears, check notifications in Firestore, then the SSE route. The writer may be fine and the transport broken.

"Is anyone being charged twice?"

Flutterwave uses the production merchant in every environment, and the admin API's webhook handler has no dedupe and does not await several writes. Reconciliation is manual:

1. payments for the reference.
2. paymentRefs for the correlation record.
3. webhookEvents for duplicate deliveries.
4. users/{uid}/subscriptions for the resulting record.

"Is the cache serving stale media?"

Compare the cached segment with the object in the output bucket, then evict the title and re-test. A re-transcoded title that still plays the old rendition is a cache question, not a worker question.

Signals people misread

Signal	What it does not mean
Two gstv-vps instances	A bug. One per environment is expected; the labels.env filter keeps them apart.
A heartbeat for a deleted VM	A running worker. It ages out in about a minute.
A green /health	That the catalog snapshot is healthy. The snapshot is in-memory and starts after boot.
A deployed tag	A specific commit. Tags move; only the OCI revision label or a digest ties an image to code.
{status:"ok"} only	A different API. It is an older build of the same contract.
An empty catalogue	A frontend bug. Check project → isReady → readinessState first.

What is missing

• No alerting on queue depth or worker crash.
• No metrics export from the worker beyond RTDB counters.
• No retention policy on notifications, staging, paymentRefs or webhookEvents.
• No test coverage to tell you a deploy broke a route before a user does.
• Secret Manager secrets are created but never read at boot.

These are open items, not configuration you have missed.

A daily five-minute check

GET /health        both APIs
GET /healthz       each frontend
GET /health        multimedia
RTDB               queue/transcode, queue/resolutionEvents, queue/*Failed
RTDB               vpsStatus/heartbeat, vpsStatus/idleSince
Admin web          notification bell shows recent outcomes

If all of that is clean, the platform is doing what it was built to do. The known risks — open identity, open pairing rules, moving tags — will not show up here.

Next: Reference — the variables, endpoints and branch rules.