Rollback

CI publishes a moving tag — :test or :prod — plus an OCI revision label. It does not retain a per-commit tag. That single fact determines everything below.

What you can roll back

Surface	Mechanism	Caveat
VPS containers	Point IMAGE_TAG in the test compose at another tag	No other tag is published by CI, so only tags you created exist
A container image	Redeploy a digest captured before the upgrade	You must have captured it
Gateway config	Revert the VPS branch and re-run generate.sh + reconcile.sh	The branch is per VPS
Native apps	Store rollback / halt rollout	Users on the bad build stay there

What you cannot roll back by tag

• Any service whose only published tag is :prod. There is no previous :prod image after CI overwrites it. A genuine rollback means rebuilding the previous commit or redeploying a digest you recorded.
• The worker. The production image is an Artifact Registry tag no repository builds automatically. If CI overwrote nothing, there is also nothing to go back to unless you captured the digest.
• Firestore and RTDB writes. Publishing sets isReady; a rollback of the code does not un-publish a title. Use the admin deletion path, which is itself a cascade.
• Firestore schema. Document IDs are API contracts. A code rollback across a re-keying is a data migration, not a deploy.

Preparing, before you deploy

This is the only reliable rollback strategy:

1. Record the current image digest per service.
2. Record the current gateway commit per host.
3. Note the environment's /health payload and version.
4. If the change touches a queue, note the queue depth and any in-flight job.
5. Keep the previous release commit hash — it is what you will rebuild.

Five minutes of this is worth more than any emergency procedure.

Rolling back a VPS service

1. Choose the target: a digest you recorded, or a tag you built.
2. Update IMAGE_TAG in stacks/greyscreentv-test/docker-compose.yml (or the production equivalent) in a working copy, not in a branch that CI will overwrite.
3. Bring the stack up with Compose; the deploy scripts hold flock, so nothing overlaps.
4. Confirm /health on APIs and /healthz on static frontends.
5. Confirm the flow you changed, not just the container status.

Do not roll back a consumer behind its producer

If the old image cannot drain a queue the API is currently writing to, you will get the 2026-10-21 loop: a queue that always looks busy, and a VM created every reconcile tick. Roll back the producer too, or drain the queue first.

Rolling back the gateway

1. Revert the VPS branch to the previous commit.
2. Run generate.sh, then reconcile.sh.
3. Verify every public path you touched, not just the root.

A gateway rollback can take a path from public to 404. That is a full outage for that route, even though every container is healthy.

Rolling back a title, not a service

A title that went live remains live after a code rollback. To remove it:

• GET /videos/:id/deletionPreview to see the blast radius;
• DELETE /videos/:id for the cascade.

The cascade refuses to run while a transcode is in flight or another deletion marker is younger than ten minutes, and it is idempotent per phase — so retrying after a partial failure is safe.

The safety nets that are not rollbacks

• The worker's 24-hour hard cap and self-delete keep a runaway VM from living forever. Neither protects against a queue entry no deployed consumer can drain.
• The moving tag means "deployed" is a mutable claim. Record digests to make it verifiable.
• A vpsStatus/heartbeat naming a VM that no longer exists ages out in about 60 seconds. It is not evidence of a running worker.

After a rollback

1. Write down which commit is deployed where.
2. Check the queue depths — a rollback can strand entries.
3. Confirm the notification feed is still delivering; a rollback of the admin API can leave the bell silent and a failed upload unreported.
4. Rebuild the :prod image properly before the next forward deploy, rather than leaving the system on an untagged digest.

Next: Health and monitoring.