Skip to main content

Troubleshooting

Most GreyScreen failures are identity, routing, or queue problems, not component bugs. Work from the outside in: which host answered, which project it talked to, which queue the work landed on.

Fast triage​

QuestionHow to answer
Which host am I actually talking to?Browser Network tab. Admin web silently falls back to greyscreentv.testapp.pro without VITE_API_BASE_URL.
Which Firebase project?GET /health on the API for appEnv; the key's project_id for the truth.
Is the API the build I think it is?/health returns {status, appEnv, production[, version]}. Only {status} means an older build.
Is the gateway routing correctly?Compare the public path with the stripped upstream path in domains.json.
Is the worker alive?RTDB vpsStatus/heartbeat.lastSeenAt, not the Cloud console list.

Setup and toolchain​

npm ci fails or Node is rejected​

Cause: Node older than 20. The APIs, Vite builds and the worker all assume Node 20. Fix: switch to Node 20 LTS, delete node_modules, rerun npm ci. Do not mix lockfiles between branches.

gstv-website build refuses to start​

Cause: npm run build:test loads .env.test; build:prod loads .env.production. A missing file aborts the build on purpose. Fix: confirm which script CI uses. Never rename a production env file to satisfy a test build.

Admin web shows data from an old deployment​

Cause: src/config/index.js falls back to https://greyscreentv.testapp.pro/api/admin when VITE_API_BASE_URL is unset. Fix: set the variable and restart Vite; the proxy target is read at boot.

API boot​

The process exits during validateEnv()​

Read the message; it names the check.

Message themeLikely cause
Service account unreadableFIREBASE_SERVICE_ACCOUNT points at a file that is not JSON or not readable by the container user.
Firebase project mismatchThe key belongs to grey-screen-tv but APP_ENV is test, or the reverse.
Stripe key modesk_live_ with APP_ENV=test or sk_test_ with prod.
Missing price IDs / Flutterwave keysEnv layer not loaded; check .env.nonprod.local exists and APP_ENV is set.
Wildcard CORSORIGIN or CORS_ORIGIN is * in prod.

APP_ENV must be set explicitly for npm start. NODE_ENV=production alone does not load the production contract.

The admin API starts but nothing processes​

Cause (most common): RESOLUTION_CONSUMER_DISABLED=true, or a second admin API instance is running against the same RTDB. The claim consumers use child_added with no lease. Fix: run exactly one admin API per database; remove the flag outside the test stack.

A second admin API "works" but duplicates events​

Same cause. Notifications are written with idempotence guards, but transcode and artwork events are not safe to double-process. Stop the extra instance.

Catalogue and playback​

The catalogue is empty​

Check in this order:

  1. Firebase project (wrong project = empty snapshot).
  2. videos.isReady — every title must be true.
  3. readinessState — failed titles are excluded.
  4. The API log for the onSnapshot listener.
  5. Only then the React layer.

The snapshot is in-memory, so a title that went live while the process was down is picked up on the next listener start, not instantly.

A title never appears after upload​

Expected: the wizard no longer waits. The bell should have a video.failed notification.

  1. GET /notifications?limit=20 for the message.
  2. RTDB queue/transcode for a stranded child.
  3. RTDB queue/transcodeFailed for a dead letter.
  4. vpsStatus/heartbeat for a worker that never started.
  5. GCS input bucket for a master that was never written.

A video.failed on a re-transcode of a live title keeps the old renditions serving.

The player shows a playlist but no picture​

Cause: a ladder problem, not a routing problem. Fix: open the master playlist.m3u8 directly. If it lists rungs that do not exist, the worker wrote the master before the renditions completed — re-run the job.

404 from /stream/...​

Cause: the gateway strips /stream and forwards to gstv-multimedia:8000. Fix: check the container is running, the route exists in domains.json, and generate.sh has been re-run. A route added to a repo is not public until the gateway knows it.

CORS errors on a new origin​

Cause: CORS is an allowlist, not a wildcard. Fix: add the origin to CORS_ORIGIN in the environment that must accept it. Do not widen to *; production validation rejects it.

Upload and artwork​

completeUpload answers 409​

Cause: the object's generation changed during conversion — the file was replaced mid-flight. Fix: re-run the crop and upload. A retried request with the same generation returns the stored result instead of 409.

completeUpload answers 422​

Cause: the conversion failed. Staging records image[kind].status = "failed" with the reason. Fix: re-upload the source. If the failure is dimension-related, confirm the artwork matches the crop for its kind: portrait 3:4 or landscape 16:9, within a 1.5% aspect tolerance.

saveVideo answers a missing-object error​

Cause: the objects are verified on the server before enqueueing. Fix: upload in the order the wizard uses — thumbnail, thumbnailPreview, trailer, video — then save. Check the bucket suffix: a -test API cannot see a -prod object.

A queue/imageJobs child keeps the VM alive​

Cause: legacy lane. The current upload path writes nothing there, but VM creation still counts it as work. A child no deployed worker can claim is an infinite signal. Fix: confirm the child belongs to a real abandoned fileId, then remove it after checking the two-key guard described in the incident record. Do not add a producer.

The 2026-10-21 create/delete loop was exactly this: an old worker image could not drain artwork jobs, so the scheduler created a VM every 60 seconds. Deploy the worker first when a queue contract changes.

Auth and payments​

Sign-in fails on the phone but the web works​

Cause: the native apps select Firebase by build variant, and signup failure falls back to a random local user. A "signed-in" user may not exist in Firebase. Fix: check the EAS profile / APP_VARIANT; then check users for the email.

Stripe checkout returns an error after the redirect​

Cause: the return URL does not match the origin that created the session, or the price IDs belong to another mode. Fix: confirm the frontend publishable key prefix matches the API's secret key mode, and the success/cancel URLs point at the same origin.

A mobile-money payment is charged but no subscription appears​

Cause: the Flutterwave webhook did not reach the API, or the admin API's handler did not finish. Flutterwave uses the production merchant in every environment, so the charge is real. Fix: check /payments for the reference, then the webhook log. Do not retry a charge before reconciling.

A user has an entitlement the plan does not grant​

Cause: /users/verifySubcription mints a signed token for any user ID, and clients mostly trust client-supplied state. See Security notes.

TV and pairing​

The phone cannot join a TV session​

  1. The code is six characters from ABCDEFGHJKLMNPQRSTUVWXYZ23456789.
  2. The session must still be status: "waiting"; the phone does not check expiresAt.
  3. Both devices must use the same RTDB instance. The three copies of the pairing contract differ only in RTDB_URL — a mismatch here looks like "invalid code".
  4. RTDB rules are currently world read/write; a rules change that tightens them will break clients that do not authenticate.

The remote does nothing except Back​

Expected. Only back and navigate.screen are executed; other actions are published on RemoteEffectBus with no subscribers. See TV pairing.

CI and deployment​

Test deploy succeeded but the site is unchanged​

Cause: the deploy ran against the wrong VPS branch, or the browser is on a cached bundle with a moving image tag. Fix: read the deploy log for the resolved branch; hard-reload once.

Production deploy has no approval​

Cause: the GitHub production environment has no protection rule configured, or the job does not reference it. Fix: check the environment settings before assuming the workflow changed.

Rollback to a previous build is not possible by tag​

Expected. CI publishes moving tags (:test, :prod) and an OCI revision label — no retained per-commit tag. Rollback means rebuilding the previous commit or redeploying a digest captured before the upgrade. See Rollback.

The worker image on the VM is older than the repository​

Expected. The instance template pulls the Artifact Registry tag, and no repository builds it automatically. The main workflow of gstv-vps pushes a different registry image and has no deploy job.

Getting unstuck​

  • Reproduce with the smallest setup that shows the bug (Running locally).
  • Read /health, then the network tab, then the log — in that order.
  • If the answer is "the test stack behaves like production", that is correct: it shares the production Firebase project.

Next: Architecture — start with the request lifecycle.