Post-mortem: eleven hours of Seantral on the wrong backend
· 4 min · post-mortem, seantral, deploy, supabase
On 21 September 2026 a fallback deploy built the Seantral bundle without its backend variables. Timeline, cause, diagnosis and the guard that now stops the build.
Timeline
On 21 September 2026 CI on GitHub was stopped by an account billing problem, and the Seantral release went through the terminal fallback deploy, npm run deploy:seantral. That build started without the Seantral backend variables in the environment. For eleven hours seantral.com showed “Restricted access”, zero locations, no photos, and the statistics did not answer.
The recovery was a new build with the Seantral environment loaded, released through the same fallback channel. The guard that prevents a repeat entered the repo the same evening, at 23:42, with commit e5640f4.
Cause
The client reads the backend address and key from build variables. When they are missing it falls back to the address written in the code, which is the RiftSeed backend. Since 1 September Seantral lives on a backend of its own, and the ws_* tables no longer exist on the old one. CI knows the right variables because they sit in its secrets; the terminal does not, and the client fallback said nothing.
How it was spotted
From the app itself: the symptom was on screen. The automated probes were green, and rightly so: the uptime probe checks that seantral.com answers and that the forecast pipeline is healthy on the Seantral backend, and both things were true. Only the served bundle was talking to the wrong backend.
The diagnosis came from two checks on the served product, without reading the code. First: query the tables over REST with the public key embedded in the live bundle. The answer was PGRST205, meaning the table is absent from the schema, which is different from a denied permission. Second: read the project reference inside the served JavaScript, which was the RiftSeed one.
Fix
A new build with the Seantral variables loaded from the local environment file, and a new deploy. No data lost: the Seantral backend had never been touched, it was only unreachable from the bundle.
Ratchet added
scripts/guardia-backend-seantral.mjs is now the first step of build:seantral. The build stops if the backend address is missing, if it is not the Seantral one, or if the public key is missing, and the message says which command to run. Four tests in scripts/guardia-backend-seantral.test.ts hold the rule in place. The terminal fallback goes through the same build, hence the same guard: the mistake of 21 September now fails the build instead of going live.