Safer Deploys: Test Releases Before Switching
Résumé
Pixel’s announcement covers a safer deployment workflow that tests each new release before switching production traffic. It copies the live SQLite database with VACUUM INTO, starts the release on a free localhost port with outbound jobs disabled, and checks /healthz; a deliberately broken release failed preflight while the site continued serving HTTP 200.
DEV Community
Boot the Release Against a Copy of Production Before You Switch
The deploy script already had a rollback. Switch the current symlink, restart the systemd unit, poll /healthz, and if that fails, switch back and restart again. On paper that's a safe deploy. In practice, every broken release still produced an outage. The reverse proxy returned 502 from the moment the old process stopped until the rollback had brought it back — up to about 35 seconds in the worst case, and a few seconds even after the health check learned to fail fast. A post-switch health check can't prevent that. It runs after the damage, and its only job is to bound it. The fix was to move the question "does this release start?" to before the switch, and to ask it against the real data. TL;DR: Before touching current, the deploy now copies the live SQLite database with VACUUM INTO, starts the new release on a free localhost port against that copy with a throwaway config and session secret and every outbound job disabled, and waits for /healthz. If it doesn't come up, the deploy aborts with the test server's last log lines and production never notices. The first check in the preflight is whether the release is readable by the runtime group — because the preflight runs as the deploy user and would otherwise miss permission bugs entirely. A deliberately broken release was then pushed through the real pipeline: the deploy failed, the site served HTTP 200 throughout. What Does a Post-Switch Health Check Actually Protect? Less than it looks like. The sequence is: switch current to the new release restart the service — the old process is gone now poll /healthz on failure: switch back, restart, poll again Everything between step 2 and the end of step 4 is downtime if the release is broken. The health check decides whether you stay down, which is valuable, but the outage itself is already happening by the time it reports anything. What breaks a release in a small Node.js app is rarely subtle. A missing package after npm ci, a syntax error in a file the tests didn't load, a schema migration that throws on real data, a file the runtime user can't read. All of those fail within the first second of node server.js. None of them needs the release to be live to show up. They just need the release to start. How Do You Start a Second Copy Without Touching the First? Three things have to be separate from production: the port, the data, and anything the app does to the outside world. The port. Ask the kernel for one instead of picking a number: port="$(node -e 'const s=require("net").createServer().listen(0,"127.0.0.1",()=>{console.log(s.address().port);s.close()})')"
There's a tiny race between closing that socket and the app binding it, and on a single-tenant VM it doesn't matter. Binding to 127.0.0.1 matters more — the preflight instance must never be reachable from outside, not even for the few seconds it lives. The data. The app runs on SQLite in WAL mode, which means copying the .db file with cp while the app is running can produce a file that's missing whatever is still sitting in the -wal file, or one that's torn mid-write. VACUUM INTO asks SQLite itself for a consistent snapshot, WAL contents included, as a single self-contained file: vacuum_into() {
TARGET="$1" node -e '
const { DatabaseSync } = require("node:sqlite");
const db = new DatabaseSync(process.env.DB_FILE);
db.exec("VACUUM INTO \x27" + process.env.TARGET.replace(/\x27/g, "\x27\x27") + "\x27");
db.close();
'
}
It reads while the live app keeps writing, and it's the same call the deploy already used for its pre-switch backup. Using the real data rather than a fixture is the point: migrations run on startup, and a migration that works on the test schema but fails on a production row is precisely the bug a fixture can't find. The side effects. This is the part that's easy to get wrong. A second instance of the app is a second instance of everything the app does on startup — including its scheduler. This one syncs orders from an external marketplace ev...

Powered by Readybot.io