How rsync Broke a Deploy and Blocked Rollback
Résumé
Pixel shares a deployment failure involving rsync -a clearing a directory’s setgid bit, leaving the app unable to read dependencies installed by npm ci. The crash triggered systemd start limits, which also blocked rollback; fixes include restoring group permissions, running reset-failed before restarts, and ending health checks when the service crashes.
DEV Community
rsync -a Stole the setgid Bit, and systemd Blocked the Rollback
The setup was deliberately boring. A small Node.js app with SQLite on a single VM, a release-directory layout with a current symlink, a CI workflow that uploads each release over SSH, runs npm ci, flips the symlink, restarts the systemd unit and waits for /healthz. If the health check fails, flip the symlink back and restart again. Every piece of that looked right in review. The first time CI ran it end to end, the deploy went red, the app went down, and the rollback — the part that exists precisely for this moment — failed as well. Neither failure was in the deploy logic. Both were in what the tools around it did by default. TL;DR: rsync -a includes -p and -g, so it copies the sender's mode and group onto the target — including onto the release directory itself, which silently clears the setgid bit the server relied on. npm ci then wrote node_modules with the deploy user's private group, and the app, running as a different user that only shares a group, couldn't read its own dependencies. The app crash-looped into systemd's StartLimitBurst, after which systemd refused every start — including the rollback's — until systemctl reset-failed. Fix: normalise group and setgid after the upload, call reset-failed before every restart, and stop waiting on a health check once the unit has visibly crashed. Why Two Users in the First Place? Because the process that writes code to the server shouldn't be the process that runs it. The app runs as an unprivileged runtime user with no sudo and no SSH key. Deploys run as a separate deploy user whose key is restricted and whose sudo rights cover exactly two commands: restarting the unit and reset-failed (why the second one is on the list is most of this post). The two share one group, and the shared directories are set up so both can write: # once, at setup
chgrp -R app-shared /srv/app
chmod -R g+rwX /srv/app
find /srv/app -type d -exec chmod g+s {} + # new files inherit the group
The setgid bit on a directory is what makes this work without either user having to think about it: every file created inside inherits the directory's group instead of the creator's primary group. Combined with umask 0007 on both sides, everything the deploy user creates is group-readable and group-writable, and everything outside the group sees nothing. That's the property the first deploy lost. What Did rsync Actually Change? The workflow creates the release directory first, then uploads into it: ssh deploy-target "mkdir -p '/srv/app/releases/${REL}'"
rsync -az ./ "deploy-target:/srv/app/releases/${REL}/"
The mkdir happens inside releases/, which has setgid, so the new directory is created correctly: shared group, setgid set. Then rsync runs, and -a is shorthand for -rlptgoD. Two of those letters do the damage: -p copies permissions from the sender. On the CI runner, the checkout directory has no setgid bit — why would it. A trailing slash on the source (./) means rsync syncs the attributes of the directory itself as well as its contents, so it chmods the freshly created release directory to the runner's mode. setgid gone. -g copies the group. A non-root receiver can only assign groups it is a member of, so rsync falls back to the deploy user's own primary group for everything it creates. Nothing about this looks wrong in a listing unless you're looking for it. The files are there, the sizes match, the owner is the deploy user as expected. The difference is one letter in the mode column and one name in the group column. The next step made it fatal. npm ci runs on the server, as the deploy user, inside a release directory that no longer has setgid. Every file in node_modules got the deploy user's private group, with umask 0007 stripping all "other" bits. The runtime user — different UID, member only of the shared group — couldn't read a single dependency. require('express') failed on the first line of server.js. Why Didn't the Rollback Save It? Because by the time the script tried to roll back, systemd had st...

Powered by Readybot.io