Rotate DB and Redis Passwords Without Downtime

DEV Community

How to Rotate DB and Redis Passwords Without Stopping Connections

📝 Originally published (in Japanese) at forge.workstyle.tech. TL;DR The DB admin password and Redis password were stored in plaintext in the infrastructure configuration repository history. DB and Redis are services that applications connect to. I kept the history unchanged and rotated the credentials in order: Secret, DB and Redis, then application restarts. TL;DR Because the database and Redis passwords were left in the Git history, I rotated the credentials instead of rewriting the history. By updating the Secret, switching the values on the database and Redis sides, and then gradually restarting the app, I was able to make the change without interrupting connections. The object storage root credentials required a restart, so I scaled the Pods down to zero before bringing them back up. What to Decide Before Rewriting Git History Rewriting Git history changes everyone’s clones (local copies of the repository), pull requests (proposed changes), and commit IDs. History can also remain in GitHub caches and forks (separately copied repositories), so rewriting history alone may not remove all traces of an exposure. This time, we left the history unchanged and replaced the credentials. We recorded them as “replaced” in the secret scanner’s allowlist. Before starting, we prepared rollback steps and a decision threshold. In this case, the condition was: “Roll back if login fails for more than five minutes.” Deciding what state triggers a rollback in advance helps avoid delays when problems arise. How to Prepare New Passwords for the DB and Redis I generated the new passwords with Python’s secrets module (a standard library for creating secret values) and wrote them to files. I made sure not to include a trailing newline, so I wouldn’t have to type the values directly into the CLI (the screen where you enter commands) or put plaintext in SQL. For the DB change, I used a SCRAM-SHA-256 verifier (a value used for authentication instead of the password itself). I prepared the verifier and set it with ALTER ROLE … PASSWORD ''. This avoids including the plaintext password in the SQL statement. For Redis, I read the value from a file and ran redis-cli -x config set requirepass. The -x option passes the value through standard input (the input channel used to provide data to a command). In What Order Should You Switch DB and Redis? If you’re only changing passwords, neither the DB nor Redis needs to be restarted. Existing sessions can keep using their current connections. So we updated both credentials in a single maintenance window. First, we updated the Secret (a Kubernetes resource for managing credentials and other sensitive data). At that point, the Pods (the units that run the application) hadn’t been restarted, so the apps were still using the old values. We then changed the credentials on the DB and Redis before restarting the apps. Reversing this order would cause apps that had loaded the new values to try connecting to the DB or Redis while they still had the old credentials. We restarted the apps in batches of three or four. Apps that use GPUs were restarted one at a time. We also changed environment variables (env) that had been configured in plain text to use secretKeyRef (a setting that passes a Secret’s value to an environment variable). The work took place on October 13, from 00:24 to 00:54 Japan time. The 15 apps that don’t use GPUs had all been restarted by 00:27. The six GPU-using apps were restarted between 00:30 and 00:53. There were zero authentication failures and zero restarts. The number of DB connections dropped from 55 to 20–24. We monitored the systems from 00:54 to 01:04 and confirmed there were no authentication or connection errors. Before the work, we had estimated it would take 45–60 minutes. Why We Scale Object Storage to Zero Pods The object storage root was read only from the environment, so changing its value required a restart. There was no way to switch it while the app kept running, as we...

Powered by Readybot.io