Rotating passwords without locking yourself out

mariadbdockersecurity

Starting point

The new, strong database passwords were already prepared in .env — generated and documented, but not active. The running MariaDB instances still used the old ones. The reason: MariaDB only reads its password environment variables on the very first start with an empty data directory, and WordPress only when the container is created. A docker restart is not enough for either.

Decision

Rotation in two phases: application users first, root only afterward. root is the repair access — had it been changed in the same pass and something gone wrong, there would have been no administrative access left to fix it. And deliberately no docker restart, because an existing container does not re-read its environment variables on restart; only a real recreation counts.

Before touching anything: knowing where you stand

Two things were checked before anything changed. First, the images:

docker inspect --format='{{.Image}}' projects-mariadb-odabas-1
docker images --no-trunc | grep mariadb

The image IDs of the running containers matched the locally tagged images exactly — no docker pull had happened since creation. That mattered because an unintended version jump on MariaDB, against an existing data directory, could have been destructive.

Second, a full dump of both instances with mariadb-dump --all-databases (3.3 MB and 3.8 MB, 42 tables each, including the system tables of the mysql database), then a comparison: the password values in the container environments were compared against those in .env, via SHA-256 — not in plaintext. Container and .env matched. What the database itself would accept only showed up at the later login.

One correction against the original plan showed up right there: inside the container the variable is called MYSQL_ROOT_PASSWORD, not MARIADB_ROOT_PASSWORD as on the host side in .env.

Implementation

Phase 1 — application users. Two separate instances, two separate statements:

-- on the odabas instance
ALTER USER 'odabasuser'@'%' IDENTIFIED BY '<new_password>';
-- on the waverider instance
ALTER USER 'waverideruser'@'%' IDENTIFIED BY '<new_password>';

Then docker compose up -d, to recreate the WordPress containers with the new environment variables. In the process, without the plan calling for it, the two MariaDB containers were recreated as well — Compose does that automatically once environment variables change. For the data this was inconsequential, since it lives in the bind mount. But it explains why container environment and database had drifted apart by Phase 2.

Phase 2 — root. MYSQL_ROOT_PASSWORD only takes effect on the first start of an empty data directory. After the unplanned container recreation from Phase 1, the container environment and the database had therefore already drifted apart, and access had to go through the previously saved old values. Both databases had two root accounts each (root@localhost and root@%), rotated in a single session — after the first ALTER USER, a second login with the old password would no longer have been possible.

According to the closing note, there was a gap of about 17 seconds between the ALTER USER and the completion of docker compose up -d, with no noticed effects.

How I know it’s right

“It worked” is not verification, only hope. What was checked instead:

  • HTTP 200 on both sites at every check.
  • Manual login to both wp-admin interfaces succeeded — the login writes session data to wp_usermeta and isn’t cached, so it proves real write access.
  • The old credentials fail:
  docker exec -it mariadb-shared mariadb -u odabasuser -p odabasdb

Typing the old password: ERROR 1045 (28000): Access denied for user 'odabasuser'@'localhost' (using password: YES). The new value works.

  • A dump with the new root password from the container environment succeeds.
  • The Traefik test stack remained unaffected throughout the rotation.

The third point is the actual proof: a rotation is only proven once the old password demonstrably fails, not once the new one happens to work.

Cleanup

The temporarily saved old passwords were deleted with shred -u, the shell variables removed. The two SQL dumps remain at mode 600 in a directory at mode 700 and are excluded from the repository via *.sql in .gitignore — because of --all-databases, they contain the password hashes from mysql.user.

Takeaway

The administrative access gets rotated last, not first — it’s the only way back if something goes wrong with the rest. A planned change can trigger an unplanned one along the way, as the database containers’ recreation here did as a side effect of Phase 1 — and a checksum says more about the starting state than the assumption that everything is already in sync.