The two most recent PianoTell outages were related to disk storage issues:
The first problem was directly related to the fact that pianotell-cloudbackup was producing frequent backups of the database, storing them locally, and uploading them to the cloud. Originally I had not implemented a local retention policy, so the backups were kept indefinitely on disk.
So the fix for the first issue was to update pianotell-cloudbackup to implement a local retention policy. So far so good.
However, I didn't implement any retention policy for the cloud backup itself. Once in a while I would manually clean it up. The second outage was partly a consequence of this — when the cloud storage filled up, the backup container couldn't upload new backups, and the local disk filled up with dumps that couldn't be offloaded.
So I decided to properly solve this once and for all. Enter pianotell-offsitebackup.
This is a new, independent container that runs on a separate machine offsite. It has two responsibilities:
- Pull backups from the cloud to the offsite machine, creating a second copy of all backups (database dumps and assets).
- Enforce a tiered retention policy on the cloud backups, keeping the storage tidy without manual intervention.
The cloud retention policy keeps recent backups at full granularity, then progressively thins them out — keeping one per week, then one per month, then eventually deleting very old ones. Before deleting anything from the cloud, the script verifies that the backup exists locally — so we never lose the only copy of a file.
On the offsite machine itself, backups are kept indefinitely as long as disk space permits. If local storage ever exceeds 100 GB, a separate retention script progressively thins older backups — oldest first, stopping the moment we're back under the limit. This means we always keep the maximum number of backups that fit.
The offsite machine is my daughter's old MacBook with a broken screen, running headless via SSH. It's actually the perfect use for a machine that was otherwise going unused. 🙂
As before, the whole thing is containerized and can be deployed with a single command.
With this change, PianoTell now has a proper three-tier backup strategy:
- Local (on the server) — the most recent backups, refreshed frequently
- Cloud — tiered retention managed automatically
- Offsite (separate machine) — a full copy of everything, retained as long as space allows
No more manual cleanup, no more surprise outages from full disks. Hopefully. 🤞
