[DOWNTIME] Disk Storage Outage on 6 February 2025

For the first time since 8 June 2024, we had another unplanned outage on PianoTell, this 6 February 2025.

According to status.pianotell.com, we were down for about 29 minutes. I estimate we were down from 19:06 to 19:35 Pacific Time.

We apparently ran out of disk space. PianoTell creates backups every 15 minutes and uploads them to the cloud. Unfortunately, I never implemented logic to delete unneeded backups on the local disk.

I took manual action to delete unneeded backups and cleared up other space by rotating logs and such. The service didn't recover by itself, because I assume the database got in a bad state. Fortunately, after a safe reboot, all was well.

Separate from this, I've been investigating why our bill was €1 higher for January. I finally discovered it's because our network traffic has gone through the roof... we exceeded 20TB in January. The bad news is that this is likely not just web traffic, but most likely related to the increasing size of the cloud backups. The good news is that I can definitely optimize this.

Sorry for the downtime!

My priority for today was to mitigate and get the service back up as quickly as possible. I'll look to invest more in preserving the robustness of PianoTell over the next weeks.