🎹 Piano | Tell — Hosting Updates

navindra · 2024-06-03T03:06:27+00:00

Update 1: 2 June 2024 If you’re following the feature requests thread, you might know that Hosting is at the top of my priority list. As background, PianoTell is currently hosted on FreeFlarum. This amazing service has hosted over 30K unique forums over a span of 7 years. Although many of these forums are no longer active or have moved to new hosting, this basically means that PianoTell shares resources with many other forums. Some of these forums are quite a bit bigger than PianoTell and will eventually bring things down. We’ve been very lucky so far, but it’s likely a matter of time before we experience a temporary outage or slowdowns. It’s also possible our forum will eventually outgrow the generous per-forum limits. Incidentally, if PianoTell ever goes down, you can follow https://status.freeflarum.com So what’s my plan? I’m containerizing PianoTell. The dream is that all I have to do is run a single command to deploy PianoTell on any server hosted anywhere. While I grew up on Linux and open source, I’m completely alien to these modern container concepts. Fortunately, it’s been a blast to learn and refresh. It's simply stunning to witness just how pervasive Linux has become today. This weekend I was able to containerize almost the entirety of PianoTell. Now I have two Docker containers: pianotell-web: This is the server with PHP/nginx/Flarum. I based it on shinsenter’s PHP Docker Images. shinsenter also has Flarum images but these are just a thin layer on the PHP image. I eventually concluded I could optimize better by starting with the PHP ones and building my own PianoTell images. pianotell-database: This is the SQL database container and I’m using the official MariaDB images for this. These images are basically perfection and didn’t need any work. When I just started out, I thought this Docker stuff was a whole lot of hype and nonsense, but now that I can run a command and have the entirety of PianoTell running on my laptop, it’s simply magic. If only it will be so easy on the server. The other progress I made this weekend was to identify a top leading candidate for hosting PianoTell. If it works out, we’ll be on an ARM64 server in Finland and price should be about €3.79 per month at Hetzner Cloud. Hetzner replaced my leading candidate, Namecheap, when Kevin shared that he hosts the entirety of FreeFlarum on a server there. So PianoTell will be at the same hosting company but on a new virtual server. I have quite a bit left to do, so the next updates will be next weekend. I have containers to build that will handle: Cloud backup. I’m thinking I’ll use OneDriveMapper. SMTP to handle all the outgoing emails. I have a few options here. Apparently there’s a utility to use a personal Gmail for this. Amazon SES is also a possibility. However, I think I’ll set up our own Postfix server despite there being some downsides to this. Scheduler. This will be for maintenance tasks like certificate updates and backups. Finally, I’ll need to get us set up on CloudFlare. I may or may not look into AWS for storage but that’s for a future time. After all this, the new PianoTell will look like the old PianoTell. 🙂 Here it is running on my laptop: Special thanks to @rogerch for helping with consulting, Kevin/FreeFlarum, and Daniël/Flarum. Update 2: 7 June 2024 I'm going into the weekend with some significant tasks already accomplished. I guess I couldn't wait until I actually had time to work on this. The main things I've addressed are: I've found a mail solution which doesn't require the painful task of setting up a mail transfer agent (MTA) like Postfix and won't cause me to end up with a $100K bill from AWS. Configuring an MTA properly can be tricky and especially the encryption/authentication part can be laborious to get right. FreeFlarum has such a setup and still gets rejected by some mail providers. I found that I could just use an existing cloud provider like Gmail or Outlook. I already made the switch on Wednesday! That's time and money saved. Eventually I can look at an MTA if it's really needed but for now it's not a blocker. We're on CloudFlare! Hooray! I definitely messed things up when I first set it up on Thursday night. forum.pianotell.com was looping infinitely. Not good, but all fixed now. When I switched DNS from Namecheap to CloudFlare, Namecheap decided it would no longer handle email for pianotell.com either. I fixed that by moving the MX records over to CloudFlare as well. We should be decently protected from bots now. I also got an SSL certificate for *.pianotell.com from Let's Encrypt. I figured out some interesting things: Because we're on CloudFlare, I could have saved on the €0.50 Hetzner charge for IPV4... so monthly cost would have been €3.29 instead of €3.79. However I've got no native IPV6 at home and setting up Tunnelbroker is annoying enough that I may be painting myself in a corner. I found rclone for cloud backups. It looks pretty awesome but unfortunately the official Docker container is silly beyond belief. I might just install rclone locally on the server but I'm motivated to find or create a saner container implementation. As you might tell, I've been forging ahead pretty recklessly. On the other hand, there isn't a whole lot left to do! I may switch to new hosting this weekend. If so, there will be some downtime, but I'll try to minimize it. Apologies in advance! Update 3: 9 June 2024 I only have incremental progress to report this weekend. I rewrote the pianotell-web Docker container. I ditched the shinsenter/PHP + nginx container and rebased on the official PHP containers + Caddy. The shinsenter container is great to start with as it already solves a lot of problems you’re likely to encounter when starting from scratch. I discovered that the hard way and had to read up a bit on PHP production settings and also figure out how to properly install PHP extensions on Docker. On the other hand, shinsenter was too complex and had too many layers of scripts that got in my way. Every time I tried to change something straightforward, I had to chase down several layers of scripts to figure out why that caused something else to break. However, the primary reason I made this change was to adopt Caddy for the web server instead of nginx. I was struggling a bit with nginx, but Caddy is such a breath of fresh air! Not only is it extremely easy to configure, it magically takes care of retrieving and updating TLS certificates on-demand. So I solved a big problem by adopting Caddy — no need for me to implement Certbot afterall to manage the certificate. It would have have yet another thing to monitor, but Certbot is also a problem to properly test locally. I can easily test Caddy because it even issues self-signed certificates for localhost. The other nice thing I achieved with the rewrite was to make my container and config definitions work both for my local laptop instance and for the production deployment. So on my laptop, the containers assume PianoTell is running on https://localhost/ but now the very same containers can handle https://forum.pianotell.com/ on the server. Including taking care of the certificate. I had an assumption that I would need to build my Docker images for multiple platforms. For some reason, I thought the images I was using on my laptop were AMD64 (i.e. targeting Intel/AMD CPUs) but my chosen server runs on ARM64 (i.e. the stuff that powers many smartphones). After going down the route of enabling multi-platform builds, I realized that Docker was already using an ARM64 build of Linux locally because my Mac is on Apple silicon. That makes perfect sense, but I was confused because Linux seems to identify ARM64 as aarch64 internally and I didn’t connect the dots. So this work ended up be being unnecessary. Finally, I added a proper healthcheck dependency on mariadb. The backup story is a work in progress. A long time ago, @rogerch had warned me that managing the SQL server would be the most challenging part of all this, and only now, at the end, do I understand. There is certainly a lot to read and understand. To over-simplify a bit, the best way for resilience here would be to have a secondary replica of the primary server that’s getting a constant stream of updates from the primary. If the primary dies, then the secondary can be made primary with a new secondary replica. The next best thing would be to have a streaming backup of every update that can be restored manually. Finally, the simplest is to take a point in time snapshot backup and hope you never have to restore and potentially lose data given that the backup would only be periodical. There are other hacky and unreliable ways of backing up like copying the database files on the filesystem. It’s not an either/or situation, and there are a lot of nuances, but you get the idea. I plan to take a periodic cloud backup to start — perhaps once every hour. I looked at a few solutions like this one, but I wasn’t too comfortable with them for various reasons. The one I linked is quite excellent, but the client bits for MariaDB are out of date and MariaDB recommends using the same versions across the board. So instead, I’m working on rolling my own solution based on the MariaDB official images. That way I’m in full control. I’ve created a backup user on the SQL server and I’ve figured out the mariadb-dump command that I’m going to use. I prefer mariadb-dump instead of mariadb-backup because the backup format is human readable and very portable. It's so simple to understand. Now I just need to add cron, rclone, and get it all working together. This is the kind of silly stuff I find myself doing on the weekend now!: FROM mariadb:11.1.5 AS cloudbackup-base # we only need the MariaDB client components + # we need cron for the scheduler and rclone for the cloud backup RUN apt-get -y purge mariadb-server && apt -y autoremove && \ apt-get update && apt-get -y --no-install-recommends install mariadb-client cron rclone && \ rm -rf /var/cache/apt/archives /var/lib/apt/lists # logic to set up the scheduler will go here #COPY ./crontab /etc/crontab #RUN crontab /etc/crontab #RUN touch /var/log/cron.log # a possibly futile attempt to reduce the final image size FROM ubuntu:jammy COPY --from=cloudbackup-base / / CMD cron -f if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } Overall, I feel like I’m a little behind from where I wanted to be at this point. My goals for this weekend were to deploy to the cloud and validate for a few days, but I want to focus on completing the backup part of the story before I start working on that. Thanks for listening to my gibberish! It helps me document my decisions, where I’m at, and clarify my own understanding on the next steps! Update: 14 June 2024 I’m going into the weekend with quite a bit accomplished. In fact, I believe I now have a minimum viable implementation of PianoTell! On top of the pianotell-web and pianotell-database containers, I implemented pianotell-cloudbackup and pianotell-tunnel. pianotell-cloudbackup does exactly what the name says. It backs up PianoTell to the cloud every 15 minutes if anything has changed. I believe this is a pretty robust backup story. Let me rephrase that, pianotell-cloudbackup is nothing short of pure magic! 😉 pianotell-tunnel is based on an idea from @Sophia! It allows Cloudflare to communicate with the PianoTell server over a tunnel, no matter where it lives. I deployed PianoTell on a test domain and Cloudflare served it straight from my laptop, from a private network, as if it was any other public website. Pretty awesome! Other than pianotell-tunnel, I can also run PianoTell via normal Cloudflare proxy functionality. Although I needed to poke a hole in my firewall so that Cloudflare could reach my laptop, the proxy works great as well. I spent several hours getting pianotell-tunnel to work. Despite that, or probably because of that, I’m more inclined to go with the simpler proxy option. The tunnel is too closely coupled with my PianoTell implementation. I feel like PianoTell should work independently of Cloudflare with Cloudflare being an extra level of protection. I shouldn't need to deploy and configure a Cloudflare binary on my network for primary functions. The tunnel configuration can be quite tricky. I had to chase down various settings and enable debug logs at various levels to figure out why things were not working as expected. The tunnel seems like too much overhead, introduces the potential for bugs, and is one more thing that can fail in the system. Compare this with simply opening some ports on a firewall. Cloudflare also creates a very strange looking CNAME DNS entry to make the tunnel work. Maybe this doesn’t matter to anyone but me, but I want the PianoTell DNS entry to look legit. 🙂 I had to force Caddy to create self-signed certificates with the tunnel, but Caddy can get legit certificates just fine in the proxy configuration. The tunnel certainly has advantages once it’s been set up. I don’t have to worry about direct access to my server. For the proxy solution, I need to configure the firewall to only accept connections from Cloudflare. The are various network oddities that I had to deal with as a consequence of running Caddy in a Docker container. This makes locking down direct access to the server even more important. Fortunately, there’s a script for that. I have this sleek Mac mini from the Intel days that is lying idle at home. I plan to set this up as an asynchronous replica database and also as a second backup site for PianoTell. I might be able to use pianotell-tunnel to help with this. This is more of a future project though and not a blocker. All in all, I’m quite pleased with where things are at now! I have to focus a bit more on validation and then summon up some courage to take the plunge. There would be no going back to FreeFlarum. Update: 16 June 2024 I think this is it. PianoTell is ready to graduate to new hosting. I created pianotell-primary-01 on Hetzner Cloud on Friday. Hetzner is beyond amazing as a hosting service and have exceeded my expectations. I’m very impressed by how fast and easy everything is. Linux on ARM64 is a dream. I installed Docker and pulled down the new pianotell-* containers and everything just worked! Mostly. For one, the networking is less strange than on Mac, but that meant I needed to make some changes to accommodate Cloudflare. I built a new Caddy with Cloudflare support and that’s all working now. For cloudbackup, I had to refresh some tokens, but that’s not too unexpected. The primary issue that caused me the most distress is the security model between the host and the containers. This isn’t evident on Mac because it only hosts the containers indirectly via Linux on VM. Without Mac in the picture, the containers appear to be much less isolated from the host than I had expected. The root user in the container appears to have many similar privileges to root in the host. So I invested quite a bit of time in terms of fixing ownerships, privileges, and such. It’s mostly an incomplete patch job, with the real solution being to run Docker in rootless mode. Unfortunately, that is exactly the kind of custom work I was trying to avoid on the host in the first place — Docker should just install that way by default. I’ll live. All of that was before I could even get to what I thought would be my primary task this weekend — configure the Hetzner firewall so that only Cloudflare can proxy the traffic. Fortunately, this was very easy to do. And that’s it! I’ve been running a test version of PianoTell on Hetzner Cloud since Friday and it’s been great! I still have to decide when I’ll make the switch. Ideally, it would have been at the beginning of the weekend rather than the end. When ready, I will take down PianoTell for about an hour with a redirect message. This is to allow all traffic to drain and make sure I have the latest copy of the database. Then I’ll update Cloudflare so that the new server starts taking traffic. And hopefully that will be it! Update: 17 Jun 2024 For the first time in a long time, I'm feeling rather idle! I don't foresee the hosting change happening before next Saturday night. In the meantime, I've been wondering how I will stop traffic to the old site and prevent traffic to the new host while things are in flux. I've figured out a couple of things: 1) I can add a redirect in the custom header settings for Flarum. This will take care of the old site in case DNS doesn't propagate, etc. 2) I can use Cloudflare Workers to intercept traffic to the new site while I'm working on it. Here's my idea for the status page... this is very simple HTML, but everything is being handled by Cloudflare Workers. https://forum.pianotell.com/status Let me know if you have any cool ideas for the status page. 😎 At least I'm getting to practice piano a bit more! Update: 18 Jun 2024 PianoTell has officially graduated to independent hosting! It took some work to get here, but we're finally out of beta and now at V1. 🙂 We're on Hetzner Cloud for €3.79 per month with no contract or minimum term. That's quite some value for money, less than I typically spend for a single Matcha Green Tea Latte. So if I just halt my matcha habit, I'll be saving money. 😃 With this move, I've made a couple of tentative tweaks. Teacher Talk is now Teachers Board. This is based on feedback that TT could mean either the tech forum or the teacher forum. Sadly, we still have no teachers here. Most teachers seem to prefer to launch their own boards or groups, but I hope this can be a welcome home for them. Meta is now Piano | Tell — Café. This was based on the feedback that Meta was an ugly name and the suggestion to name it Pub... Café just works for teetotalers as well. 🫖 If folks like these changes, I can make them permanent. Also, a small bonus... since we now have plenty of space on the new server, I've opened up the ability for Gold Members to upload a Profile Cover in addition to the Profile Picture. Let me know if you find any new issues or if old bugs resurface. Next focus is on Recital #1! Update: 2 Jul 2024 Just to interrupt with a little happy dance... 🕺 Got my first bill from Hetzner a couple of days ago... and I've got to say, I feel a deep sense of satisfaction for what can be accomplished with €2.31 these days! PianoTell was not on Cloudflare at launch, but we have had over half a million requests to PianoTell since we onboarded and Cf started tracking. I bet there are a lot of bots even though I enabled Crawler Hints to help cut down on that, but still! And we are close to the 200 user mark... technically the number has to hit 202 to exclude the 2 admin accounts. 😃 Update: 3 Aug 2024 Minor update this month. We've comfortably hit 200+ users, 10000+ posts, 1 million+ requests[*], and launched the first recital. This is what the bill for a full 31-day month looks like: We haven't had any unplanned downtime since we moved to the new hosting -- we're all green at status.pianotell.com. I'm happy with that because all-in-all, it means this friendly and precious community is totally sustainable! [*] Just going by the rough Cloudflare numbers. Update: 19 November 2025 🎹 Piano | Tell — Image hosting situation, Migrating from Imgur to Cloudflare I learned that our image host Imgur was blocked in the UK, which meant some of our members couldn't see any images on the forum. Imgur had served us well, but this wasn't acceptable. I downloaded all 3,500+ images (about 1 GB) from Imgur and migrated them to Cloudflare R2, a new S3-compatible storage provider. The new home for our images is cdn-01.pianotell.com. I also switched the upload extension so new images go directly to PianoTell instead of Imgur. After copying all the images, I went ahead and updated every Imgur link directly in the database — so all existing posts now point to the new location. The whole migration took about a week, and everything is working. Images are now served worldwide through Cloudflare's CDN. Update: 2 Mar 2026 After 2 years, Hetzner recently informed me of a price increase effective 1 Apr 2026. The reality is that it's a very different and difficult world now, with attack on all fronts, not the least of which is the emergence of AI. The costs to operate our infrastructure and to buy new hardware have both increased dramatically. Therefore, our price adjustment will affect both your existing products and new orders. The underlying causes of the increased costs are, among others, the exploding demand for AI-related computing power and for cloud services. In addition, raw material prices and production costs have also generally risen for manufacturers. The costs for RAM and SSDs especially have risen by a large amount. For example, the cost for DRAM memory has increased up to 500% since September 2025. And according to market researchers like TrendForce, this price trend will continue throughout the year. We have genuinely tried hard to optimize our costs and to prevent increasing our prices for as long as possible. But we can no longer compensate for the strain that it has placed on our operations. We want to continue to deliver quality products that meet both our standards and your expectations, so we must take this step. The price for hosting PianoTell is increasing 36% from € 3.29 to € 4.49 per month. On the other hand, the monthly cost of PianoTell is still less than the price of a Matcha Latte, so there's no cause for alarm. I also remain confident in Hetzner and that they provide some of the best value possible. There will be no changes planned or necessary as part of this. This is merely an informational update for full transparency, and part of my pledge in keeping PianoTell a sustainable service. Update: 29 Mar 2026 The two most recent PianoTell outages were related to disk storage issues: [DOWNTIME] Disk Storage Outage on 6 February 2025 [DOWNTIME] Disk Storage Outage on 22 February 2026 The first problem was directly related to the fact that pianotell-cloudbackup was producing frequent backups of the database, storing them locally, and uploading them to the cloud. Originally I had not implemented a local retention policy, so the backups were kept indefinitely on disk. So the fix for the first issue was to update pianotell-cloudbackup to implement a local retention policy. So far so good. However, I didn't implement any retention policy for the cloud backup itself. Once in a while I would manually clean it up. The second outage was partly a consequence of this — when the cloud storage filled up, the backup container couldn't upload new backups, and the local disk filled up with dumps that couldn't be offloaded. So I decided to properly solve this once and for all. Enter pianotell-offsitebackup. This is a new, independent container that runs on a separate machine offsite. It has two responsibilities: Pull backups from the cloud to the offsite machine, creating a second copy of all backups (database dumps and assets). Enforce a tiered retention policy on the cloud backups, keeping the storage tidy without manual intervention. The cloud retention policy keeps recent backups at full granularity, then progressively thins them out — keeping one per week, then one per month, then eventually deleting very old ones. Before deleting anything from the cloud, the script verifies that the backup exists locally — so we never lose the only copy of a file. On the offsite machine itself, backups are kept indefinitely as long as disk space permits. If local storage ever exceeds 100 GB, a separate retention script progressively thins older backups — oldest first, stopping the moment we're back under the limit. This means we always keep the maximum number of backups that fit. The offsite machine is my daughter's old MacBook with a broken screen, running headless via SSH. It's actually the perfect use for a machine that was otherwise going unused. 🙂 As before, the whole thing is containerized and can be deployed with a single command. With this change, PianoTell now has a proper three-tier backup strategy: Local (on the server) — the most recent backups, refreshed frequently Cloud — tiered retention managed automatically Offsite (separate machine) — a full copy of everything, retained as long as space allows No more manual cleanup, no more surprise outages from full disks. Hopefully. 🤞 Update: 5 April 2026 When I moved PianoTell to self-hosting back in mid-2024, the priority was getting the site up and running. The containers worked well, but the implementation was imperfect — config management was finicky, there was no log rotation, file permissions were too open, and most importantly, state like PHP sessions and logs wasn't safely preserved across container updates. Because of that, I ended up freezing the production web and database containers on the original 0.1 images and resorted to making all updates in-place rather than risking updates. It worked, but also meant I was accumulating tech debt — every change was a one-off fix applied directly to the running container and patches had to be applied manually. I was still diligently capturing changes in new image builds, but only deploying them to developer instances. Over the past couple of weeks I've been cleaning all of this up, getting the infrastructure into a state where container updates are safe, repeatable, and tested. Here's what changed: Security hardening — Tightened up file permissions and process isolation across all containers. Log management — Access logs now rotate automatically and persist across container updates. Session persistence — Even more important, sessions are saved too! PianoTell patches baked into the image — Patches are applied at build time instead of manually after each deployment. File mount to directory mount migration — Docker file bind mounts don't update when you edit a config file on the host. Directory mounts fix that. Cron scheduler — Replaced cron with Supercronic, which actually works properly in Docker without the need for hacks. Backup optimization — Cloud backups now exclude regenerable assets. Deployment verification tests — Automated test suite is run after every deployment. All of this to say, PianoTell has finally been upgraded to the latest container images with zero drama. Big milestone! 🎹 Ultimately, everything above is groundwork for Flarum 2, which should now be as simple as an image update. Update: 15 July 2026 PianoTell has now been upgraded from Flarum 1.8 to Flarum 2! If all went according to plan, your first reaction might be: "What changed?" That is intentional. My goal was to make this rather large upgrade as boring as possible for everyone using the forum. Accounts, posts, discussions, private discussions, drafts, uploads, profile pictures, covers, bookmarks, edit history, and the other important bits should all be where we left them. Underneath the familiar exterior, quite a lot has changed. PianoTell is now running on the newest generation of Flarum, along with an updated PHP 8.5, an updated web server, and new or upgraded versions of nearly every extension we depend on. Since Flarum 1.8 was on obsolete software dependencies, this is a much needed upgrade and gives us a firmer security stance. A few things you may notice: Dark mode is available for experimentation. Search has been enhanced. The icon set has been updated to Font Awesome 7. Controls may look or behave slightly differently. New Recital workflow starting with PIYW #10. Slight performance regressions that will be ironed out over time. Most of the differences should be subtle, but this is still a major platform upgrade. There may be small regressions or corners that I missed during testing. If something appears missing, broken, or simply odd, please report it here. A screenshot and your browser/device information would be especially helpful. If you notice any glitches or errors, try hard-refreshing the page once so your browser picks up the new files. This upgrade has been a long-running project. I’ve been documenting the technical journey over on the Flarum community forum. A tremendous amount of work went into making Flarum 2 and bringing its extensions forward, and I’m extremely grateful to all the Flarum and Friends of Flarum developers and contributors who made this possible. Hopefully this will be one of those updates where almost everything stays familiar, while quietly opening the door to new features and improvements. Let me know what you find!

navindra

rsl12 especially thank you for dark mode! Even in this clunky state, my eyes thank you!

JustinVG For example in my note taking tool Obsidian I use the theme AnuPpuccin. I don't use the contrasting headers or folders, that is too much for me. But for text markup it adds readability (for me):

I've deployed some small tweaks for now, loosely inspired by AnuPpuccin/Solaris. Hopefully it makes dark mode a little less clunky now. You can choose your preferred mode in Settings.

rsl12

navindra looks better!

« Previous Page