Update 1: 2 June 2024

If you’re following the feature requests thread, you might know that Hosting is at the top of my priority list.

As background, PianoTell is currently hosted on FreeFlarum. This amazing service has hosted over 30K unique forums over a span of 7 years. Although many of these forums are no longer active or have moved to new hosting, this basically means that PianoTell shares resources with many other forums. Some of these forums are quite a bit bigger than PianoTell and will eventually bring things down. We’ve been very lucky so far, but it’s likely a matter of time before we experience a temporary outage or slowdowns. It’s also possible our forum will eventually outgrow the generous per-forum limits.

Incidentally, if PianoTell ever goes down, you can follow https://status.freeflarum.com

So what’s my plan? I’m containerizing PianoTell. The dream is that all I have to do is run a single command to deploy PianoTell on any server hosted anywhere.

While I grew up on Linux and open source, I’m completely alien to these modern container concepts. Fortunately, it’s been a blast to learn and refresh. It's simply stunning to witness just how pervasive Linux has become today.

This weekend I was able to containerize almost the entirety of PianoTell. Now I have two Docker containers:

pianotell-web: This is the server with PHP/nginx/Flarum. I based it on shinsenter’s PHP Docker Images. shinsenter also has Flarum images but these are just a thin layer on the PHP image. I eventually concluded I could optimize better by starting with the PHP ones and building my own PianoTell images.
pianotell-database: This is the SQL database container and I’m using the official MariaDB images for this. These images are basically perfection and didn’t need any work.

When I just started out, I thought this Docker stuff was a whole lot of hype and nonsense, but now that I can run a command and have the entirety of PianoTell running on my laptop, it’s simply magic. If only it will be so easy on the server.

The other progress I made this weekend was to identify a top leading candidate for hosting PianoTell. If it works out, we’ll be on an ARM64 server in Finland and price should be about €3.79 per month at Hetzner Cloud. Hetzner replaced my leading candidate, Namecheap, when Kevin shared that he hosts the entirety of FreeFlarum on a server there. So PianoTell will be at the same hosting company but on a new virtual server.

I have quite a bit left to do, so the next updates will be next weekend.

I have containers to build that will handle:

Cloud backup. I’m thinking I’ll use OneDriveMapper.
SMTP to handle all the outgoing emails. I have a few options here. Apparently there’s a utility to use a personal Gmail for this. Amazon SES is also a possibility. However, I think I’ll set up our own Postfix server despite there being some downsides to this.
Scheduler. This will be for maintenance tasks like certificate updates and backups.

Finally, I’ll need to get us set up on CloudFlare. I may or may not look into AWS for storage but that’s for a future time.

After all this, the new PianoTell will look like the old PianoTell. 🙂 Here it is running on my laptop:

Special thanks to @rogerch for helping with consulting, Kevin/FreeFlarum, and Daniël/Flarum.

Update 2: 7 June 2024

I'm going into the weekend with some significant tasks already accomplished. I guess I couldn't wait until I actually had time to work on this.

The main things I've addressed are:

I've found a mail solution which doesn't require the painful task of setting up a mail transfer agent (MTA) like Postfix and won't cause me to end up with a $100K bill from AWS.
- Configuring an MTA properly can be tricky and especially the encryption/authentication part can be laborious to get right. FreeFlarum has such a setup and still gets rejected by some mail providers.
- I found that I could just use an existing cloud provider like Gmail or Outlook.
- I already made the switch on Wednesday!
- That's time and money saved. Eventually I can look at an MTA if it's really needed but for now it's not a blocker.
We're on CloudFlare! Hooray!
- I definitely messed things up when I first set it up on Thursday night. forum.pianotell.com was looping infinitely. Not good, but all fixed now.
- When I switched DNS from Namecheap to CloudFlare, Namecheap decided it would no longer handle email for pianotell.com either. I fixed that by moving the MX records over to CloudFlare as well.
- We should be decently protected from bots now.
I also got an SSL certificate for *.pianotell.com from Let's Encrypt.

I figured out some interesting things:

Because we're on CloudFlare, I could have saved on the €0.50 Hetzner charge for IPV4... so monthly cost would have been €3.29 instead of €3.79. However I've got no native IPV6 at home and setting up Tunnelbroker is annoying enough that I may be painting myself in a corner.
I found rclone for cloud backups. It looks pretty awesome but unfortunately the official Docker container is silly beyond belief. I might just install rclone locally on the server but I'm motivated to find or create a saner container implementation.

As you might tell, I've been forging ahead pretty recklessly. On the other hand, there isn't a whole lot left to do!

I may switch to new hosting this weekend. If so, there will be some downtime, but I'll try to minimize it. Apologies in advance!

Update 3: 9 June 2024

I only have incremental progress to report this weekend.

I rewrote the pianotell-web Docker container.

I ditched the shinsenter/PHP + nginx container and rebased on the official PHP containers + Caddy.

The shinsenter container is great to start with as it already solves a lot of problems you’re likely to encounter when starting from scratch. I discovered that the hard way and had to read up a bit on PHP production settings and also figure out how to properly install PHP extensions on Docker.

On the other hand, shinsenter was too complex and had too many layers of scripts that got in my way. Every time I tried to change something straightforward, I had to chase down several layers of scripts to figure out why that caused something else to break.

However, the primary reason I made this change was to adopt Caddy for the web server instead of nginx. I was struggling a bit with nginx, but Caddy is such a breath of fresh air! Not only is it extremely easy to configure, it magically takes care of retrieving and updating TLS certificates on-demand.

So I solved a big problem by adopting Caddy — no need for me to implement Certbot afterall to manage the certificate. It would have have yet another thing to monitor, but Certbot is also a problem to properly test locally. I can easily test Caddy because it even issues self-signed certificates for localhost.

The other nice thing I achieved with the rewrite was to make my container and config definitions work both for my local laptop instance and for the production deployment. So on my laptop, the containers assume PianoTell is running on https://localhost/ but now the very same containers can handle https://forum.pianotell.com/ on the server. Including taking care of the certificate.

I had an assumption that I would need to build my Docker images for multiple platforms. For some reason, I thought the images I was using on my laptop were AMD64 (i.e. targeting Intel/AMD CPUs) but my chosen server runs on ARM64 (i.e. the stuff that powers many smartphones). After going down the route of enabling multi-platform builds, I realized that Docker was already using an ARM64 build of Linux locally because my Mac is on Apple silicon. That makes perfect sense, but I was confused because Linux seems to identify ARM64 as aarch64 internally and I didn’t connect the dots. So this work ended up be being unnecessary.

Finally, I added a proper healthcheck dependency on mariadb.

The backup story is a work in progress.

A long time ago, @rogerch had warned me that managing the SQL server would be the most challenging part of all this, and only now, at the end, do I understand.

There is certainly a lot to read and understand.

To over-simplify a bit, the best way for resilience here would be to have a secondary replica of the primary server that’s getting a constant stream of updates from the primary. If the primary dies, then the secondary can be made primary with a new secondary replica. The next best thing would be to have a streaming backup of every update that can be restored manually.

Finally, the simplest is to take a point in time snapshot backup and hope you never have to restore and potentially lose data given that the backup would only be periodical. There are other hacky and unreliable ways of backing up like copying the database files on the filesystem.

 It’s not an either/or situation, and there are a lot of nuances, but you get the idea.

I plan to take a periodic cloud backup to start — perhaps once every hour.

I looked at a few solutions like this one, but I wasn’t too comfortable with them for various reasons. The one I linked is quite excellent, but the client bits for MariaDB are out of date and MariaDB recommends using the same versions across the board.

So instead, I’m working on rolling my own solution based on the MariaDB official images. That way I’m in full control. I’ve created a backup user on the SQL server and I’ve figured out the mariadb-dump command that I’m going to use.

I prefer mariadb-dump instead of mariadb-backup because the backup format is human readable and very portable. It's so simple to understand.

Now I just need to add cron, rclone, and get it all working together.

This is the kind of silly stuff I find myself doing on the weekend now!:

FROM mariadb:11.1.5 AS cloudbackup-base

# we only need the MariaDB client components +
# we need cron for the scheduler and rclone for the cloud backup
RUN apt-get -y purge mariadb-server && apt -y autoremove && \
    apt-get update && apt-get -y --no-install-recommends install mariadb-client cron rclone && \
    rm -rf /var/cache/apt/archives /var/lib/apt/lists

# logic to set up the scheduler will go here
#COPY ./crontab /etc/crontab
#RUN crontab /etc/crontab
#RUN touch /var/log/cron.log

# a possibly futile attempt to reduce the final image size
FROM ubuntu:jammy
COPY --from=cloudbackup-base / /
CMD cron -f

Overall, I feel like I’m a little behind from where I wanted to be at this point.

My goals for this weekend were to deploy to the cloud and validate for a few days, but I want to focus on completing the backup part of the story before I start working on that.

Thanks for listening to my gibberish! It helps me document my decisions, where I’m at, and clarify my own understanding on the next steps!

Update: 14 June 2024

I’m going into the weekend with quite a bit accomplished. In fact, I believe I now have a minimum viable implementation of PianoTell!

On top of the pianotell-web and pianotell-database containers, I implemented pianotell-cloudbackup and pianotell-tunnel.

pianotell-cloudbackup does exactly what the name says. It backs up PianoTell to the cloud every 15 minutes if anything has changed. I believe this is a pretty robust backup story. Let me rephrase that, pianotell-cloudbackup is nothing short of pure magic! 😉
pianotell-tunnel is based on an idea from @Sophia! It allows Cloudflare to communicate with the PianoTell server over a tunnel, no matter where it lives. I deployed PianoTell on a test domain and Cloudflare served it straight from my laptop, from a private network, as if it was any other public website. Pretty awesome!

Other than pianotell-tunnel, I can also run PianoTell via normal Cloudflare proxy functionality. Although I needed to poke a hole in my firewall so that Cloudflare could reach my laptop, the proxy works great as well.

I spent several hours getting pianotell-tunnel to work. Despite that, or probably because of that, I’m more inclined to go with the simpler proxy option.

The tunnel is too closely coupled with my PianoTell implementation. I feel like PianoTell should work independently of Cloudflare with Cloudflare being an extra level of protection. I shouldn't need to deploy and configure a Cloudflare binary on my network for primary functions.
The tunnel configuration can be quite tricky. I had to chase down various settings and enable debug logs at various levels to figure out why things were not working as expected.
The tunnel seems like too much overhead, introduces the potential for bugs, and is one more thing that can fail in the system. Compare this with simply opening some ports on a firewall.
Cloudflare also creates a very strange looking CNAME DNS entry to make the tunnel work. Maybe this doesn’t matter to anyone but me, but I want the PianoTell DNS entry to look legit. 🙂
I had to force Caddy to create self-signed certificates with the tunnel, but Caddy can get legit certificates just fine in the proxy configuration.

The tunnel certainly has advantages once it’s been set up. I don’t have to worry about direct access to my server. For the proxy solution, I need to configure the firewall to only accept connections from Cloudflare.

 The are various network oddities that I had to deal with as a consequence of running Caddy in a Docker container. This makes locking down direct access to the server even more important. Fortunately, there’s a script for that.

I have this sleek Mac mini from the Intel days that is lying idle at home. I plan to set this up as an asynchronous replica database and also as a second backup site for PianoTell. I might be able to use pianotell-tunnel to help with this. This is more of a future project though and not a blocker.

All in all, I’m quite pleased with where things are at now! I have to focus a bit more on validation and then summon up some courage to take the plunge.

There would be no going back to FreeFlarum.

Update: 16 June 2024

I think this is it. PianoTell is ready to graduate to new hosting.

I created pianotell-primary-01 on Hetzner Cloud on Friday. Hetzner is beyond amazing as a hosting service and have exceeded my expectations. I’m very impressed by how fast and easy everything is. Linux on ARM64 is a dream.

I installed Docker and pulled down the new pianotell-* containers and everything just worked! Mostly.

For one, the networking is less strange than on Mac, but that meant I needed to make some changes to accommodate Cloudflare. I built a new Caddy with Cloudflare support and that’s all working now. For cloudbackup, I had to refresh some tokens, but that’s not too unexpected.

The primary issue that caused me the most distress is the security model between the host and the containers. This isn’t evident on Mac because it only hosts the containers indirectly via Linux on VM.

Without Mac in the picture, the containers appear to be much less isolated from the host than I had expected. The root user in the container appears to have many similar privileges to root in the host. So I invested quite a bit of time in terms of fixing ownerships, privileges, and such. It’s mostly an incomplete patch job, with the real solution being to run Docker in rootless mode. Unfortunately, that is exactly the kind of custom work I was trying to avoid on the host in the first place — Docker should just install that way by default.

I’ll live.

All of that was before I could even get to what I thought would be my primary task this weekend — configure the Hetzner firewall so that only Cloudflare can proxy the traffic. Fortunately, this was very easy to do.

And that’s it! I’ve been running a test version of PianoTell on Hetzner Cloud since Friday and it’s been great!

I still have to decide when I’ll make the switch. Ideally, it would have been at the beginning of the weekend rather than the end.

When ready, I will take down PianoTell for about an hour with a redirect message. This is to allow all traffic to drain and make sure I have the latest copy of the database. Then I’ll update Cloudflare so that the new server starts taking traffic.

And hopefully that will be it!

Update: 17 Jun 2024

For the first time in a long time, I'm feeling rather idle!

I don't foresee the hosting change happening before next Saturday night. In the meantime, I've been wondering how I will stop traffic to the old site and prevent traffic to the new host while things are in flux.

I've figured out a couple of things:

1) I can add a redirect in the custom header settings for Flarum. This will take care of the old site in case DNS doesn't propagate, etc.
2) I can use Cloudflare Workers to intercept traffic to the new site while I'm working on it.

Here's my idea for the status page... this is very simple HTML, but everything is being handled by Cloudflare Workers.

https://forum.pianotell.com/status

Let me know if you have any cool ideas for the status page. 😎

At least I'm getting to practice piano a bit more!

Update: 18 Jun 2024

PianoTell has officially graduated to independent hosting!

It took some work to get here, but we're finally out of beta and now at V1. 🙂

We're on Hetzner Cloud for €3.79 per month with no contract or minimum term. That's quite some value for money, less than I typically spend for a single Matcha Green Tea Latte. So if I just halt my matcha habit, I'll be saving money. 😃

With this move, I've made a couple of tentative tweaks.

Teacher Talk is now Teachers Board. This is based on feedback that TT could mean either the tech forum or the teacher forum. Sadly, we still have no teachers here. Most teachers seem to prefer to launch their own boards or groups, but I hope this can be a welcome home for them.

Meta is now Piano | Tell — Café. This was based on the feedback that Meta was an ugly name and the suggestion to name it Pub... Café just works for teetotalers as well. 🫖

If folks like these changes, I can make them permanent.

Also, a small bonus... since we now have plenty of space on the new server, I've opened up the ability for Gold Members to upload a Profile Cover in addition to the Profile Picture.

Let me know if you find any new issues or if old bugs resurface.

Next focus is on Recital #1!

Update: 2 Jul 2024

Just to interrupt with a little happy dance... 🕺

Got my first bill from Hetzner a couple of days ago... and I've got to say, I feel a deep sense of satisfaction for what can be accomplished with €2.31 these days!

PianoTell was not on Cloudflare at launch, but we have had over half a million requests to PianoTell since we onboarded and Cf started tracking. I bet there are a lot of bots even though I enabled Crawler Hints to help cut down on that, but still!

And we are close to the 200 user mark... technically the number has to hit 202 to exclude the 2 admin accounts. 😃

Update: 3 Aug 2024

Minor update this month. We've comfortably hit 200+ users, 10000+ posts, 1 million+ requests[*], and launched the first recital.

This is what the bill for a full 31-day month looks like:

We haven't had any unplanned downtime since we moved to the new hosting -- we're all green at status.pianotell.com.

I'm happy with that because all-in-all, it means this friendly and precious community is totally sustainable!

_[*] _Just _going _by _the _rough _Cloudflare _numbers.