🎹 Piano | Tell — Hosting Updates

navindra · 2024-06-03T03:06:27+00:00

Update 1: 2 June 2024 If you’re following the feature requests thread, you might know that Hosting is at the top of my priority list. As background, PianoT...

twocats

navindra I'm glad you're finding this fun and challenging! 🙂

Sophia

Nice update! As for CloudFlare, you might want to look into utilizing their tunnel solution. It eliminates any need for ssl certificates or open ports. It can be run in a Docker container as well.

I changed our NAS that runs Plex etc and it's eliminated all intruder attempts because CloudFlare seems to catch m all 😃

navindra

Sophia As for CloudFlare, you might want to look into utilizing their tunnel solution. It eliminates any need for ssl certificates or open ports. It can be run in a Docker container as well.

Wow! I did not know about that option and at first I was confused. 😃

It's very interesting indeed. I need to research a bit into the pros and cons of CloudFlare Tunnel vs CloudFlare Proxy, but this is very cool.

Thanks a lot!

Sophia

Yeah it's like magic... it is almost as if you are running TailScale or WireGuard but without the need for clients... best of all worlds! The Docker container is as simple as

#
# CloudFlare - Zero Trust Cloudflare tunnel
#
  tunnel:
    image: cloudflare/cloudflared:latest
    container_name: cloudflare-tunnel
    restart: unless-stopped
    command: tunnel run
    environment: 
      - TUNNEL_TOKEN=yourtokenhere
    networks:
      - yournetwork

Then all you need to do is point your CloudFlare domain (or subdomains, if you plan on adding more sites) to the corresponding port. Done!

Actually I only found out about this because I thought our ISP doesn't allow port 80/443 and I wanted to run a site. AFTER I set up the CF tunnel on our NAS, I learned that we are allowed any port now. Still... we haven't had a single intruder attempt since, so we're not complaining! 😃

rogerch

Sophia Very cool!

navindra

Sophia This sent me down some interesting investigation paths.

I realized that I could probably host the entirety of PianoTell on a home server safely using this CloudFlare tunnel. Of course, I wouldn't do that for various reasons, but interesting thought.
Even more interesting, I realized that an asynchronous replica setup was very much doable via the tunnel idea.

So even if I don't host the live version of PianoTell at home, I can still host the replica database (on a Raspberry Pi with Docker even) using similar tunneling technology from Tailscale. Of course, the replica can actually be anywhere, like on some cloud service where I have free credits... the possibilities are intriguing.

Thanks very much for this idea!

Sophia

Yes, that's one of the beauties... no need to run any service like No-IP even if you have a dynamic IP. And as you said, as long as you have a replica, switching to it is as easy as disabling the CF container on the one server and enabling it on the other. Not that I have done that, but in theory that should work - and it should make maintaining your server that much easier 🙂

I saw that Tailscale has a similar service now, but I think CF is a little more forgiving when it comes to resource hungry applications. Plus it's so much easier to set up - no need to fuss with Caddy or LetsEncrypt etc. No need for a reverse proxy like Nginx or Traefik either. Apparently, you can even use it with SSH, but I would imagine that won't be possible with your VPS or you'd be locked out the moment something goes wrong 🙂

Please keep us updated, I'm very interested 😃

navindra

I’m going into the weekend with quite a bit accomplished. In fact, I believe I now have a minimum viable implementation of PianoTell!

On top of the pianotell-web and pianotell-database containers, I implemented pianotell-cloudbackup and pianotell-tunnel.

pianotell-cloudbackup does exactly what the name says. It backs up PianoTell to the cloud every 15 minutes if anything has changed. I believe this is a pretty robust backup story. Let me rephrase that, pianotell-cloudbackup is nothing short of pure magic! 😉
pianotell-tunnel is based on an idea from @Sophia! It allows Cloudflare to communicate with the PianoTell server over a tunnel, no matter where it lives. I deployed PianoTell on a test domain and Cloudflare served it straight from my laptop, from a private network, as if it was any other public website. Pretty awesome!

Other than pianotell-tunnel, I can also run PianoTell via normal Cloudflare proxy functionality. Although I needed to poke a hole in my firewall so that Cloudflare could reach my laptop, the proxy works great as well.

I spent several hours getting pianotell-tunnel to work. Despite that, or probably because of that, I’m more inclined to go with the simpler proxy option.

The tunnel is too closely coupled with my PianoTell implementation. I feel like PianoTell should work independently of Cloudflare with Cloudflare being an extra level of protection. I shouldn't need to deploy and configure a Cloudflare binary on my network for primary functions.
The tunnel configuration can be quite tricky. I had to chase down various settings and enable debug logs at various levels to figure out why things were not working as expected.
The tunnel seems like too much overhead, introduces the potential for bugs, and is one more thing that can fail in the system. Compare this with simply opening some ports on a firewall.
Cloudflare also creates a very strange looking CNAME DNS entry to make the tunnel work. Maybe this doesn’t matter to anyone but me, but I want the PianoTell DNS entry to look legit. 🙂
I had to force Caddy to create self-signed certificates with the tunnel, but Caddy can get legit certificates just fine in the proxy configuration.

The tunnel certainly has advantages once it’s been set up. I don’t have to worry about direct access to my server. For the proxy solution, I need to configure the firewall to only accept connections from Cloudflare.

 The are various network oddities that I had to deal with as a consequence of running Caddy in a Docker container. This makes locking down direct access to the server even more important. Fortunately, there’s a script for that.

I have this sleek Mac mini from the Intel days that is lying idle at home. I plan to set this up as an asynchronous replica database and also as a second backup site for PianoTell. I might be able to use pianotell-tunnel to help with this. This is more of a future project though and not a blocker.

All in all, I’m quite pleased with where things are at now! I have to focus a bit more on validation and then summon up some courage to take the plunge.

There would be no going back to FreeFlarum.

Pallas

navindra Amazing! That all sounds so intricate and interesting, and I'm so grateful that you're doing this work. Thank you!

Sophia

I'm very excited about the backup solution! Congratulations on everything you accomplished in such a short time 😊

navindra

I think this is it. PianoTell is ready to graduate to new hosting.

I created pianotell-primary-01 on Hetzner Cloud on Friday. Hetzner is beyond amazing as a hosting service and have exceeded my expectations. I’m very impressed by how fast and easy everything is. Linux on ARM64 is a dream.

I installed Docker and pulled down the new pianotell-* containers and everything just worked! Mostly.

For one, the networking is less strange than on Mac, but that meant I needed to make some changes to accommodate Cloudflare. I built a new Caddy with Cloudflare support and that’s all working now. For cloudbackup, I had to refresh some tokens, but that’s not too unexpected.

The primary issue that caused me the most distress is the security model between the host and the containers. This isn’t evident on Mac because it only hosts the containers indirectly via Linux on VM.

Without Mac in the picture, the containers appear to be much less isolated from the host than I had expected. The root user in the container appears to have many similar privileges to root in the host. So I invested quite a bit of time in terms of fixing ownerships, privileges, and such. It’s mostly an incomplete patch job, with the real solution being to run Docker in rootless mode. Unfortunately, that is exactly the kind of custom work I was trying to avoid on the host in the first place — Docker should just install that way by default.

I’ll live.

All of that was before I could even get to what I thought would be my primary task this weekend — configure the Hetzner firewall so that only Cloudflare can proxy the traffic. Fortunately, this was very easy to do.

And that’s it! I’ve been running a test version of PianoTell on Hetzner Cloud since Friday and it’s been great!

I still have to decide when I’ll make the switch. Ideally, it would have been at the beginning of the weekend rather than the end.

When ready, I will take down PianoTell for about an hour with a redirect message. This is to allow all traffic to drain and make sure I have the latest copy of the database. Then I’ll update Cloudflare so that the new server starts taking traffic.

And hopefully that will be it!

Sophia

navindra configure the Hetzner firewall so that only Cloudflare can proxy the traffic

Yay! My biggest paranoia is break-ins, so I'm really pleased you found a way 🙂 Sorry you had the run around with the tunnel suggestion, but then again one can never learn too much, right?

navindra

Sophia It was very interesting and informative for sure!

I still need a tunnel solution though, because I'm way less inclined to open the MariaDB port outside of the private network. So for this, I'll either need Cloudflare Access or Tailscale. That way I can have an asynchronous SQL replica running as well.

navindra

For the first time in a long time, I'm feeling rather idle!

I don't foresee the hosting change happening before next Saturday night. In the meantime, I've been wondering how I will stop traffic to the old site and prevent traffic to the new host while things are in flux.

I've figured out a couple of things:

1) I can add a redirect in the custom header settings for Flarum. This will take care of the old site in case DNS doesn't propagate, etc.
2) I can use Cloudflare Workers to intercept traffic to the new site while I'm working on it.

Here's my idea for the status page... this is very simple HTML, but everything is being handled by Cloudflare Workers.

https://forum.pianotell.com/status

Let me know if you have any cool ideas for the status page. 😎

At least I'm getting to practice piano a bit more!

rogerch

navindra Thanks for doing all this work Navindra! The status page looks good. Nice idea to give us some cool videos to watch while we wait for the transition to complete!

Happy practicing!

navindra

Just to interrupt with a little happy dance... 🕺

Got my first bill from Hetzner a couple of days ago... and I've got to say, I feel a deep sense of satisfaction for what can be accomplished with €2.31 these days!

PianoTell was not on Cloudflare at launch, but we have had over half a million requests to PianoTell since we onboarded and Cf started tracking. I bet there are a lot of bots even though I enabled Crawler Hints to help cut down on that, but still!

And we are close to the 200 user mark... technically the number has to hit 202 to exclude the 2 admin accounts. 😃

twocats

navindra I feel a deep sense of satisfaction for what can be accomplished with €2.31 these days!

That's crazy, how can it be so cheap?!

navindra And we are close to the 200 user mark

Woohoo!!!

rogerch

Awesome! Thanks Navindra!

Sophia

twocats Those are large servers that rent out a tiny chunk of it. They are usually a little oversold so they are banking on customers not using it to full capacity at all times.

Agree they're pretty great considering the price!

twocats

Sophia I'm just surprised that it's not at least 5 Euros a month 🙂

navindra

Sophia Those are large servers that rent out a tiny chunk of it. They are usually a little oversold so they are banking on customers not using it to full capacity at all times.

Clever. Fortunately we're nowhere near stretching out the virtual CPU capacity we've been rented out!

twocats It's close to that per month... but no problem, all we would have to do is double the number of ads to cover it. 😉

« Previous Page Next Page »