Adam's blog: Stopped self-hosting everything

29 Dec 2024, 1961 words

Self-hosting everything is fun – at first. After a while and a few days of downtime, cloud starts to seem more shiny than before. This is why I decided to move from self-hosting everything to a more balanced approach.

The big three categories of services that I used to self-host are email, all websites and VPN that connected my home lab with VPSs. As you can probably imagine, there is a different kind of nerves when one of the things enters its downtime. You need to always know that everything is working properly, because if all your mail servers are down for longer period of time, you could miss something important (such as a university assignment, what a horror!). My personal breaking point was when I was hit with a two-week outage on one of my VPS due to the provider having storage issues right at the time I was supposed to give a lecture from a website hosted on it. This does not mean that I have stopped self-hosting everything, just that I now better judge what is worth my time and nerves versus what for what I am willing to give a few bucks to someone to deal with the struggle for me.

For better orientation, I have divided my services into three main categories: Email, VPN and websites. For each of them, I am first going to briefly describe what was my approach in the past and what I am using now:

Email

The first rule of email is that you always have at least two servers. I had a main Linux VPS with Postfix that taken care of receiving and sending email with IMAP access through Dovecot. Then, I had a secondary Linux VPS also with Postfix, but this secondary VPS just saved everything that it received locally and waited for a cron job from the primary server to take it for ingestion. In the unlikely situation that both my VPSes were down at the same time, I had a tertiary MX record pointing at JunkEmailFilter’s MX Backup service. This service promised to keep all received mails and forward them to the primary email once you sort your outage out. Great stuff! Saved me a couple of times during the years. Even though I have nailed the receiving part of email most of the time, there were still hiccups, e.g. when somebody sent me a large file as an attachment. What is worse is sending your email from your own IP address – nowadays, all major mail providers are using block lists, and they are sending new IP addresses to SPAM folder by default (to combat SPAM, which makes sense). For the several years I was running my self-hosted mail provider, sending mail to @gmail.com address was always a roll of a dice.

Where have I moved instead? To Proton mail. I have purchased their 2-year plan during the Black Friday sale and haven’t regretted it since. I really like Proton because of their privacy focus. Plus, it is easy to add custom domain(s) together with catchall addresses and to set up some basic Sieve-like filters (the only thing missing is filters by mail content, which in the context of Proton makes sense). The only inconvenience is that I need to self-host a Proton mail bridge in order to be able to send and read my mail from Thunderbird.

However, I operate more domains than my Proton plan would allow for, so I cannot migrate everything over there. For my other domains, I am using two services:

Purelymail offers a pricing based on how many mails you have stored or sent, so when I keep forwarding everything to my Proton mail account and only occasionally send a mail, the overall price per year for multiple domains is far less than $10 per year for me. Unbeatable. Except for that, I am often using DuckDuckGo email protection for throwaway accounts or SimpleLogin for ones I want to use more often, but still keep my real address private.

VPN

As I have mentioned in a past article, there are multiple options how to access applications running on LAN from whole Internet. In the past, I have opted to use a setup with one central VPS that had a public IP address. This VPS had also running WireGuard, to which all my LAN-only servers were connected. Then, by using HAProxy on the VPS, I have decided which domain should be forwarded where. A second layer is that I want my LAN to be accessible from my parent’s LAN and vice versa. Because I am running MikroTik routers everywhere, I have again used the WireGuard on the central VPS to connect all the networks together. Then, when the routers could see each other, I have set up a second WireGuard layer just between them and created relevant forwarding rules for IP ranges. As you can probably already see, the main problem is the VPS being a single point of failure and often also a bottleneck because of it limited 100 Mbit/s bandwidth.

Instead, I have mainly migrated to three different solutions:

I opted not to use the ZeroTier directly for connecting networks because I did not want ZeroTier to be able to see all traffic that goes between my LANs. Instead, the additional WireGuard layer provides post-quantum security, while also giving me full access control on every endpoint. However, if I opted to use the ZeroTier directly, I would not have to purchase the two additional VPSes (one primary and one secondary) to connect to my LAN from WAN, I would only need the ZeroTier application.

Web apps

Static pages, such as this blog, are all hosted through Cloudflare Pages. Because the pages can be build through CI/CD directly from GitHub repositories, all I have to provide is the command to build the pages, set the alias and then forget about it. This reduces load on my servers while increasing uptime (no single point of failure on my side!).

Dynamic pages, such as APIs and alike, are running on my VPSes or local servers, which have cloudflared installed. Then, I route relevant subdomains to relevant ports of docker containers on the target machines. This gives me the advantage of not having to configure reverse proxies and also automatically generates TLS certificates for me. Furthermore, I can secure my applications behind SSO through Google or Microsoft or similar if I want to prevent whole Internet reaching my application without proper authentication.

Another option that I use is to use the Cloudflare Access login as a source of truth which I use within my project as an OIDC identity provider, so you can log-in with your Google/Microsoft/w/e account into your locally hosted Nextcloud or Gitea.

Conclusion

In short, even though I still fully support anybody who wants to try to self-host anything and everything, I have decided to move away from it. It was a great learning exercise and gave me much understanding of the inner workings of many of the applications, but in the end it because almost a full-time job to keep everything running and caused problems when the bottlenecks hit. Now, with my mixed solution, I can sleep much better because I know that even when some parts of my network are offline, some services are still running and a good chance is that I won’t need to fix them first thing in the morning.

DNS Bonus

When you are on your LAN you may want to use the LAN-only IP addresses of your servers for faster access, or if you just want to simply block known malicious domains, you may want to overwrite local DNS records. In the past, I have used dnscrypt-proxy as a self-hosted DNS server. Now, I have migrated to setting custom DNS rules directly inside my MikroTik routers. As this task needs to be applied periodically for multiple routers, I have created a simple script that is able to take a hosts file and set the same records inside the MikroTik routers:

#!/usr/bin/env python3
import ipaddress
from pathlib import Path
from sys import argv
from re import sub
def is_valid_ipv4_address(ip_string: str) -> bool:
    try:
        ipaddress.IPv4Address(ip_string)
        return True
    except ipaddress.AddressValueError:
        return False
def conver_hosts_line(line: str) -> str:
    line = line.replace('"', '')
    line = sub('\s+', ' ', line)
    line = line + ' '
    try:
        domain, target, _ = line.split(' ', 3)
    except ValueError:
        return None
    matching_rule = f'name="{domain}"'
    if '*' in domain:
        domain = domain.replace('.', r'\\.').replace('*', '.*')
        matching_rule = f'regex="{domain}"'
    if is_valid_ipv4_address(target):
        return f'/ip/dns/static/add type=A {matching_rule} address="{target}" comment="autogen" ttl=30'
    else:
        return f'/ip/dns/static/add type=CNAME {matching_rule} cname="{target}" comment="autogen" ttl=30'
def main() -> None:
    file_in = Path(argv[1])
    assert file_in.exists()
    print('/ip dns static remove [find comment="autogen"]')
    with file_in.open('r') as f:
        for line in f.read().split('\n'):
            line = conver_hosts_line(line)
            if not line:
                continue
            print(line)
main()

Then you can simply take your hosts file and use the script in bash: ./hosts-to-routeros.py hosts.txt | ssh -T mikrotik-router. It will remove all previously autogenerated rules and replace them with the new content. An example of the hosts.txt file may be:

immich.example.com        10.0.0.2
my-server                 10.0.0.3
nextcloud.example.com     my-server
*.example.org             my-server
homeassistant.example.com external.example.com

As you can see, it supports both wildcards and aliases. Furthermore, you can then use DoH inside your MikroTik router for more private resolving or use some WireGuard-based VPN with in-build malware and adblock such as ProtonVPN as your DNS server.

Discuss on Mastodon and Bluesky