NixOS Backup Server

Continuing on my side obsession with NixOS, I decided it was time to make my backup strategy for my homelab (as well as pictures, documents, etc) more robust. Currently all my Proxmox VMs, workstations, and other data is being backed up to an premises TrueNAS box hosting an NFS share. This works great for a primary backup strategy, but best practice as far as backups go follow the 3-2-1 rule. My understanding of this rule is that there should be three copies of the data (including the original) on two separate storage mediums and one of the copies of the data should be offsite.

Currently I have two copies of my data on two separate storage devices in one location so I could use some improvement. Enter NixOS backup server project. Breaking this project down into smaller goals looked like this:

1. Disk Configuration
   1. BTRFS
      1. OS Disk
      2. NFS Disk
   2. USB Auto Unencrypt
2. NixOS Configuration
   1. SOPS Nix
   2. Tailscale
   3. NFS Server
   4. Disable sleep on Lid Close
   5. Syncthing
   6. Impermenance
      1. OS Impermenance
      2. User Home Impermenance

I don't want to bore you with every single thing that I did on this - but I did want to highlight some of the cooler things I discovered. If you're interested my entire nix configurations are available on my github.

Disk Configuration

This first consideration in this project is configuring the disks. I want this project to be modular and easy to install and configure on multiple devices if desired. Luckily, I discovered the disko project by listening to the Linux Unplugged podcast (best Linux podcast out there imo). Disko allows for declarative disk configuration. This means I can write my config, run the disko tool on the machine, and have all disks, OS and NFS shares, configured with the filesystem and partitions of my liking! I can even set up encryption on the devices as well. More on that below!

USB Auto Unencrypt

Encrypting your data is always a good idea. Doing so on a remote server presents some interesting challenges considering the password prompt on reboot. However, unlocking the luks partition is possible through a permanently plugged in USB that will automatically enter the key when the prompt appears. In NixOS, you can specify this in the disko config and have the box unencrypt the drives at boot. Neat!

Impermanence

I wanted to add in something new to expand my understanding of Nix. This project introduced me to the impermanence module which allows for an (almost) complete rebuild of the server every time the server boots. I'm a big fan of this idea and very curious to see how it well it does or doesn't work. The specifics are in the nix-config located in my github, but the long and short of it is that upon disk creation I create a blank btrfs snapshot of the root partition. I then create a systemd process that restores to the blank snapshot and moves all the necessary files over from the /persist subvol - thus making my system have that "fresh OS install" feel to it.