State of the Homelab 2023

I started my homelab back in June of 2021 with an old HP Proliant G6 that I fished out of a dumpster at my work. I didn't have a great router at the time so I connected it to a raspberry pi that I used as a bridge to the wifi network (You can read more if you're interested here). I didn't have much in the way of hardware and didn't know a ton about systems administration, virtualization, or networking at the time. Armed with a borderline obsessive desire to know more about technology, I've learned an exponential amount about all of those things since then and gotten newer/better hardware to run a variety of different services. This post is to give an update on the current state of my homelab (spoiler alert: no more dumpster servers) and some plans I have in the future to improve things.

Current State

I trashed my dumpster server a while back once I got some decent hardware. I've been through a few iterations of setups with my servers and networking equipment. But everything looks like this in the rack:

Networking

Ubiquiti is the backbone of my homelab. I'm running a Dream Machine Pro SE which allows for me to have up to 8 PoE (2 PoE+) devices powered by this guy. I'm running an AP 6 Lite as my main Access Point and a Unifi HD Beacon as an extender. It works well, but eventually I'd like to add another AP 6 Lite instead of the repeater, but that's not an entirely necessary upgrade so I'll wait on that. I also have a 24 Port non-PoE switch that I grabbed for a good deal off eBay. I'm using a patch panel which was my newest addition to the rack and I think makes a world of difference as far as organization and aesthetic.

Hypervisor & NAS

Proxmox is at the heart of my homelab. At any given time I'm probably running ~15VMs with a max of 20. I'm currently running it in a three node cluster that acts more like a two node cluster with an HP Elitedesk I got as a gift that acts as a third vote for quorum. I can't overstate how much I like Proxmox. Heck I almost virtualized my NAS on it as well (and I still don't think it's a bad idea, just not what I ultimately wanted). It's been rock solid for me and I can't wait to start integrating some of the more advanced features as needed.

I run a dedicated piece of hardware (an old Dell R410) as my NAS. I'm running TrueNAS Scale on it and it's been great so far as well. I really don't use many of the features it offers beyond the basics (NFS shares is about it). But it gives me a nice WebUI, supported ZFS, and allowed me to sync with my IdM solution for ldap. It doesn't have a ton of space (~2.75TB) but I'm not using it to it's full extent at the moment so that's not an issue. I'd like to more advantage of docker on TrueNAS scale, but the way docker works uses Kubernetes I think and I don't have the knowledge to run or troubleshoot that... yet.

Hardware

I mentioned some of my hardware before but I'll detail it a little more here. My Dell R410 has 6 cores and 48GB of memory. For storage it's got 3 2TB NAS drives from Western Digital that are placed in RAIDZ1. I mentioned earlier that I don't use it for much and that's true, but overall I'm happy with the performance.

My main Proxmox node is an HP Proliant DL360 G7. It's got 24 cores between the two CPUs, 152GB of RAM, and 240GB of local storage for VMs. It's a great server, but as I'll talk about in my storage section, a little tight on storage. I'm hesitant to throw too much more into the server because while it runs great, it is an old server and who knows how much longer it will reliably run.

My second Proxmox node I made with consumer parts I had lying around after upgrading my main workstation. It's got a B450M motherboard, a Ryzen 7 2700X (8cores, 16threads), and 64GB of RAM. It's got the most storage of any of my Proxmox nodes. It has a 1TB M.2 SSD and 2 more 256GB SATA SSDs in a mirrored RAIDZ configuration. It could probably handle the entire workload of my VMs by itself if I absolutely needed it to. Honestly there are times where I like this node more than my main node. While it is consumer hardware, it's more modern than my HP server. One of the bigger issues I've had with it though is graphics. The CPU doesn't have integrated graphics so I've had to use an old 660 OC graphics card in it. I'd like to setup a Jellyfin server that can do hardware transcoding but I've not had much luck passing through things from Proxmox. This graphics card also takes up a PCIe slot, restricting what I can do to increase the functionality of the server.

Finally I've got the little EliteDesk. It's not a bad little PC with 16GB of RAM, a Core i5 6th gen, and 60GB of storage. Because of it's storage restrictions, I only run one or maybe two VMs on it. I've tested out a little bit with utilizing the TrueNAS box as VM storage and it's been fine for 1 or 2 VMs that aren't used much. I'll have more to say on that below.

Future Plans

One of my favorite parts of having a homelab is to see how much it's improved over time. I like asking the question "What can I do to make this setup better?". While there are an infinite amount of things that I could do to make my homelab better I'll be focusing on three areas in particular: automation, shared storage, and backups.

Automation

When it comes to automation, ansible is one of the best engines out there. It's simple, agentless, cross platform, and a marketable skill to have in the IT industry considering it's ability to function with other automation technologies. Not to mention it's ability to work on prem and with cloud based technologies. I have big plans for using ansible to configure my servers and workstations. I already have a series started that deals with deploying an Active Directory environment using ansible. If you're interested you can view the first post in that series here. But considering I don't use Windows for any production in my homelab, I'll need to create a some more playbooks for ubuntu, rhel and arch later on.

I'd also like to start learning something like packer or terraform. To be honest I don't know much about either of them so I can't talk about which would work best for my use case. That being said I do know is that packer can run ansible as a provisioner to set up systems which sounds intriguing.

Shared Storage

Each of my Proxmox nodes have their own dedicated local VM storage, which is fine until I need to migrate a VM in order to do maintenance on one of my nodes. Proxmox will migrate, it just takes a long time depending on the VM and stops the VM during the migration. Sometimes taking a backup to my NAS and restoring is quicker so I do that from time to time as well.

The solution to this issue is shared storage. One way I could solve this is to have a dedicated box that deals with all VM storage (another NAS box as iSCSI target perhaps?). However given my current wired LAN speeds max out at 1Gb/s, this is less than ideal for having shared storage. Ideally I would upgrade to 10Gb/s on my LAN for this to work well. The problem with this is cost. Upgrading to 10Gb/s would be very expensive right now for me, at least the way that I would like to upgrade, and I don't see it in the cards for quite some time. I have a few ideas as to what I could do for a potential 'better than now' solution, and maybe for one or two VMs I could run the NFS server as it's storage over 1Gb/s without too much of an issue. I've thought about Ceph storage as well, but again that pesky 10Gb/s is recommended.

Backup Solution

Finally we've reached the backup section. As an IT professional, I know that the importance of backups cannot be overstated. But as is the case in homelabs and life I got sidetracked and/or busy with things. I admit that I have neglected my backup solution. But I'm hoping to correct that this year. I have some ideas in mind like Borg backup and proxmox backup server and who can count out good ole rsync when it comes to backups. I've also been thinking about offsite backups. I'd like to buy some cheap old workstation computers like an EliteDesk and place those at some relatives' houses as offsite backups, but that'll require some planning. For the time being I still need a more robust backup solution at my own home first

Final Thoughts

Back in 2021, I got my first IT job as a sys admin. I didn't know a ton, but was passionate and wanted to learn everything I could about how an enterprise environment worked and looked. I've learned a lot since then and have built up a homelab that I'm proud to experiment and learn with. That being said one of the best (and worst) parts of a homelab is that it could always be improved. My focuses this year revolve around automation, shared storage, and backups. Looking forward to posting more this year!