State of my Homelab: 2024H1, Part 1

But, why?

Having a dedicated environment to tinker and experiment with random ideas is always nice. Many people do this using their local environments (laptops or workstations), but these setups have limitations:

1. Long running jobs can be a pain to setup, especially on a computer that is setup for multiple use cases (eg. dev in the day, game in the evening).
2. Experimenting with clusters (multiple nodes) using VMs can get impractical/overwhelming for a single machine.
3. Local environments have to be functional at all times, so you cannot do risky changes which might break the environment.

In addition, having separate, always-available compute and storage besides your main dev machine can have the following advantages:

1. Ability to self host services, like the following:
   • Remote, 1-click-setup coding environments
   • Password manager
   • Github actions runner for CI/CD
   • Photo library
   • Local LLM
2. Local, fast-access network attached storage (NAS) for backups, large datasets, media storage etc.
3. Dev, staging (or even prod, if you are ambitious enough) environments for personal projects.

All of these without the worry of paying $$ to host things in the cloud. According to my estimates, the monthly cost (mainly electricity) of running these machines is 10-20x lower than the same infra in any cloud env.

Here’s a glimpse of a few services I am running as of this moment.

In this post, I will detail out the hardware components which make up the lab. In a followup, I will detail out the software bits.

Most of the software and some documentation lives in an IaC repo here: shikharbhardwaj/infra.

Hardware overview

Build photos

Compute

3 nodes in a Proxmox (VM hypervisor) cluster. Specs:

Name	Overview	CPU	RAM	Local SSD storage
arete	Dell 5080 Micro	i5-10500T, 6C/12T	32 GB	500 GB
phoenix	Lenovo M900 Tiny	i5-6500T, 4C/4T	16 GB	256 GB
thor	Custom PC	R7-3700X, 8C/16T	64 GB	256 GB

So a total of 32vCPUs, 112GB of RAM available.

Storage

The NAS is hosted as a Proxmox VM running TrueNAS. Overall usable capacity of ~10 TB, powered by a pool consisting of 2 mirrored vdevs with the following disks.

Model	Capacity	Count
WD DC HC510	10TB	2
Seagate BarraCuda	1TB	3

The Seagate drives are a leftover from a previous version of this storage array, which I keep running because why not.

Networking

The main device is a TP-Link ER-605, which accepts WAN from two ISPs and handles failover. There’s a 2.5GbE unmanaged switch which connects to other devices (like my workstation), but everything is still running 1GbE right now. Plan is to upgrade this in the future.

Everything runs on IPv4 right now, although both the ISPs do provide some form of IPv6 support AFAICT. Another item to tackle in the future.

For accessing services outside of the home network, I use Tailscale. It has been an excellent service in my experience, requiring minimal effort to setup a Wireguard based VPN containing all my servers/VMs which can be accessed from anywhere in the world over regular internet.

However, CG-NAT makes it hard for different nodes to obtain a direct connection which can be problematic in case something needs high-bandwidth/low-latency connections. If a direct connection cannot be established, it has to go via a DERP relay which is highly limited in terms of bandwidth.

Here are some bandwidth numbers for a direct vs relayed connection (local connection is 1GbE):

Connection	Bandwidth
Direct (LAN)	931 Mbits/sec
Direct (Tailscale, same LAN)	883 Mbits/sec
Direct (Tailscale, across ISP)	51 Mbits/sec
Relayed (On mobile internet)	210 Kbits/sec

To avoid this slowdown, I have a cheap (~10$/mo) Digitalocean VPS which acts as a relay for services which need the fast access. It also doubles as an external uptime monitoring agent, along with hosting some other observability and authentication related services.

I deliberately keep the internet bits of my home separate from my homelab, as I am not very keen to make internet access for my family depend on my experimental setup at this point.

Power

Since the hardware here is low power enough, everything is powered from a single socket, via a cheap 1kVA UPS that provides a few minutes of backup, which has been enough so far to ride the small blips in power that we see here.

I use a Shelly Plus 1PM to keep an eye on the power metrics used by the entire system. This includes my workstation as well, so the cost/energy numbers shown below include that as well.

This dashboard is created from data scraped from the plug using its API via a simple Python script checked into the IaC repository here.

The homelab itself uses around 150W of baseline power, which is not too low but the power bill is very justifiable for the amount of compute and storage available.

Next up

This wraps up the hardware overview of the homelab. In the next post, I will go over the software components of the lab.