The day my homelab kerploded

About a week ago I had a home server catastrophe! But, this is a good story with a happy ending thanks to good backups.

First, a little history.

I have been pretty vocal about my homelab setup and in particular old enterprise laptops as affordable and power efficient home servers. My (now previous) primary homelab setup was using a Dell Precision M4700. Originally released in 2013, this thing has worked great for the last 4 years or so.

Rough overview of specs (link above has more info):

• CPU: Core i7-3740qm @2.7GHz
  • Boost up to 3.7GHz
  • 4 cores/8 threads
• 24 GB DDR3-1333
• NVIDIA GK107GLM Quadro K1000M

The Precision M4700 has an mSATA slot, a 2.5" HDD slot, and the ability to add a third HDD by removing the DVD drive and using an adapter for another 2.5" HDD. My configuration as a server used the primary 2.5" slot for root and /home, while being large enough to be a data drive for VMs.

This became important later as my self-hosted services became more important. I no longer only needed the server to manage files, I also used it as my primary “cloud” and has become an integral part of my life and for my wife who uses a handful of the services regularly.

But, as you can tell, there isn’t a lot of room for large storage disks. For that I use a Syba 4-bay HDD enclosure. This holds four 3.5" 4 TB disks and connects via USB-3.0. I then create a single mount point using mergerfs. The data speeds were good enough for my usecases. I kept some things on the primary drive to help with speed, think of things like GoToSocial, Searxng, and Nextcloud.

For the most part, speed wasn’t an issue since almost all of my services are single user or for file syncing. Even streaming media would do just fine over a USB-3.0 connection. All of this has worked great until about a week ago when my Syba enclosure just died for some unknown reason.

The catastrophe

Last week I was in the process of running updates on my homelab. I turned the Precision M4700 and the Syba off after the updates and then turned both back on, as I usually do after doing updates. The Syba DAS never came back.

When the Syba powered on it would continuously cycle through the drives as if it was doing some sort of hardware check. The lights would cycle around and around endlessly and never connect to any device. I connected it to my desktop, a laptop, and even my mini iKool R1.

Nothing.

The benefit of having a DAS (direct attached storage) is I can move it to any device. I love(d) this setup because I can move my storage array to any machine. In fact, I’ve done it several times while testing other machines for servers, or to move huge files around without using a network transfer. But it didn’t matter what device I connected the Syba to, it was borked.

Since the Syba DAS doesn’t have any operating system or a way to connect for an output of errors, I can only assume what went wrong. My conclusion is there is some sort of firmware issue that was caused when I powered the device off. It may have been in the middle of something firmware related and then I killed the power.

The solution

With my USB-3.0 DAS dead, I have no way to connect my storage array to any of the laptops (my preferred way of self-hosting) I have around. I’m fortunate to have a laptop that can hold 3 drives, but it cannot hold 4 3.5" disks

I was truly up a creek.

I was able to pop out the drives and verify the data was still good, so I just needed a PC with a case large enough to hold 4 disks, and enough SATA ports for everything. I only have one device that can accomplish that and its my primary desktop.

I’ve bounced around with my desktop setup for years since I can’t afford to invest in anything substantial. I’d love to build something that is of 2024 specs, but my budget for the entire year for any computing hardware or software is about $400. And that’s for every computer in the house and can only be spent about $40 at a time.

Then about 6 months ago my oldest daughters boyfriend gave me his old rig, which is night and day better than anything I’ve ever owned. Here’s the specs:

• AMD Ryzen 5 3500X
• 40GB DDR4-3200 RAM
• 1TB NVMe

It also just so happens to have 4 SATA ports on the motherboard.

The transfer

That night I decided I’d move my homelab to my desktop and dedicate the desktop to being the server since I have no idea when I could afford to buy any PC parts. Its the slow season for my contracting work, so spending any money is out of the question for at least another 60 days.

I have a routine that once a month I use Clonezilla to make a 1:1 image of my desktop, primary laptop, and any servers I have running. Thankfully I had just completed one, so setting up the server was a snap. Believe it or not, I was done transferring my server in about an hour.

1. I pulled the drives out of the Syba and connected in the desktop case.
2. I restored my Clonezilla image to the NVMe.
3. Booted into a live gparted environment and resized the partitions.
4. Booted into an Ubuntu live environment, mounted the primary OS drive, and edited my fstab and interfaces settings to match the new devices and mount points.
5. Booted the server and it was if nothing happened.

Sucess! It has been running from the new setup perfectly for the last week. In fact, because of the better hardware, it is running better than ever.

Now my primary desktop

I am now using the Precision M4700 as my primary desktop (as seen in the pic here) My configuration as a desktop uses the mSATA slot for root, the primary 2.5" slot for /home, and a third data drive for misc. I did so I can use a small 64 GB mSATA drive for root and then can use any HDD size for /home.

It chugs occasionally, but for the most part it works as expected. Not bad for an 11 year old laptop.

Closing thoughts

I am happy with the setup and that I was able to get everything back together so quickly. I am extremely happy with my backup strategy, which was the real test in this ordeal. Having clones of everything makes restoring so much quicker. Yes, if I wanted to learn Ansible or Terraform, I could automate backups. Yes, I could use a declarative OS like Nix.

But I don’t want to learn those things and I don’t have to. For what its worth, I’ve heard people say they use some or all of those things and still have all kinds of troubles restoring broken systems. This is mostly because they spend all their time fiddling with their automation instead of just getting down to work. My solution is having good copies of everything that I verify at time of backup and have great notes on all my systems.

Notes are important to me not just for computers. I’m an avid user of the GTD methodology from David Allen for all things in my life. I never try to remember anything, it all goes in calendars, tasks, and notes. Then I can recall it at any time.

For as much that went right with restoring my server I am sad about what I lost. I lost a great DAS that made working on things fun. I’m sad I lost the best desktop I’ve ever had and I don’t know when I can get my hands on anything similar.

Which, final thought, is why I’m falling out of love with most of my tech feeds. In particular the Self Hosted Show. I love Chris and I’m sure Alex is a fantastic guy. But Alex lives in a different income bracket than me. In fact it is several income brackets above me. I know for many tech people they would just throw money at the problem I had and buy either new server or desktop hardware. It is not interesting to me to hear someone talk about how they just throw money at their homelab or home network, especially when I will never have that option. It makes me feel bad about myself, the life I have, and what I have built. I’m not going to allow that any more.

I’m poor and I’m happy with my tech life, even if its old and slow.

- - - - -

Did you like this post? Give it an upvote by clicking on the arrows below! Sending me an upvote is like you and I giving each other a high five.

🙏 😎

If you enjoy the random stuff I write here, post to Mastodon, or watch on YouTube, and are feeling generous, I am open to tips of Ko-fi.