June 21st, 2022:
Homelab Outage and Recovery

Last Wednesday my AC didn’t work either because of the load buffer module (SMM) that delays turning the AC on if the generator has been activated. Normally, it keeps the AC from switching on immediately and drawing too much juice when the generator hasn’t fully started. In this case, it failed and defaulted to “open” meaning no power to the AC. Hopefully, Generac will extend warranty work because of the failed units and replaced it with a default “closed” SMM (so the AC won’t be unavailable if it fails again). I called the generator service company and they had a repairman schedule to visit the next day.

esxi vmware homelab

That repair was supposed to take “5 – 10 minutes.” The UPS on my homelab server rack is good for 15 minutes or so. Around 12 minutes into the repair, it started beeping like crazy and my whole lab shut off without powerdown. (ESXi VM server which includes media server, automation server, web server and DNS/Pihole VMs plus the SAS array of storage.) Servers don’t like when that happens. It wouldn’t post when I tried to boot. I really, really should have shut everything down in advance… but I wanted to save my over 600 days of uptime on the VM host!

After I got home from a weekend of racing, I spent several hours getting it back up and running by pulling hardware, re-installing VMware to new SD cards (it boots from internal SD cards), running diagnostics on the RAID arrays, etc. Luckily I had the DNS and Web Server VMs backed up to another server and powered that on to cover me for the weekend… but I thought I was screwed. (None of that effort contributed to the solution.)

The fix was to run the Lifecycle Manager… which is a Dell EMC server feature that walks you through server setup and maintenance. I don’t think I used it since I set up the server! It walked through the automated firmware upgrade and it reinstalled BIOS, etc. That unexpected process allowed me to post. I just had to mark all of the disks in my RAID arrays as “good, offline” and re-scan the foreign configuration and I was back online.

Lessons learned:

  1. Backups are important! I have the backup server which allowed me to get up and running on Thursday so I could keep working from home. But, I had no home automation or media server… they rely on hardware that doesn’t work in the spare server. If all else failed, I also have online “cold storage” backups for everything, but the recovery time for those can be weeks or months.
  2. Don’t get too dependent on home automation. 2 of my automated light switched have gone bad (Insteon units which I need to replace and re-configure with Z-Wave). I was able to control them with Amazon Echo commands though… if my server was online. Also, our automated cat feeder is dependent on the automation server. We were gone for a weekend and had to leave bowls of food out for them. I bet it was gone in the first 24 hours.
  3. My daughter’s birthday was on Sunday (6 years old!). While at the racetrack. she made up her mind that she wanted to watch dvd rips of the Gummi Bears when she got home. When we got home on Sunday afternoon, her world fell apart because I didn’t get the server back online yet. I’m working on setting her expectations that the world of entertainment offered by the server is a convenience and isn’t always available when she has screen time.
  4. Practice makes perfect. I haven’t had to perform any significant maintenance on my ESXi host for almost 2 years. In the future, I’ll practice more frequent roll-overs to my backup host and add a USB controller that matches the one in the primary host for automation interfaces. Then, I’ll only be without a media server if this happens again (see #3 above).

Like any other hobby, running a homelab can be a lot of work. I’ve become VERY dependent on the main VMs that run on my host. Most of the experimental hosts could be lost and I could easily start over with them to play with Docker and K8s, etc. I’ll have to make the routine maintenance part of the “fun” of my homelab, too.

- Duane

November 2nd, 2010:
Return to the Basement Project

After a brief hiatus from our basement project to work on larger projects for clients, Christian is going to squeeze some time in to work on hanging the ceiling and install the trim and doors in our basement. Once this is complete, we’ll be able to apply the final coats of paint, install the overhead can lighting, and get the carpet installed. If each remaining part of the project takes a week (or weekend), we should be able to schedule our projector and screen installation by the end of the year! (Pics of the project here.)

Basement Floorplan

- Duane

September 7th, 2010:
Racing, Theater Progress, and Detroit…

We’ve been exceptionally busy this summer. Somehow, we’ve managed to mix weddings, racing, home improvements into our already hectic work schedules to make updating my blog (or even posting to Twitter) a luxury. My race season has ended with 4 events in the bag (plus a practice session). I learned a lot about my new race car as well as myself. With some maintenance on the car and a little more seat time, I’m hoping to have an exciting season next year.

In other racing news, Glenn (my father-in-law) had a spectacular “racing incident” last weekend at Mid-Ohio. Another driver spun just ahead of him in a blind uphill corner. Without enough time to avoid the collision, Glenn hit the side of the other car between 60 and 80 mph, casing significant damage to his car, but no injuries (apart from a little soreness). We were very relieved to see him climb out of his car once the corner workers had gotten the remaining cars on track under control.

Christian (my brother) and his drywall crew finished the walls in our basement a few months ago. Since then, we’ve put a coat of paint on the walls (to make it easier to paint after the trim is installed), finished the remaining electrical wiring, routed and installed speaker connections, consulted with Eric at AVIO, tested the speaker locations (awesome) and built our 2nd-row riser. We’re on-track to have our theater finished by the end of the year!

Our company moved to downtown Detroit about 3 weeks ago, doubling my daily drive time. The good news is, that’s my only complaint. Detroit is great… and that’s coming from somebody that gave up on the D when the currently imprisoned mayor was re-elected. I went to school here and grew up on the edges of the city, so it’s nice to re-connect with my hometown. There’s hundreds of restaurants to visit at lunch, traffic in the city itself isn’t bad (compared to Chicago, NYC, etc.), and our new building is simply awesome. Our founder, Dan Gilbert, is trying to get more innovators and entrepreneurs downtown. If that happens, it’s only going to get better down here.

- Duane

February 17th, 2009:
As the basement project continues, the bug to build an arcade cabinet returns.

I’ve wanted to build a M.A.M.E. cabinet since the first time I downloaded the premier arcade emulation package. I just never had anywhere to put one, but thanks to my brother, Christian’s, help, we’re looking forward to a warm and functional basement space. Our new basement will be divided into 3 rooms, plus utility closets and storage: a dedicated projection theater, a walk-up bar, and a small arcade/air hockey area. George shares my excitement and we’ve been plotting our home-built arcade shrines in-between projects at work. We’ll be visiting Marvin’s Marvelous Mechanical Museum for a little recon soon. Perhaps we can learn something from his years of arcade and coin-op attraction restoration.

- Duane

February 9th, 2009:
Basement project continues. Wife demands that I do hard labor.

My brother and master builder, Christian, came over last friday to survey the electrical work we need done in the basement. He gave us a list of the stuff we needed to buy and are edging closer to completing the basement. We tried to pick up as much as possible from Home Depot yesterday, but came up short in the specialty lighting department. I’m just kidding about the demanding part. Though I admit I was pretty cranky when the time came to hang out insulated curtains in our family room. Perhaps the only chore that I dislike more than hanging curtains is painting. It went well none-the-less and we got Austin remodeling contractors to install a pet door to let our cats get to the basement without leaving the door open as well as finally fixing the broken trim on our kitchen stairs.

- Duane