June 21st, 2022:
Homelab Outage and Recovery

Last Wednesday my AC didn’t work either because of the load buffer module (SMM) that delays turning the AC on if the generator has been activated. Normally, it keeps the AC from switching on immediately and drawing too much juice when the generator hasn’t fully started. In this case, it failed and defaulted to “open” meaning no power to the AC. Hopefully, Generac will extend warranty work because of the failed units and replaced it with a default “closed” SMM (so the AC won’t be unavailable if it fails again). I called the generator service company and they had a repairman schedule to visit the next day.

esxi vmware homelab

That repair was supposed to take “5 – 10 minutes.” The UPS on my homelab server rack is good for 15 minutes or so. Around 12 minutes into the repair, it started beeping like crazy and my whole lab shut off without powerdown. (ESXi VM server which includes media server, automation server, web server and DNS/Pihole VMs plus the SAS array of storage.) Servers don’t like when that happens. It wouldn’t post when I tried to boot. I really, really should have shut everything down in advance… but I wanted to save my over 600 days of uptime on the VM host!

After I got home from a weekend of racing, I spent several hours getting it back up and running by pulling hardware, re-installing VMware to new SD cards (it boots from internal SD cards), running diagnostics on the RAID arrays, etc. Luckily I had the DNS and Web Server VMs backed up to another server and powered that on to cover me for the weekend… but I thought I was screwed. (None of that effort contributed to the solution.)

The fix was to run the Lifecycle Manager… which is a Dell EMC server feature that walks you through server setup and maintenance. I don’t think I used it since I set up the server! It walked through the automated firmware upgrade and it reinstalled BIOS, etc. That unexpected process allowed me to post. I just had to mark all of the disks in my RAID arrays as “good, offline” and re-scan the foreign configuration and I was back online.

Lessons learned:

  1. Backups are important! I have the backup server which allowed me to get up and running on Thursday so I could keep working from home. But, I had no home automation or media server… they rely on hardware that doesn’t work in the spare server. If all else failed, I also have online “cold storage” backups for everything, but the recovery time for those can be weeks or months.
  2. Don’t get too dependent on home automation. 2 of my automated light switched have gone bad (Insteon units which I need to replace and re-configure with Z-Wave). I was able to control them with Amazon Echo commands though… if my server was online. Also, our automated cat feeder is dependent on the automation server. We were gone for a weekend and had to leave bowls of food out for them. I bet it was gone in the first 24 hours.
  3. My daughter’s birthday was on Sunday (6 years old!). While at the racetrack. she made up her mind that she wanted to watch dvd rips of the Gummi Bears when she got home. When we got home on Sunday afternoon, her world fell apart because I didn’t get the server back online yet. I’m working on setting her expectations that the world of entertainment offered by the server is a convenience and isn’t always available when she has screen time.
  4. Practice makes perfect. I haven’t had to perform any significant maintenance on my ESXi host for almost 2 years. In the future, I’ll practice more frequent roll-overs to my backup host and add a USB controller that matches the one in the primary host for automation interfaces. Then, I’ll only be without a media server if this happens again (see #3 above).

Like any other hobby, running a homelab can be a lot of work. I’ve become VERY dependent on the main VMs that run on my host. Most of the experimental hosts could be lost and I could easily start over with them to play with Docker and K8s, etc. I’ll have to make the routine maintenance part of the “fun” of my homelab, too.

- Duane

February 21st, 2020:
Fifth Generation Subaru Legacy GT Bypass Valve Upgrade

This write-up applies to 2010 – 2013 Legacy GT models with 2.5 turbocharged engine. A stock engine engine/tune is unlikely to benefit from changing you bypass valve.

Replacing the factory turbo Bypass Valve (aka blow-off valve or BOV) is far easier than I initially thought. Whether or not you should is hotly debated, but it’s really up to you. My custom tuned Cobb Stage II was leaking boost and this fixed it, but that may have been due to the age of the original more than the design. Here’s how you do it (since I couldn’t find any other overview):

Parts needed:
1.) New bypass valve. I went with the GFB Hybrid (T9233) because it allows me to use either hybrid or full recirculation easily.
2.) Spare plastic body fasteners. (You may break some during removal of the intake snorkel)
3.) Zip tie or small hose clamp (optional, see step 11)

Tools needed:
1.) 8mm socket or nut driver (snicker)
2.) 10mm (socket and ratchet)
3.) 12mm (socket and ratchet)
4.) Screwdriver or similar tool to pry fasteners and hoses
5.) Vice grips (recommended) or pliers

1.) Remove the 2 plastic fasteners for the cool air intake snorkel, then lift it out.
2.) Remove the passenger side radiator fan by disconnecting the electrical connector in the bottom corner. Then remove the 2 10mm bolts on the top. Lift the radiator fan straight up to remove.
3.) You can now easily access the Bypass Valve from above:

Factory Bypass Valve

4.) Slip the vacuum line off of the Bypass Valve.
5.) Use an 8mm socket or nut driver to loosen the upper hose clamp on the Bypass Valve.
6.) Wiggle the upper Bypass Valve tube out of the hose while gently prying the hose with a screwdriver. There’s a lot more flex in the lower hose than you expect. It will allow you to pull the upper hose off of the Bypass Valve and pull it aside to access the lower hose clamp.
7.) Use your vice grips to lock the spring-style lower hose clamp open… or try to do the same with pliers. Wiggle the lower hose off of the factory Bypass Valve.
8.) Compare the orientation of the inlet and outlet to your factory Bypass Valve. Also ensure that the vacuum line is aligned. Most aftermarket Bypass Valves allow you to rotate the vacuum line as needed if you remove a few screws. Follow the manufacturer’s instructions. The GFB Hybrid (T9233) did not require any adjustment.
9.) Insert the new Bypass Valve into the lower hose and align it with the upper hose. Remove your vice grips from the lower spring-style lower hose clamp.
10.) Insert the new Bypass Valve into the upper hose. Tighten the 8mm hose clamp so that it is snug. No need to go gorilla on it.
11.) Slip the vacuum line onto the vacuum inlet on the new Bypass Valve. A zip tie or small hose clamp may be necessary to ensure a good seal if the outer diameter of the inlet is smaller than the stock Bypass Valve.
12.) Install the passenger-side radiator fan: slide it straight down into place. Reinstall the 2 10mm bolts on the top. Re-connect the electrical connector on the lower corner of the fan.
13.) Re-install the intake snorkel and secure it with 2 plastic fasteners.
14.) If your Bypass Valve spring requires tuning, follow the instructions provided by the manufacturer. My throttle response benefited from a slightly tigher spring. It held boost without any adjustment, though.

Visibility Was Better With Fan Removed

Here’s how the hybrid sounds (not too loud) with a full Invidia exhaust:

2011 Subaru Legacy GT w/ GFB Bypass Valve Interior, Stationary
2011 Subaru Legacy GT w/ GFB Bypass Valve Exterior Rear, Stationary
2011 Subaru Legacy GT w/ GFB Bypass Valve Exterior Front, Stationary

- Duane

February 14th, 2019:
On Aging Parents, Life and Loss

I’ll be honest with you, this post is going to be challenging for me to write and for some to read. I’ll do my best if you do the same. Let’s get started:

When I last updated, I explained that we had a lot going on. As it turns out, I didn’t know the half of it. In October of 2017, both of my parents were celebrating Halloween, handing out candy and commenting on everyone’s costumes with our family in my brother’s house. They still lived in their home that my grandfather built. 15 months later… my mom has passed away and dad is alone in assisted living.

Within a month, my parents’ health began to worsen in alternating increments. The first big scare was Christmas morning (2017) when my mother was too weak to get out of bed and ended up in the hospital with a UTI. A few months later and we were taking her to urgent care because she “fell and hurt her knee”… in retrospect, she probably had a small stroke. We took her to the ER that night (another UTI, too) and decided in the ER with the whole family that it was time to find an independent living community.

We briefly pursued this direction a few years earlier, but my father was very resistant. They had too much “stuff” and weren’t ready to part with it. In hindsight, the additional care might have improved their quality of life significantly. This time, everyone was very serious about the endeavor. We kept my mom apprised of our progress while she get better in the hospital. We established a budget and timeline, then researched a few communities. After a handful of visits, we arranged a company to move them and their 2 pets. In just 3 weeks, they were living in a moderately-sized 1-bedroom apartment on the ground floor about 15 minutes from their house. At first, it seemed promising. The meal plan and care from the staff helped, but my parents continued to “take turns” in the hospital each needing care a few times over the next 3 months. The peak of this situation occurred mid-summer, 2018. My father was hospitalized (and later in long-term physical rehab) for a serious infection for almost 2 months while my mom was at the apartment alone. We maxed out the care program offered by the community, but the staff wasn’t trained to the level of care she truly needed. We didn’t realize that her dementia/Alzheimer’s had progressed to moderate. She wasn’t drinking enough water, not eating enough food and just not taking care of herself. When we visited, she put on a great show. We didn’t know how much help she really needed.

In late August, the day before my father returned to their apartment, my mom went to the ER by ambulance. She was having a dementia-related panic attack exacerbated by confusion from yet another infection. This was the first time mom looked really bad in the hospital… but she appeared to get better quickly, even joking in the hospital. Her care was extended by a stay in a rehab facility near their apartment. Eventually, she was even walking (with help) and making plans for Halloween in a few months. She was in the hospital and rehab for 5 weeks.

My mom taking Kara for a walk in post-hospital rehab.

During my mom’s recovery, our family decided to upgrade my parents’ care to assisted living with guidance from the hospital staff. We identified a handful of assisted living locations and decided to move to a community very close to their house (which were just finishing cleaning out to prepare for sale). The cost was significantly greater, but the level of care increased accordingly. The staff is amazing, their apartment is very nice (2 bedroom to accommodate all of their stuff) and their pets were welcome.

Things seemed to calm down for a few weeks, but it was obvious that mom’s dementia was worsening exponentially. In the evenings she was more confused and prone to repeatedly calling out. For years, she could use an iPad for crossword puzzles, video calls, email and web browsing… now she had trouble turning it on or figuring out how to charge it. The staff tried to monitor her fluid intake, but could only do so accurately during meals. Mom, always the social perfectionist, continued to put on a good show in the dining hall.

Mom’s plans for Halloween would never happen: on October 30th mom was back in the hospital. Their new apartment was closer to their “old neighborhood” and allowed for mom to be transported to a familiar hospital nearly around the corner from my brother’s house. We didn’t know then how much time we would be spending at that hospital, but were very glad to have the care, guidance and support of the staff at Beaumont in Grosse Pointe. This stay was different: mom was having trouble swallowing and the speech therapist wasn’t able to get her to follow instructions consistently. We discussed moving mom to the memory care area of their community for closer monitoring and therapy. She was in the hospital/rehab through November, but we were able to drive her to their community for Thanksgiving dinner. This time, mom went back to the hospital (followed by rehab) briefly before returning home.

We wanted to celebrate mom’s 80th birthday a week-or-so before Christmas, but realized that her health was very complicated and unstable… and had to call it off when she returned to the hospital 3 days before her birthday. Again, this stay in the hospital took its toll on mom. We didn’t know it, but the repeated infections weakened her to the point where she couldn’t fight off pneumonia. It just wasn’t detected with tests or examinations and she went home (against protest) instead of rehab.

A week later (Jan 2) and Mom was in the ER with respiratory failure. We all rushed to the hospital and the staff explained that she needed a breathing tube. Her kidneys weren’t working well and her (existing) heart condition was the root cause. I can still remember the minute-by-minute details, but will spare you. After a few days, her condition improved slightly and the breathing tube was removed, but her respiration never fully recovered. Her other organs were not working correctly. At the same time, my father was admitted to the hospital. With all of the stress, he wasn’t taking his medications and had to be admitted. This turned out to be a blessing: he was able to spend time with my mom.

While on the topic of blessings, climate change provided us with oddly warm weather so a smooth 50+ mile commute each morning for updates from the ICU staff was possible. The construction on ALL of the expressways between our house and the hospital wrapped up, further shortening travel time. Mom was able to recognize all of us until she became unresponsive and that time meant a lot to us.

Gina and I went home to have dinner and rest for a night after a few days at the hospital, but we got a call that mom had taken a turn. I went to the hospital and joined by brother and dad. The doctors told us that we only had a few days (at most)…

Mom’s pastor was out of the country, so our pastor and his wife visited, talked to the kids and spent time with us. We moved to a peaceful room and my brother and I stayed with mom until she passed peacefully. We cried a bit, but were grateful for the time we had with her. It was hard telling my dad, but he was able see her one more time, too. We started to make arrangements and my dad’s health improved enough to go to a nearby rehab for a few weeks. He’s home now and we’re preparing for mom’s memorial this weekend.

I never prepared for my parents to be “old” or get sick. I knew one day they would be gone and tried to brace myself, but caring for them and putting together a plan was more than I expected. The past 15 months have been a roller coaster of work, fear, worry, stress, sadness and (sometimes) relief. We should have planned better but luckily my mom had clear instructions for us to follow as her health declined. We’ve learned a lot and still have a lot to take care of, but I want to share some of the basics in case they help others now that I’ve had some time to reflect and gather my thoughts:

  • Be active. Visit your aging parents as frequently as possible. Get them out and about.
  • Don’t take anything lightly. The big things send them to the hospital, the small things are all warnings. Some of the major flags for our situation were trouble swallowing, not drinking enough fluids, missing medications, “small confusions” that ended up being UTIs and eventually dementia.
  • Be involved in their medical care and listen to the doctors. Ask questions. If possible, keep a journal and share it with their healthcare providers.
  • Know their medications. Make sure all of the meds are taken when they are supposed to be.
  • Keep track of their important documents including government-issued ID.
  • Make a budget plan. Update it frequently.
  • Don’t rush into things. Evaluate all of your options: care, housing, medical treatment, funeral arrangements, etc. At the same time, don’t postpone things unnecessarily. This seems contradictory, but there’s a balance to be found.
  • Value the time you have together and do everything you can to preserve your memories. (I’ve never been a big photo/video creator.)

Our family grieves in waves. Happy memories outnumber the difficult ones from recent months… but those memories are most fresh and both bring tears. In time, I know the hard times will fade and happy thoughts will once again take over. I crafted a pair of photo boards for my mom’s memorial service and this helped me more than I expected. We trying our best to keep dad happy; his grand-kids are a constant source of support and new, happy memories. We are working with him to improve his health so that he can more actively participate in our family events and, hopefully, outings. We’ve been dreading the future for so long. It’s nice to look forward to things again and hope for continued improvement and family time.

- Duane

August 31st, 2017:
Late Summer 2017 Update

Well, no racing for us this year. The combination of a 1-year old and new engine/new racing class didn’t really work out. We’ve been able to spend some quality time with Kara at the state and metro parks, plus a few great trips to Northport, MI. The weather has been nice, but my work schedule (especially having my home office in the Mountain time zone) has severely limited my time outside during the week. Hopefully this will ease up soon in my new-ish role (promoted back into Sr. Management again: Sr. Director).

I’ve done a lot of work cleaning up our workshop this year. I’ve meant to clean it up and prepare to insulate/heat the shop so I can install the new engine in my racecar… but it’s turned into a lot of cleanup and corrections for lack of ventilation from our new roof (circa 2009). I also spent a few hours replacing the suspension and lifting my Jeep due to a need for new tires and shocks.

I’ve been keeping up with my retro computer collection: everything is still working and I find an few hours per week to enjoy my Amigas, PCs, Macs and Commodore machines. I’m almost through my Leisure Suit Larry bucket list: wrap up the series. I’ve owned about 3 or 4 versions of the game series (or parts of it) over the years and it’s time to finally wrap it up! (I took a short break for Space Quest and Police Quest in there, too.) I added a SX-64 Executive "e;portable" to my collection earlier this year and have had my eye on few Apple IIc computers, but the price or condition haven’t been up to snuff.

Most recently, I’ve been working on capturing/digitizing a few boxes of 8mm/Hi8/DV that literally fell out of my closet a week ago. I’ve found some amazing stuff on those tapes: lots of high school wrestling, some hanging out with friends, and an emotional collection of holiday footage with many family members that have since passed. The process is relatively slow as I must capture each tape in real-time. However, my setup is surprisingly effective: a Sony Hi8 camera connected via analog S-Video and Audio (when capturing 8mm and Hi8) to my Sharp DV Camera, connected via Firewire 800 to an Apple Thunderbolt Lightning Cinema Display, connected to my 2011 Macbook Air via Thunderbolt with an external Thunderbolt drive for storage. Both cameras are time base corrected resulting in decent quality considering the source media. The capturing process is taking a long time, but it’s mostly hands-off… the real work will be sorting, editing and finalizing for DVD (all source is 480p at best). Some of the content might make it to YouTube, but I’ll likely keep it private for friends.

That sums up a lot from the past few months. Seems like each season I hope to make more regular updates to my blog, but I end up doing this instead. More to come?

- Duane

February 8th, 2017:
Lessons Learned By A New Father (Part 2)

Christmas Eve Kara and DuaneWe’re almost at the 7 month mark with Kara. Everyone said that we should take lots of pictures because time moves so quickly. It’s true! It’s hard to remember the tough parts in the first 3 months and the progress made in months 4 – 6 is incredible! Here are more lessons learned:

  1. I mentioned in my last article that it gets a lot better after week 10… that’s true! But, 3 months is like cresting a hill and seeing nothing but sunshine, green trees, and blue water. You finally feel like your effort is paying off.
  2. That bedtime routing really starts to pay off after 3 months. We got a few full nights of sleep and it was amazing. We also woke up the next morning, double checked the baby monitor, and went to her room to make sure she was ok. We couldn’t believe she slept for 11 hours!
  3. Don’t get cocky, though. Babies occasionally regress. Our first experience with a regression was also Kara’s first cold. She just started sleeping through the night (at least a few times in the same week) and then she got an upper-respiratory/head cold. She barely slept for an hour at a time. It was rough. Then, we got sick. I was afraid of coughing and waking her up, so I slept on a sofa in the basement. It took a lot of effort to get back into a routine after Kara recovered from that cold.
  4. If your baby attends daycare, they’re going to get sick. Our line of thinking is that she will have a better developed immune system eventually… but it can be tough. She’s had 3 colds in the past 4 months… each time making her fussy and depriving us all of sleep.
  5. Despite the previous note on daycare, it’s so worth it. We think that Kara is very well socialized. The schedule at daycare helps establish a rhythm that we continue on weekends and holidays and it keeps things running smoothly. She participates in a lot of activities and the staff has helped her make a number of keepsakes. Kara is genuinely happt to see the staff when we drop her off in the morning.
  6. Around 4 months or so, Kara needed less constant handling and was happier to be on her tummy to play (with supervision, of course). She also started playing in an exer-saucer… kind of a bouncy seat that helps develop leg and core muscles. We could set Kara in her exer-saucer or on a play mat within eyeshot while we cook dinner or clean.
  7. Diaper rash and dry skin… we’ve figured out the few occasions that we’ve encountered to far… but it’s probably going to happen for you too. We found that a 2-layer approach to diaper rash works best: thick coat of bag balm followed by a zinc-oxide moisture barrier. Spread that stuff on like cake frosting! The dry skin ended up being a combination of winter (dry air and furnace heat indoors) and the baby-safe liquid soap we were using. We switched soap and mom lotions Kara up with Aquaphor every night before getting into her sleeper and sleep sack… and no more blotchy dry skin.
  8. At 6 months we started to introduce real food: blended sweet peas, bananas, pears, carrots, squash, mangos, chicken w/ sweet potato, etc. Each are blended and frozen into little 1 oz. servings. We can then pop them into the microwave for 10 seconds and mix with a little nursing milk. Kara has been very excited about this new activity. It’s mostly for fun, as the majority of her calories still come from nursing.
  9. Baby isn’t the only one to enjoy new activities with real food… get ready for baby turds. I noted in my last article that diapers from nursing weren’t too bad. While not the most horrifying experience, that first turd can be quite a surpise… especially if baby just enjoyed carrots! The smells are a little worse, but honestly, if you’ve been changing baby’s diapers regularly, you’re pretty well conditioned for the task by now
  10. At 6 months, Kara is focused on things around her. She anticipates things (like the next knee bounce or kiss). She watches the door and waits for us to get her when she cries in her crib. She is more engaged with us when playing and she gets a focused, intense expression on her face when she’s trying to figure something out (like reaching for blocks, trying out a new toy, or discovering a button on her outfit).
  11. We’ve gone on a few outings together and she’s had a blast: a week ago we went to the computer store for the first time. I was pretty confident that it would go well as Kara has been going to the supermarket with us since she was a few weeks old. She generally enjoys looking at all of the colorful boxes on the shelves as we roll down the isles. The computer store was a little more challenging. We went on the observed New Year holiday… along with what seemed like every other technophile in Southeast Michigan. Kara was starting a cold, but was still an absolute angel. She wanted to get out of her carrier and have me hold her… but really wanted to see all of the exciting new things on the shelves of the store. It was a little scary… especially because we ventured out on our own. I asked my wife if she would like to see a movie while Kara and I went out. This weekend, we went to Legoland in a nearby mall with my brother and his family. It was our first “kid activity”… also a huge success, it’s given us confidence in our decision to visit Disneyworld in April!

Kara is just starting to babble. She’s holding herself up on all fours, turning around on the floor and sitting up on her own. She’s not crawling yet, but we feel like it’s right around the corner. Gina and I refer to Kara as “smooshy” for the first 3 months. She had not yet developed much of a personality, lacked eye contact, and rare smiles were like oxygen to a drowning person. At 6 months, it can be hard to remember those times when your baby smiles when she wakes up and recognizes your face, giggles when you play together, or reaches for you when she’s ready to be picked up.

Stay tuned. I’m pretty sure I’ve barely scratched the surface of this whole new dad thing. I doubt I’m right about a lot of this, but it’s working pretty well for me at the moment. Let’s see what happens in a few more months…

- Duane