From the field...

For What It's Worth...

April 11th, 2024:
Pets Page Updated

After a very long break in updates, I’ve made a few changes to my lab and servers. (More on that later…)

In the meantime, I’ve updated our pets page.

- Duane

June 21st, 2022:
Homelab Outage and Recovery

Last Wednesday my AC didn’t work either because of the load buffer module (SMM) that delays turning the AC on if the generator has been activated. Normally, it keeps the AC from switching on immediately and drawing too much juice when the generator hasn’t fully started. In this case, it failed and defaulted to “open” meaning no power to the AC. Hopefully, Generac will extend warranty work because of the failed units and replaced it with a default “closed” SMM (so the AC won’t be unavailable if it fails again). I called the generator service company and they had a repairman schedule to visit the next day.

esxi vmware homelab

That repair was supposed to take “5 – 10 minutes.” The UPS on my homelab server rack is good for 15 minutes or so. Around 12 minutes into the repair, it started beeping like crazy and my whole lab shut off without powerdown. (ESXi VM server which includes media server, automation server, web server and DNS/Pihole VMs plus the SAS array of storage.) Servers don’t like when that happens. It wouldn’t post when I tried to boot. I really, really should have shut everything down in advance… but I wanted to save my over 600 days of uptime on the VM host!

After I got home from a weekend of racing, I spent several hours getting it back up and running by pulling hardware, re-installing VMware to new SD cards (it boots from internal SD cards), running diagnostics on the RAID arrays, etc. Luckily I had the DNS and Web Server VMs backed up to another server and powered that on to cover me for the weekend… but I thought I was screwed. (None of that effort contributed to the solution.)

The fix was to run the Lifecycle Manager… which is a Dell EMC server feature that walks you through server setup and maintenance. I don’t think I used it since I set up the server! It walked through the automated firmware upgrade and it reinstalled BIOS, etc. That unexpected process allowed me to post. I just had to mark all of the disks in my RAID arrays as “good, offline” and re-scan the foreign configuration and I was back online.

Lessons learned:

  1. Backups are important! I have the backup server which allowed me to get up and running on Thursday so I could keep working from home. But, I had no home automation or media server… they rely on hardware that doesn’t work in the spare server. If all else failed, I also have online “cold storage” backups for everything, but the recovery time for those can be weeks or months.
  2. Don’t get too dependent on home automation. 2 of my automated light switched have gone bad (Insteon units which I need to replace and re-configure with Z-Wave). I was able to control them with Amazon Echo commands though… if my server was online. Also, our automated cat feeder is dependent on the automation server. We were gone for a weekend and had to leave bowls of food out for them. I bet it was gone in the first 24 hours.
  3. My daughter’s birthday was on Sunday (6 years old!). While at the racetrack. she made up her mind that she wanted to watch dvd rips of the Gummi Bears when she got home. When we got home on Sunday afternoon, her world fell apart because I didn’t get the server back online yet. I’m working on setting her expectations that the world of entertainment offered by the server is a convenience and isn’t always available when she has screen time.
  4. Practice makes perfect. I haven’t had to perform any significant maintenance on my ESXi host for almost 2 years. In the future, I’ll practice more frequent roll-overs to my backup host and add a USB controller that matches the one in the primary host for automation interfaces. Then, I’ll only be without a media server if this happens again (see #3 above).

Like any other hobby, running a homelab can be a lot of work. I’ve become VERY dependent on the main VMs that run on my host. Most of the experimental hosts could be lost and I could easily start over with them to play with Docker and K8s, etc. I’ll have to make the routine maintenance part of the “fun” of my homelab, too.

- Duane

February 21st, 2020:
Fifth Generation Subaru Legacy GT Bypass Valve Upgrade

This write-up applies to 2010 – 2013 Legacy GT models with 2.5 turbocharged engine. A stock engine engine/tune is unlikely to benefit from changing you bypass valve.

Replacing the factory turbo Bypass Valve (aka blow-off valve or BOV) is far easier than I initially thought. Whether or not you should is hotly debated, but it’s really up to you. My custom tuned Cobb Stage II was leaking boost and this fixed it, but that may have been due to the age of the original more than the design. Here’s how you do it (since I couldn’t find any other overview):

Parts needed:
1.) New bypass valve. I went with the GFB Hybrid (T9233) because it allows me to use either hybrid or full recirculation easily.
2.) Spare plastic body fasteners. (You may break some during removal of the intake snorkel)
3.) Zip tie or small hose clamp (optional, see step 11)

Tools needed:
1.) 8mm socket or nut driver (snicker)
2.) 10mm (socket and ratchet)
3.) 12mm (socket and ratchet)
4.) Screwdriver or similar tool to pry fasteners and hoses
5.) Vice grips (recommended) or pliers

1.) Remove the 2 plastic fasteners for the cool air intake snorkel, then lift it out.
2.) Remove the passenger side radiator fan by disconnecting the electrical connector in the bottom corner. Then remove the 2 10mm bolts on the top. Lift the radiator fan straight up to remove.
3.) You can now easily access the Bypass Valve from above:

Factory Bypass Valve

4.) Slip the vacuum line off of the Bypass Valve.
5.) Use an 8mm socket or nut driver to loosen the upper hose clamp on the Bypass Valve.
6.) Wiggle the upper Bypass Valve tube out of the hose while gently prying the hose with a screwdriver. There’s a lot more flex in the lower hose than you expect. It will allow you to pull the upper hose off of the Bypass Valve and pull it aside to access the lower hose clamp.
7.) Use your vice grips to lock the spring-style lower hose clamp open… or try to do the same with pliers. Wiggle the lower hose off of the factory Bypass Valve.
8.) Compare the orientation of the inlet and outlet to your factory Bypass Valve. Also ensure that the vacuum line is aligned. Most aftermarket Bypass Valves allow you to rotate the vacuum line as needed if you remove a few screws. Follow the manufacturer’s instructions. The GFB Hybrid (T9233) did not require any adjustment.
9.) Insert the new Bypass Valve into the lower hose and align it with the upper hose. Remove your vice grips from the lower spring-style lower hose clamp.
10.) Insert the new Bypass Valve into the upper hose. Tighten the 8mm hose clamp so that it is snug. No need to go gorilla on it.
11.) Slip the vacuum line onto the vacuum inlet on the new Bypass Valve. A zip tie or small hose clamp may be necessary to ensure a good seal if the outer diameter of the inlet is smaller than the stock Bypass Valve.
12.) Install the passenger-side radiator fan: slide it straight down into place. Reinstall the 2 10mm bolts on the top. Re-connect the electrical connector on the lower corner of the fan.
13.) Re-install the intake snorkel and secure it with 2 plastic fasteners.
14.) If your Bypass Valve spring requires tuning, follow the instructions provided by the manufacturer. My throttle response benefited from a slightly tigher spring. It held boost without any adjustment, though.

Visibility Was Better With Fan Removed

Here’s how the hybrid sounds (not too loud) with a full Invidia exhaust:

2011 Subaru Legacy GT w/ GFB Bypass Valve Interior, Stationary
2011 Subaru Legacy GT w/ GFB Bypass Valve Exterior Rear, Stationary
2011 Subaru Legacy GT w/ GFB Bypass Valve Exterior Front, Stationary

- Duane

February 14th, 2019:
On Aging Parents, Life and Loss

I’ll be honest with you, this post is going to be challenging for me to write and for some to read. I’ll do my best if you do the same. Let’s get started:

When I last updated, I explained that we had a lot going on. As it turns out, I didn’t know the half of it. In October of 2017, both of my parents were celebrating Halloween, handing out candy and commenting on everyone’s costumes with our family in my brother’s house. They still lived in their home that my grandfather built. 15 months later… my mom has passed away and dad is alone in assisted living.

Within a month, my parents’ health began to worsen in alternating increments. The first big scare was Christmas morning (2017) when my mother was too weak to get out of bed and ended up in the hospital with a UTI. A few months later and we were taking her to urgent care because she “fell and hurt her knee”… in retrospect, she probably had a small stroke. We took her to the ER that night (another UTI, too) and decided in the ER with the whole family that it was time to find an independent living community.

We briefly pursued this direction a few years earlier, but my father was very resistant. They had too much “stuff” and weren’t ready to part with it. In hindsight, the additional care might have improved their quality of life significantly. This time, everyone was very serious about the endeavor. We kept my mom apprised of our progress while she get better in the hospital. We established a budget and timeline, then researched a few communities. After a handful of visits, we arranged a company to move them and their 2 pets. In just 3 weeks, they were living in a moderately-sized 1-bedroom apartment on the ground floor about 15 minutes from their house. At first, it seemed promising. The meal plan and care from the staff helped, but my parents continued to “take turns” in the hospital each needing care a few times over the next 3 months. The peak of this situation occurred mid-summer, 2018. My father was hospitalized (and later in long-term physical rehab) for a serious infection for almost 2 months while my mom was at the apartment alone. We maxed out the care program offered by the community, but the staff wasn’t trained to the level of care she truly needed. We didn’t realize that her dementia/Alzheimer’s had progressed to moderate. She wasn’t drinking enough water, not eating enough food and just not taking care of herself. When we visited, she put on a great show. We didn’t know how much help she really needed.

In late August, the day before my father returned to their apartment, my mom went to the ER by ambulance. She was having a dementia-related panic attack exacerbated by confusion from yet another infection. This was the first time mom looked really bad in the hospital… but she appeared to get better quickly, even joking in the hospital. Her care was extended by a stay in a rehab facility near their apartment. Eventually, she was even walking (with help) and making plans for Halloween in a few months. She was in the hospital and rehab for 5 weeks.

My mom taking Kara for a walk in post-hospital rehab.

During my mom’s recovery, our family decided to upgrade my parents’ care to assisted living with guidance from the hospital staff. We identified a handful of assisted living locations and decided to move to a community very close to their house (which were just finishing cleaning out to prepare for sale). The cost was significantly greater, but the level of care increased accordingly. The staff is amazing, their apartment is very nice (2 bedroom to accommodate all of their stuff) and their pets were welcome.

Things seemed to calm down for a few weeks, but it was obvious that mom’s dementia was worsening exponentially. In the evenings she was more confused and prone to repeatedly calling out. For years, she could use an iPad for crossword puzzles, video calls, email and web browsing… now she had trouble turning it on or figuring out how to charge it. The staff tried to monitor her fluid intake, but could only do so accurately during meals. Mom, always the social perfectionist, continued to put on a good show in the dining hall.

Mom’s plans for Halloween would never happen: on October 30th mom was back in the hospital. Their new apartment was closer to their “old neighborhood” and allowed for mom to be transported to a familiar hospital nearly around the corner from my brother’s house. We didn’t know then how much time we would be spending at that hospital, but were very glad to have the care, guidance and support of the staff at Beaumont in Grosse Pointe. This stay was different: mom was having trouble swallowing and the speech therapist wasn’t able to get her to follow instructions consistently. We discussed moving mom to the memory care area of their community for closer monitoring and therapy. She was in the hospital/rehab through November, but we were able to drive her to their community for Thanksgiving dinner. This time, mom went back to the hospital (followed by rehab) briefly before returning home.

We wanted to celebrate mom’s 80th birthday a week-or-so before Christmas, but realized that her health was very complicated and unstable… and had to call it off when she returned to the hospital 3 days before her birthday. Again, this stay in the hospital took its toll on mom. We didn’t know it, but the repeated infections weakened her to the point where she couldn’t fight off pneumonia. It just wasn’t detected with tests or examinations and she went home (against protest) instead of rehab.

A week later (Jan 2) and Mom was in the ER with respiratory failure. We all rushed to the hospital and the staff explained that she needed a breathing tube. Her kidneys weren’t working well and her (existing) heart condition was the root cause. I can still remember the minute-by-minute details, but will spare you. After a few days, her condition improved slightly and the breathing tube was removed, but her respiration never fully recovered. Her other organs were not working correctly. At the same time, my father was admitted to the hospital. With all of the stress, he wasn’t taking his medications and had to be admitted. This turned out to be a blessing: he was able to spend time with my mom.

While on the topic of blessings, climate change provided us with oddly warm weather so a smooth 50+ mile commute each morning for updates from the ICU staff was possible. The construction on ALL of the expressways between our house and the hospital wrapped up, further shortening travel time. Mom was able to recognize all of us until she became unresponsive and that time meant a lot to us.

Gina and I went home to have dinner and rest for a night after a few days at the hospital, but we got a call that mom had taken a turn. I went to the hospital and joined by brother and dad. The doctors told us that we only had a few days (at most)…

Mom’s pastor was out of the country, so our pastor and his wife visited, talked to the kids and spent time with us. We moved to a peaceful room and my brother and I stayed with mom until she passed peacefully. We cried a bit, but were grateful for the time we had with her. It was hard telling my dad, but he was able see her one more time, too. We started to make arrangements and my dad’s health improved enough to go to a nearby rehab for a few weeks. He’s home now and we’re preparing for mom’s memorial this weekend.

I never prepared for my parents to be “old” or get sick. I knew one day they would be gone and tried to brace myself, but caring for them and putting together a plan was more than I expected. The past 15 months have been a roller coaster of work, fear, worry, stress, sadness and (sometimes) relief. We should have planned better but luckily my mom had clear instructions for us to follow as her health declined. We’ve learned a lot and still have a lot to take care of, but I want to share some of the basics in case they help others now that I’ve had some time to reflect and gather my thoughts:

  • Be active. Visit your aging parents as frequently as possible. Get them out and about.
  • Don’t take anything lightly. The big things send them to the hospital, the small things are all warnings. Some of the major flags for our situation were trouble swallowing, not drinking enough fluids, missing medications, “small confusions” that ended up being UTIs and eventually dementia.
  • Be involved in their medical care and listen to the doctors. Ask questions. If possible, keep a journal and share it with their healthcare providers.
  • Know their medications. Make sure all of the meds are taken when they are supposed to be.
  • Keep track of their important documents including government-issued ID.
  • Make a budget plan. Update it frequently.
  • Don’t rush into things. Evaluate all of your options: care, housing, medical treatment, funeral arrangements, etc. At the same time, don’t postpone things unnecessarily. This seems contradictory, but there’s a balance to be found.
  • Value the time you have together and do everything you can to preserve your memories. (I’ve never been a big photo/video creator.)

Our family grieves in waves. Happy memories outnumber the difficult ones from recent months… but those memories are most fresh and both bring tears. In time, I know the hard times will fade and happy thoughts will once again take over. I crafted a pair of photo boards for my mom’s memorial service and this helped me more than I expected. We trying our best to keep dad happy; his grand-kids are a constant source of support and new, happy memories. We are working with him to improve his health so that he can more actively participate in our family events and, hopefully, outings. We’ve been dreading the future for so long. It’s nice to look forward to things again and hope for continued improvement and family time.

- Duane

August 31st, 2017:
Late Summer 2017 Update

Well, no racing for us this year. The combination of a 1-year old and new engine/new racing class didn’t really work out. We’ve been able to spend some quality time with Kara at the state and metro parks, plus a few great trips to Northport, MI. The weather has been nice, but my work schedule (especially having my home office in the Mountain time zone) has severely limited my time outside during the week. Hopefully this will ease up soon in my new-ish role (promoted back into Sr. Management again: Sr. Director).

I’ve done a lot of work cleaning up our workshop this year. I’ve meant to clean it up and prepare to insulate/heat the shop so I can install the new engine in my racecar… but it’s turned into a lot of cleanup and corrections for lack of ventilation from our new roof (circa 2009). I also spent a few hours replacing the suspension and lifting my Jeep due to a need for new tires and shocks.

I’ve been keeping up with my retro computer collection: everything is still working and I find an few hours per week to enjoy my Amigas, PCs, Macs and Commodore machines. I’m almost through my Leisure Suit Larry bucket list: wrap up the series. I’ve owned about 3 or 4 versions of the game series (or parts of it) over the years and it’s time to finally wrap it up! (I took a short break for Space Quest and Police Quest in there, too.) I added a SX-64 Executive "e;portable" to my collection earlier this year and have had my eye on few Apple IIc computers, but the price or condition haven’t been up to snuff.

Most recently, I’ve been working on capturing/digitizing a few boxes of 8mm/Hi8/DV that literally fell out of my closet a week ago. I’ve found some amazing stuff on those tapes: lots of high school wrestling, some hanging out with friends, and an emotional collection of holiday footage with many family members that have since passed. The process is relatively slow as I must capture each tape in real-time. However, my setup is surprisingly effective: a Sony Hi8 camera connected via analog S-Video and Audio (when capturing 8mm and Hi8) to my Sharp DV Camera, connected via Firewire 800 to an Apple Thunderbolt Lightning Cinema Display, connected to my 2011 Macbook Air via Thunderbolt with an external Thunderbolt drive for storage. Both cameras are time base corrected resulting in decent quality considering the source media. The capturing process is taking a long time, but it’s mostly hands-off… the real work will be sorting, editing and finalizing for DVD (all source is 480p at best). Some of the content might make it to YouTube, but I’ll likely keep it private for friends.

That sums up a lot from the past few months. Seems like each season I hope to make more regular updates to my blog, but I end up doing this instead. More to come?

- Duane