Eric Stewart: Running Off At The Mouth

So … we have this nasty habit of not rebooting boxes.  We use yum-cron to keep the systems up to date.  This works fine for the most part, except that when new kernel packages come in, we tend to not pay that much attention.

From a security standpoint, we tend to prevent unauthorized access to the systems by locking down who they’ll actually talk to, so updated kernels are less of an issue for us than they might be for others.

This silly practice, though, does bite us in the butt once in a while.

Our kindly VM Infrastructure manager sent me an email asking me to upgrade the VMWare Tools on our virtual machines.  So, I did what he said to do and rebooted the boxes that I could (as several others will have to wait until scheduled downtime to be rebooted).  When the systems came back up … uh oh, no network.

This is not unheard of in the world of VMWare.  In my case, here’s what caused the issue.

1. A long running CentOS box updates it kernel, but never gets rebooted to use that new kernel.
2. Idiot System Admin Who Should Have Known Better (ISAWSHKB) comes up, installs VMWare Tools, which politely compiles/configures devices/virtual hardware for the currently running kernel.
3. ISAWSHKB reboots virtual server per instructions.
4. Server politely comes up with newer kernel.  VMWare fails to deal with this issue, and boom no network available.

So, here’s what I suggest (not yet tested, so you’re on your own):

1. If virtual hardware might need upgrading, power off the system.  Otherwise, just reboot.
   1. While system is powered off, upgrade the hardware.  Power system back on.
   2. It’s possible you’ll have no networking at this point.  No biggie – log in through your vSphere Client to the server.
2. Once you’re logged back into the system, go through the process of upgrading your VMWare Tools (if you’ve done this in a different order, you will have to reinstall reconfigure the tools after rebooting/upgrading hardware).
3. Reboot again.  You should be back to normal.

I’m also checking to see if this can be automated.  Using a script someone else on the intarwebs came up with, and a bit of my own know-how, I put this into “/etc/init.d/fix-vmware-tools”:

#!/bin/sh
#
# Startup/Shutdown script for fixing VMWare Tools
# 
# chkconfig: 35 02 01
# description: Fixes VMWare Tools for kernel update
#
case "${1}" in
     stop)
          echo "Nothing to stop"
          ;;
     start)
          rkernel=`uname -r`
          vmxnet="/lib/modules/`uname -r`/misc/vmxnet.ko"
          if [ ! -e $vmxnet ]; then
               echo "vmware-tools not configured for running kernel $rkernel"
               echo "running vmware-config-tools.pl"
               /usr/bin/vmware-config-tools.pl -d
               echo "vmware-tools now compiled for running kernel $rkernel"
          else
               echo "$vmxnet found"
          fi
          ;;
esac

And then ran “chkconfig fix-vmware-tools on”.  Initial tests are promising; hopefully future updates won’t break this!  Note that for startups, it starts at 2; vmware tools starts at 3 and this will need to run before they do.