I suspect many of you reading this know how these things go; you start off ascertaining what OS you’re dealing with, move on to getting them to click various components of the GUI, before progressing to some command-line stuff.

In this case, the WiFi connection was sound; the adapter was receiving a DHCP lease, and we could ping IPs on the Internet.
However, we couldn’t ping the router, or get the router to respond to DNS queries.

I was moments away from manually configuring a known-good external DNS server when I thought we should venture into a quick look at the routing table. As tough as it might be over the phone, I got my colleague to read out the entries from his “route print” output.

This revealed a few entries I didn’t like the sound of – several for 192.168.1.0/24 (the default internal subnet used by his router). So, I got him to read out his full ipconfig output – suspecting something might be up.

Sure enough, it turned out VMware Workstation had randomly chosen 192.168.1.0/24 as the subnet to use for the VMnet8 (NAT) network. I assume that when the machine was first built it was on a corporate 10.x.x.x network. So when Workstation randomly chose a couple of /24s from the 192.168 range to use for VMnet1 and VMnet8, 192.168.1.0/24 was available – but choosing this means clashing with perhaps the most commonly used private IP range in the world.

For those not familiar with VMware Workstation, it supports up to 8 virtual networks and provides a DHCP service which can run on these networks to allocate IPs to VMs connected to them. At install (or first run) it randomly selects two /24 ranges from 192.168.0.0/16 (e.g. 192.168.114.0/24) for use on these virtual networks.

A quick visit to the Virtual Network Editor (Start / All Programs / VMware / Virtual Network Editor) allowed my colleague to change the subnet used for VMnet8 to another /24 range – and the problem disappeared.

Certainly one to chalk up to experience!

In my view, Workstation should avoid using this specific subnet out of the box – I’m sure it must catch out a fair few users (who seemingly have a 2 in 254 chance of having either VMnet1 or VMnet8 picking this range).

Please let me know (leave a comment) if you’ve also suffered from this issue.

One of the specific challenges faced in ROBO deployments is network bandwidth and latency. In this case, a number of the customer’s sites have such poor connectivity that whilst vCenter will operate over the links, delivering patches via vCenter Update Manger (VUM) isn’t really viable – pushing a 250MB+ patch over a 128kbps line can saturate the link causing QoS issues for other critical traffic.

To work around this we’re making use of the customer’s existing tried and tested patch delivery method to get the VMware patch bundles to the remote sites. The patch bundles are staged onto a shared datastore at the site and then applied using PowerCLI.

One of the enhancements brought in with PowerCLI 4.0 Update 1 was the Install-VMHostPatch cmdlet. It really is very simple to use and avoids having to pass host credentials in scripts as the vihostupdate.pl command within vSphere CLI requires. Carter Shanklin posted a short video demonstrating its use – it’s definitely worth watching.

Anyway… enough scene-setting. Below is the quick-n-dirty script I pulled together (with a little help from various folks) to demonstrate the concept. This script will be improved in the near future to include error handling, and I also plan to more accurately document the project. I’ll update this post at that time.

##################################################################
# ROBO Cluster patching script by Rob Upham - r o b @ vmware.com #
# Script assumes a two-node cluster with vMotion but no DRS      #
##################################################################

##########################################
# Edit these values to suit your cluster #
# and the patch being applied            #
##########################################

$VC = "robo-vc-x64.vmwdemo.com"
$host1 = "esx-robo-1.vmwdemo.com"
$host2 = "esx-robo-2.vmwdemo.com"
$patchpath = "/vmfs/volumes/Template-FC-EMC/Patches/ESXi400-201002001/metadata.zip"
#Patch bundles should be unzipped, placed on shared VMFS or NFS storage and the metadata.zip file referenced above
# e.g. $patchpath = "/vmfs/volumes/NFS-Vol1/Patches/ESXi400-200912001/metadata.zip"

########################
# Work gets done below #
########################

#Connect to vCenter Server
Write-Host "Connecting to vCenter Server"
Connect-VIServer -Server $VC

#Get host objects
$VMHostObj1 = Get-VMHost $Host1
$VMHostObj2 = Get-VMHost $Host2

#Check current build numbers
Write-Host "Hosts to be patched are:"
Write-Host "Host1 = $host1"
$VMHostObj1 | get-view -Property Name,Config.Product | select Name,{$_.Config.Product.Fullname}
Write-Host "Host2 = $host2"
$VMHostObj2 | get-view -Property Name,Config.Product | select Name,{$_.Config.Product.Fullname}

#Host 1
#Move vms to Host 2 and enable maintenance mode
Write-Host "Moving VMs from $host1 to $host2"
$VMHostObj1 | get-vm | move-vm -destination ($VMHostObj2)

Write-Host "$host1 entering maintenance mode"
$VMHostObj1 | set-vmhost –state maintenance

#Update host 1
Write-Host "Patching $host1 with patch $patchpath"
$VMHostObj1 | Install-VMHostPatch -HostPath $patchpath

#Reboot the ESX Server host 1
$VMhostObj1 | Restart-VMHost -Confirm:$false
Write-Host "Waiting 2.5 minutes for $host1 to reboot"
Sleep 150

$VMHostState = (Get-VMHost -Name $host1 -ErrorAction SilentlyContinue).State
While ($VMHostState -eq "NotResponding"){
  Write-Host "ESX Server state is" $VMHostState
  Write-Host "Waiting 20 seconds until ESX Server starts responding"
  sleep 20
  $VMHostState = (Get-VMHost -Name $host1).State
}
Write-Host "Taking ESX Server host out of Maintenance Mode"
$VMHostObj1 | set-vmhost –state connected
Write-Host "Patching $host1 complete"

#Host 2
#Move vms to Host 1 and enable maintenance mode
Write-Host "Moving VMs from $host2 to $host1"
$VMHostObj2 | get-vm | move-vm -destination ($VMHostObj1)

Write-Host "$host2 entering maintenance mode"
$VMHostObj2 | set-vmhost –state maintenance

#Update host 2
Write-Host "Patching $host2 with patch $patchpath"
$VMHostObj2 | Install-VMHostPatch -HostPath $patchpath

#Reboot the ESX Server host 2
$VMhostObj2 | Restart-VMHost -Confirm:$false
Write-Host "Waiting 2.5 minutes for $host2 to reboot"
Sleep 150

$VMHostState = (Get-VMHost -Name $host2 -ErrorAction SilentlyContinue).State
While ($VMHostState -eq "NotResponding"){
  Write-Host "ESX Server state is" $VMHostState
  Write-Host "Waiting 20 seconds until ESX Server starts responding"
  sleep 20
  $VMHostState = (Get-VMHost -Name $host2).State
}
Write-Host "Taking ESX Server host out of Maintenance Mode"
$VMHostObj2 | set-vmhost –state connected
Write-Host "Patching $host2 complete"
Write-Host "Build numbers are now as follows:"
$VMHostObj1 | get-view -Property Name,Config.Product | select Name,{$_.Config.Product.Fullname}
$VMHostObj2 | get-view -Property Name,Config.Product | select Name,{$_.Config.Product.Fullname}
Write-Host "All patching complete"

Updated 24 March 10 with tweaks suggested by Carter and Alan

I’m sure I’ll get round to it when I think of something worth sharing with the world.