One of the specific challenges faced in ROBO deployments is network bandwidth and latency. In this case, a number of the customer’s sites have such poor connectivity that whilst vCenter will operate over the links, delivering patches via vCenter Update Manger (VUM) isn’t really viable – pushing a 250MB+ patch over a 128kbps line can saturate the link causing QoS issues for other critical traffic.

To work around this we’re making use of the customer’s existing tried and tested patch delivery method to get the VMware patch bundles to the remote sites. The patch bundles are staged onto a shared datastore at the site and then applied using PowerCLI.

One of the enhancements brought in with PowerCLI 4.0 Update 1 was the Install-VMHostPatch cmdlet. It really is very simple to use and avoids having to pass host credentials in scripts as the vihostupdate.pl command within vSphere CLI requires. Carter Shanklin posted a short video demonstrating its use – it’s definitely worth watching.

Anyway… enough scene-setting. Below is the quick-n-dirty script I pulled together (with a little help from various folks) to demonstrate the concept. This script will be improved in the near future to include error handling, and I also plan to more accurately document the project. I’ll update this post at that time.

##################################################################
# ROBO Cluster patching script by Rob Upham - r o b @ vmware.com #
# Script assumes a two-node cluster with vMotion but no DRS      #
##################################################################

##########################################
# Edit these values to suit your cluster #
# and the patch being applied            #
##########################################

$VC = "robo-vc-x64.vmwdemo.com"
$host1 = "esx-robo-1.vmwdemo.com"
$host2 = "esx-robo-2.vmwdemo.com"
$patchpath = "/vmfs/volumes/Template-FC-EMC/Patches/ESXi400-201002001/metadata.zip"
#Patch bundles should be unzipped, placed on shared VMFS or NFS storage and the metadata.zip file referenced above
# e.g. $patchpath = "/vmfs/volumes/NFS-Vol1/Patches/ESXi400-200912001/metadata.zip"

########################
# Work gets done below #
########################

#Connect to vCenter Server
Write-Host "Connecting to vCenter Server"
Connect-VIServer -Server $VC

#Get host objects
$VMHostObj1 = Get-VMHost $Host1
$VMHostObj2 = Get-VMHost $Host2

#Check current build numbers
Write-Host "Hosts to be patched are:"
Write-Host "Host1 = $host1"
$VMHostObj1 | get-view -Property Name,Config.Product | select Name,{$_.Config.Product.Fullname}
Write-Host "Host2 = $host2"
$VMHostObj2 | get-view -Property Name,Config.Product | select Name,{$_.Config.Product.Fullname}

#Host 1
#Move vms to Host 2 and enable maintenance mode
Write-Host "Moving VMs from $host1 to $host2"
$VMHostObj1 | get-vm | move-vm -destination ($VMHostObj2)

Write-Host "$host1 entering maintenance mode"
$VMHostObj1 | set-vmhost –state maintenance

#Update host 1
Write-Host "Patching $host1 with patch $patchpath"
$VMHostObj1 | Install-VMHostPatch -HostPath $patchpath

#Reboot the ESX Server host 1
$VMhostObj1 | Restart-VMHost -Confirm:$false
Write-Host "Waiting 2.5 minutes for $host1 to reboot"
Sleep 150

$VMHostState = (Get-VMHost -Name $host1 -ErrorAction SilentlyContinue).State
While ($VMHostState -eq "NotResponding"){
  Write-Host "ESX Server state is" $VMHostState
  Write-Host "Waiting 20 seconds until ESX Server starts responding"
  sleep 20
  $VMHostState = (Get-VMHost -Name $host1).State
}
Write-Host "Taking ESX Server host out of Maintenance Mode"
$VMHostObj1 | set-vmhost –state connected
Write-Host "Patching $host1 complete"

#Host 2
#Move vms to Host 1 and enable maintenance mode
Write-Host "Moving VMs from $host2 to $host1"
$VMHostObj2 | get-vm | move-vm -destination ($VMHostObj1)

Write-Host "$host2 entering maintenance mode"
$VMHostObj2 | set-vmhost –state maintenance

#Update host 2
Write-Host "Patching $host2 with patch $patchpath"
$VMHostObj2 | Install-VMHostPatch -HostPath $patchpath

#Reboot the ESX Server host 2
$VMhostObj2 | Restart-VMHost -Confirm:$false
Write-Host "Waiting 2.5 minutes for $host2 to reboot"
Sleep 150

$VMHostState = (Get-VMHost -Name $host2 -ErrorAction SilentlyContinue).State
While ($VMHostState -eq "NotResponding"){
  Write-Host "ESX Server state is" $VMHostState
  Write-Host "Waiting 20 seconds until ESX Server starts responding"
  sleep 20
  $VMHostState = (Get-VMHost -Name $host2).State
}
Write-Host "Taking ESX Server host out of Maintenance Mode"
$VMHostObj2 | set-vmhost –state connected
Write-Host "Patching $host2 complete"
Write-Host "Build numbers are now as follows:"
$VMHostObj1 | get-view -Property Name,Config.Product | select Name,{$_.Config.Product.Fullname}
$VMHostObj2 | get-view -Property Name,Config.Product | select Name,{$_.Config.Product.Fullname}
Write-Host "All patching complete"

Updated 24 March 10 with tweaks suggested by Carter and Alan

I’m sure I’ll get round to it when I think of something worth sharing with the world.