Pages Menu
TwitterRssFacebook

Popping Clouds

A blog on virtualization and cloud computing

Categories Menu

Posted by on Feb 22, 2014 in Featured, vSphere | 2 comments

Upgrading vCenter 4.1 to 5.5 – Lessons learned

I went “full cowboy” last night and executed an in-place upgrade (to 5.5) of a substantially sized production vCenter. It was running 4.1, contains about 50 UCS hosts, and around 1000 VMs. I ran into essentially every bug/issue along the way, and wanted to document for posterity what I learned. Actually I should say what “we” learned, as I went through the ordeal with a few friends of mine; Scott from Capgemini, and Danby from Honeywell.

The vCenter upgrade process is actually fairly simple, (all things considered). You basically just backup your existing database, snapshot (or clone) your existing vCenter (if virtual), mount the ISO and let ‘er rip. I wont go through that process as it already very well documented elsewhere.

Looking back the main issue really was just the size of the vCenter database, specifically the vpx_event and vpx_task tables. This was causing DBUHELPER to essentially run out of memory buffer space and crash during the database upgrade.

Had I been more careful, we would have either purged those tables as documented here, or simply reduced their size as documented here.

What we ended up doing was editing the vpxd.cfg file (in ProgramData\VMware\VMware VirtualCenter\ on 2008) and added the following code (anywhere before the closing </vpxd>):

<upgradedb>
<urlBatchSize>1200</urlBatchSize>
</upgradedb>

This will limit DBUHELPER to 1200 records at a time. It allowed the DB upgrade to complete successfully.

Other lessons learned:

  • Don’t use a “!” in your administrator@vsphere.local password. It will cause the SSO installation to roll back.
    • I actually run into this one every time with SSO. For some reason I cannot help but put an exclamation point in the password.
  • When upgrading from 4.1 to 5.x you typically have to pay attention to your SSL certificates. They are (at least in my case) usually expired. In this installation we are using self-signed certificates, so simply renaming the SSL directory in “ProgramData\VMware\VMWare VirtualCenter\” to something like “SSL.old” will cause the install to generate new SSL certificates.
  • Make sure you enable certificate validation in vCenter prior to the upgrade from 4.1. Found in the vCenter client under Administration/Server Settings/SSL settings
  • Eject or upgrade any 3.x hosts prior to the upgrade.
  • Make sure that the Microsoft Error Reporting service is set to “Manual.” If it is set to “Disabled” the installation of the SSO components will fail.

All said and done, the upgrade started at around 6PM and ended at 3AM. :) Not exactly a fun night, but we were successful in the end.

Like I mentioned earlier, our major problem was around the DB size. Upgrading that database was a real pain for us. In the end, even after the upgrade, we were able to get vCenter and all of its supporting services to start however all (50) of the hosts were disconnected. We had to go through each host and enter the credentials to reconnect it. If anyone is ever in a similar situation, here is a quick PowerShell script that will re-connect all (currently disconnected) hosts:

# Variables

$VCServer = “vcserver.yourdomain.local”

$password = “rootPassword”

#Connect to vCenter Server

$VC = Connect-VIServer $VCServer

get-vmhost | % {

$view = get-view $_.id

$arg = new-object VMware.Vim.HostConnectSpec

$arg.userName = “root”

$arg.password = $password

$arg.force = $true

$view.DisconnectHost()

$view.ReconnectHost($arg)

}

Disconnect-VIServer-Confirm:$false

Thanks for reading!

 

2 Comments

  1. Freaking amazing. Thank you!

  2. You are certainly brave! That sounds like an Awesome escapade! Hope you and yours are still doing well!

Trackbacks/Pingbacks

  1. Upgrading vCenter 4.1 to 5.5 – Lessons learned – Popping Clouds | - […] Upgrading vCenter 4.1 to 5.5 – Lessons learned – Popping Clouds. […]

Leave a Reply