So last night was a busy one thanks to the HP Chassis and “fantastic” Virtual Connect system.
Bare with this article, as its part what happens blah blah, and part, how to install firmware on Virtual Connect
So the environment consists of;
HP C7000 Chassis
6 x HP BL460c Blade Servers
2 x HP VC Flex-10 Enet Module
These VC Flex modules are connected by 2 x 10GB links in a LACP trunk to Cisco 3750X stack.
The plan of work
The customer wants to move to VMware 5.5, nice and simple, upgraded from VMware 5.0, to do this you have to also update your HP gear to the latest supported levels.
After some searching online and no thanks to the documentation of HP, it was deducted that you must do this in the following order.
1. Update your HP Chassis Onboard Administrator
2. Update all your blades using the SPP disc
3. Update your Virtual Connect Manager
4. Upgrade all ESXi hosts to 5.5, and your vCenter
Viola, and your done
So I updated the HP Onboard Administrator to 4.22 (Jun 13 2014) Firmware, this worked perfectly.
I updated all the blades using the SPP disc via the iLO and virtual media, nice and simple.
And then I update the Virtual Connect from 3.60 to 4.30. After this we lost all connectivity to the blades and all Virtual Machines died as the blades couldnt get to the iSCSI storage.
On the Virtual Connect Manager go to Hardware > Overview > Click on a VC Switch > Server Ports
When looking at the Shared Uplink Port you can see that both of them say standby aswell, meaning neither switch is acting as active. And the view from the ESXi box
(Please note I've edited the screenshot to reflect what we were seeing at the time of the issue as I didnt get a screenshot of everything)
So this meant a phone call to HP support, of which there is no exact shortcut or option to press for “HP Virtual Connect Manager”, I dont even think HP want to admit it exists.
The fix is quite simple, downgrade to where you have come from;
But I need to be on VMware 5.5, so I need to upgrade.
Yep, to downgrade its a pain, if you just try to downgrade it will fail,
You need to turn off all your blade servers first. (Yes so much for that High Availability blah blah, only exists if you have multiple chassis and the right VMware setup for failover)
Run the VCSU tool, as below, however you need to specify at the following line
Please enter Force Update options if any (eg: version,health):
Rolling back the firmware to 3.60 worked straight away which connectivity restored, I just have to do the rest of the patches now.
How to update the Virtual Connect Manager
Note: Please plan downtime for this, as in my case it took everything offline whilst updating both modules.
So to update the Virtual Connect Manager, leave the web interface, as you don’t do it there, you need to download the “Virtual Connect Support Utility”, of which I downloaded the wrong version first, you need 1.10 which can be found here.
To do this, you open up the command interface,
As you can see this launches a command prompt in the install area of the utility, which is fantastic, because it also exits the program.
1. Type vcsu.exe and enter
2. Type update and enter
3. Enter the IP address of the Onboard Administrator, username and password.
4. Drag your firmware file from its location into the CMD prompt window and this will automatically out the file location in for you.
4. Click through the options until it says “Initializing, please wait… [Step 1 of 2]”
5. When prompted it will tell you its discovered the Virtual Connect domain, enter the username and password for the Virtual Connect Manager
6. Enter yes after confirming the version you are on and version you are going to.
7. Once done it will tell you its completed and where the log files are loaded.
Here is a full screenshot of the process, with the steps highlighted in yellow with green marks for easy identification.
Things to know about the Virtual Connect Support Utility
it saves everything in your temp location, so where it tells you it’s saved the log, copy the location and open it up
So it doesn’t just output a file which is the copy of your update session, which in this case is called “vcsu-15044.log”
It saves a copy of your Virtual Connect Manager settings before the upgrade, of which mine was called “vcConfig-8895406-BLE01_vc_domain-201408130822”
And finally there is the Support Info output which can be used by HP support to determine the status of your hardware before and after the firmware update “vcSupportInfo-8895406-BLE01_vc_domain-201408130823”
If you run the “collect” command, then it produces an output to the temp location for the following files
– Session output
– ShowAll of OA
– ShowAll of Virtual Connect Manager
The cause of this is basically a faulty firmware, which the guy from HP wouldnt admit to actually being faulty, but basically said don’t go to 4.30 firmware after you have downgraded and start to upgrade again.
Who’s fault was this outage in the production network? Mine I guess, it was a scheduled maintainance window, and I’ve updated Blade chassis and switches and so forth hundreds of times, and I’ve never had and issue such as this. But its my fault because I could have dug deeper into the update etc. I read the guides on how to update everything, I read the release notes of the firmware, but I must admit HP dropped the ball here.
Going off previous experience when updating Dell devices, they usually state in their release notes and guides, supported upgrade paths, i.e how you get from ver 1.0 to ver 2.0 and any steps in between you need to take. For HP this wasnt the case, it just said, here’s the latest firmware, and here’s how to upgrade it. Hell I struggled to find the 3.6 firmware to roll back to, the HP support guy sent me the link!
End of the day, I wont be designing or offering HP Chassis and blades as a Solution any time soon, I’ll stick with Cisco UCS