ESXi server unable to join to VCenter after update 3 upgrade ?

Hi People,

I'm having some serious issue after upgrading my VMware ESXi 5.1 Update 2 Patch 5 (2000251) into ESXi 5.1 Update 3 (2323236) on my HP Blade BL 465c G7 with Emulex NC551i Dual Port FlexFabric 10 Gb adapter.

The ESXi server is not ping-able and unable to join the VCenter anymore.
I've restarted the ESXi server and restart the service console, but still it is not pingable from outside of the host and even from the ESXi service console to any outside IP.

Here's the driver and firmware version before the upgrade:

~ # ethtool -i vmnic0
driver: be2net
version: 10.2.293.0
firmware-version: 4.9.416.2
bus-info: 0000:04:00.0
~ #

Open in new window


and this is the version that I have also tried to upgrade it into: 10.2.477.20
VMW-ESX-5.1.0-be2net-10.2.477.20-offline_bundle-2541837.zip

Open in new window


Any kind of help would be greatly appreciated.

Thanks,
LVL 9
Senior IT System EngineerIT ProfessionalAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Senior IT System EngineerIT ProfessionalAuthor Commented:
Rolling back the ESXi server using Shift+R also didn't rollback to the previous working version, the build number stays at 2000251 and the server still un-pingable.

I've been upgrading using the following HP supplied .ISO files:

VMware-ESXi-5.1.0-Update2-2000251-HP-5.69.2-Nov2014
VMware-ESXi-5.1.0-Update3-2323236-HP-510.9.3.16-Jun2015

Open in new window


But for my other HP Blade BL 465c G8 with Emulex HP FlexFabric 10Gb 2-port 554FLB adapter it was upgraded successfully with no issue at all.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Are you using the OEM HP version of ESXi ?

Have you made sure, that ALL firmware on the network interfaces and Host are up to date as far as HP is concerned, and you are using all the latest VIBs (drivers) from HP?
1
Senior IT System EngineerIT ProfessionalAuthor Commented:
Yes I'm using the HP version as per my post above.

So at the moment the IP address is not pingable and I cannot roll back, does upgrading the firmware of theEmulex can bring back the network connection ?
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
have you checked the drivers are loaded, and you have visible nics in ESXi ?

and all the IP Addresses are assigned correctly ?
0
Senior IT System EngineerIT ProfessionalAuthor Commented:
have you checked the drivers are loaded, and you have visible nics in ESXi ?
using the command ethtool -i vmnic0 I can see the version.

and all the IP Addresses are assigned correctly ?
yes, because this is the second upgrade from ESXi 5.1 Update 1 to 2 and yesterday to update 3, the IP address and all of the Distributed switch & etc... should be intact not touched.

So how can I perform configuration backup from ESXi console locally if the vMA applaince cannot issue vicfg-cfgbackup command ?

Capture.JPG
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
there have been many issues with firmware for HP nics. Matched firmware and drivers must be present, otherwise traffic does not flow.

you have a few options

1. rollback to a working version of ESXi 5.1 update 1 - and restore your backup. This requires a new installation of ESXi 5.1 U1.

2. try and fix the issue, which is likely to be an incompatible driver with nic and/or firmware.
1

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I've just checked the Guide

June 2015 VMware FW and Software Recipe

and it states on Page 15

driver and firmware combos are:-

driver - 10.2.47720.-1OEM.510
firmware - 4.9.416.71

these are the recommended solution for 5.1 by HP.

So as you firmware is different, I would at least check, and update the firmware, before considering next action.
0
Senior IT System EngineerIT ProfessionalAuthor Commented:
Thanks Andrew,

So in this case, how can I perform the backup without vMA and network connectivity ?
the Shift+R restore didn't work to the previous version it is still stuck to version 2323236 (Update 3).

By updating the firmware of Emulex to the latest, does it erase the existing network configurations ?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
So in this case, how can I perform the backup without vMA and network connectivity ?

Not easily, copy the state.tgz file. This contains the backup. But it's not a lot of use, because it matches the current version running, e.g. you cannot apply this to a U1.

By updating the firmware of Emulex to the latest, does it erase the existing network configurations ?

Firmware Update is completely independent of the OS. You will need to do this from DVDROM/CDROM Smart Start Firmware disk.
1
Senior IT System EngineerIT ProfessionalAuthor Commented:
OK, I've managed to do the following:

Updating the below components as per HP latest SPP 2015.06 DVD .ISO
ESXi network driver update to 10.2.477.20 – Completed successfully
Emulex CAN adapter firmware update to 4.9.416.7 – Completed successfully

Versions
Reinstalling the ESXi 5.1 Update 3 from scratch (clean install) – Completed successfully but the network connection still the problem, hence I am reverting back to below....
Reinstalling the ESXi 5.1 Update 2 from scratch (clean install) – Completed successfully

After applying the update to both ESXi driver and Firmware versions the Network is still not working.

Despite VMware HCL says HP BL 465c G7 is supported in 5.1 Update 3: http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=server&productid=14145&releaseid=270&deviceCategory=server&details=1&partner=41&releases=270&systemTypes=1&page=1&display_interval=500&sortColumn=Partner&sortOrder=Asc

(HP NC551i) Emulex OneConnect OCe10102 10GbE NIC CNA for HP ProLiant AMD G7 BladeSystems
Updated as per this page recommendation:
http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=19068&deviceCategory=io&details=1&partner=41&releases=270&page=1&display_interval=10&sortColumn=Partner&sortOrder=Asc
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
and if you rollback to ESXi 5.1 U1 ?

does it work ?
0
Senior IT System EngineerIT ProfessionalAuthor Commented:
Andrew, the Shift+R recovery brings the ESXi to 5.1 update 3 cannot rolled back to update 2.

hence I have to completely perform new Install instead of rollback.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
if you rollback to U2 does it not work....

Rollback or re-installation ?
0
Senior IT System EngineerIT ProfessionalAuthor Commented:
Somehow the rollback doesn't revert the build version so the problem was still exist.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I'm using rollback in the manual sense of the word

REINSTALL ESXi 5.1 U1.

If that does not work, this issue is one for HP and VMware Support Escalation as the hardware in on the HCL. You are at the correct firmware and driver version, something is not correct.
0
Senior IT System EngineerIT ProfessionalAuthor Commented:
Ah yes, it works after re-installing the ESXi 5.1 Update 2 manually therefore I just have to manually configuring the vDS and other settings.

It's on VMware advanced team now. I'll let you know as well how it goes.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
We've seen issues before with drivers and firmware, and the Benet2 and Emulex driver issues with hardware.
0
Senior IT System EngineerIT ProfessionalAuthor Commented:
Thanks Andrew.

Somehow the issue as due to the ESXi NIC driver updated to the latest higher edition by using HP ESXi custom ISO image.
down grading to the earlier version as per HP VIB recipe makes it working again.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.