Server stuck on 'vmkibft loaded successfully 5.5'?

Hi,

I had to restart a host server of mine and it's now stuck with this error message when it boots ... I then tried to boot the server from the HP version of the ESXi 5.5 boot CD but it hangs on that message there too!? That ISO was only Update1 though ... I've read there is an Update2 that addresses this problem. But it doesn't exist as HP version (yet)?

So what else can I do ... ?

Thank you very much!
XeronimoAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Patch your server using the latest and last Updates (e.g. Update 2 or later ) from VMware, but use the

update flag on the command line, this will just update the drivers and files, and not overwrite any of the HP stuff.

esxcli software update -d /vmfs/volumes/datastore1/<update zip file>

see an example in my EE Article but use the command above

HOW TO: Upgrade from VMware vSphere Hypervisor ESXi 5.1 to VMware vSphere Hypervisor ESXi 5.5 for FREE
0
XeronimoAuthor Commented:
Thanks but I don't have access to the console since the host is stuck while booting? Or have I missed something?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Can you boot into Recovery Mode ? (Shift and R at ESXi boot)

have you updated this server before ? because you can rollback as above to it....

Yes, this issue is caused by the driver Emulex be2iscsi.
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

XeronimoAuthor Commented:
Haven't tried recovery mode ... I will.

I don't remember if I had updated this one before ... if I did then it work fine, at least after the first reboot.

(I did not have to reboot it in this case because of an update)
0
XeronimoAuthor Commented:
Also, that host has only a local datastore ... so how could I save this ZIP on it if I can't access it? Or could I copy it from a CD/stick?
0
XeronimoAuthor Commented:
I've pressed Shift-R but I'm not sure I've ended up in the recovery console ... ? The only option I have is to do a rollback to 5.1? (also, this is a stand-alone host, not part of a cluster, and not part of my 3-licence vSphere system)
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, you can rollback to 5.1, and then perform an Upgrade to the correct "fixed" version.

We had an issue the other week, 5.5 broke on a host, we recovered to 5.1, and then applied 5.5....fixed..
0
XeronimoAuthor Commented:
Just to be sure ... I rollback to 5.1 and then boot from the HP ESXi 5.5 boot CD again and upgrade?
0
XeronimoAuthor Commented:
So I've chosen the rollback to 5.1 now and it seems to be stuck at the same place now!?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
The bug must also exist in 5.1, but it was only reported in 5.5, with the Emulex be2iscsi driver.

You must use the new ISO Boot Disk, which does not contain the bug.

You need

ESXi 5.5 Update 1 Driver Rollup 2
0
XeronimoAuthor Commented:
Aaaargh, so now I've booted from the 5.5U1_Rollup2 ISO and it still hangs at that effing same place!? Can't, or shouldn't, I just delete the partition where the hypervisor resides and reinstall the whole thing? The datastore won't be affected and I can use it later on again, no?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
The problem is, if the installation is hanging, you'll not be able to install.

How long have you waited ?
0
XeronimoAuthor Commented:
This morning (17:37 here now) I've waited for a couple of hours in the end ... after the reboots I waited up to, I don't know for sure, 20 minutes?

I'll boot it again from the 5.5 CD and I'll check tomorrow morning where it's at ... probably still at the 'vmkibft loaded successfully' -.-

So if that doesn't work then what are my options? I should be able to do a clean install of the hypervisor, no? Or will not even the install program start now? But why did it in the past then ... ? Why wouldn't I be able to reinstall the hypervisor? And wouldn't that mean that this server is dead as far as ESXi is concerned!?

Thanks for your help ...
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
This is a good question, what has changed since, you last restarted the server?

Added an iSCSI SAN ?

Changed firmware ?
0
XeronimoAuthor Commented:
Nothing really ... I think the iSCSI connections (which the client uses) have been there before ...

I was thinking though about updating the firmware now? Should I do that or not?

But if all of this doesn't help does that mean I can't use that server with VMWare anymore ... ?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
What is the server?

What is the SAN ?

Is it on the HCL ?

I would recommend updating to the lastest firmwares available for the server.

It is very odd, why this server no longer boots correctly, and it looks like this was an issue with the Emulex driver, but it's possible it could be another fault.
0
XeronimoAuthor Commented:
It's an HP DL380G7.

As for the SAN, the server just uses a bunch of internal HDs. The virtual server running on this host is doing the backups via Veeam to the local storage and then makes a copy of the backup to a Qnap NAS.

I'll update the firmware then.
0
XeronimoAuthor Commented:
Argh, I've tried to update the firmware now but when starting from the Rompaq CD the screen says 'starting Caldera DOS' and then hangs ... seems like this server has been cursed!
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I would open a support call with HP, could be a server hardware fault.
0
XeronimoAuthor Commented:
Ok, I'm going to check that.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
This could be a "red herring" we've been chasing, with this lock-up, thinking it's the iSCSI Bug, when it's not....

hence why on restarting a working host, you've had this error, and all attempt to install ESXi has also caused issues.

Your server is on the HCL for ESXi, so should install with no issues.

Get a support call into HP!
0
XeronimoAuthor Commented:
update: I've tried booting from the ESXi 5.1 U2 CD again and now it's stuck(?) at the 'nfsclient loaded successfully'. But I give it a bit more time ... Let's hope it'll be able to finally load this!
0
XeronimoAuthor Commented:
screenshotI didn't know you could look at the log with Alt-F12 ... so I did now and this is what I saw ...

Last line says: 'no routers on vmk8. Giving up.' What's that supposed to mean?

And in the upper right corner it says 'stopped'?
0
XeronimoAuthor Commented:
Problem solved!

The reason it got stuck was because of two SSD disks in the server ... I've removed them and now the ESXi server boots again as it should.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Hardware issue with your SSDs, are these HP approved SSDs ?

or Consumer?
0
XeronimoAuthor Commented:
HP approved. I'm not what the issue is ... we only had them for testing purposes anyway. I simply keep them out of the system for now ...
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Get a support call raised with HP.

Very odd, it stops booting.....especially after they've been installed.

SSD firmware maybe?
0
XeronimoAuthor Commented:
It was the solution ...
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.