We help IT Professionals succeed at work.

VMWare ESXi Host - occasionally  stalls when loading VMWare environment at startup.

Hardware: Dell Poweredge R630 - running latest BIOS and firmware.  SD card boot;   Accessories: CD drive;  USB port;  [NO Hard Drive]

Operating System: VMWare 6.5.0    5310538


Problem with sticky boot up when restarting a Dell Poweredge R630 VMWare ESXi host which has no hard drive. [i.e. boots up from onboard SD card] It can take 3 attempts to restart the server which appears to successfully boot through all the hardware configurations but stalls when loading the VMWare environment.  I have spoken with Dell who say that this is a VMWare problem. I am not an expert on this but since the server is running at the moment I was hoping that I might be able to copy the current configuration onto another medium and boot up from it in order to eliminate the SD cards.   Does anybody know how to do this?

If the problem is with VMWare how do I diagnose the likely cause? Does anybody know the best course of action?

Thanks to anybody for taking the time to look at this.
Comment
Watch Question

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
Where does it stall, and what do you mean by stall, do you mean it stops ?

e.g. on BOOT off SD card it stops ?

Are you using SD cards provided by Dell ? Good Quality SD cards, all our Dell R630s use Mirrored SD cards, and we do not have issues with ESXi Booting.

Are you using the DELL OEM version of ESXi ?

I would not waste time backing up configuration, and restoring to a new config at present.

Remove the SD card, replace with a different brand, SanDisk, Kingston, HP - these are brands we use.

Install ESXi again, and test BOOT.

You can backup the current configuration, and restore to the same version of ESXi.

 Using the vSphere CLI, qhich you have to install.

vicfg-cfgbackup --server=ESXi_host_IP_address --username=root -s output_file_name

Open in new window


see here

https://kb.vmware.com/s/article/2042141
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
You don't say whether it keeps running once fully booted or whether you have other identical servers without problems but that is likely as you have no local disks. Andrew is probably right as usual, once the OS is read into RAM that SD card doesn't really so anything, you can pull it out and VMware will keep running. You can also use a USB stick to test/boot if you can't get to the SD card.

Author

Commented:
Thanks for the response to this question.  Basically the boot up process stops during the VMWare load/boot sequence. e.g. on one occasion it stopped when loading vmtoolsd and I left it overnight but it never progressed from there. It seems to boot through the Dell hardware config screen okay stopping at an apparent random point during the  VMWare environment load.  [Note: it normally loads correctly after 2 or 3 startup attempts]
I am using the original Dell SD cards that came with the new server. We are using VMWare's ESXi not a Dell OEM version.
One thought - does this sound like an SD card problem or is it more likely to be a VMWare problem?

It seems that installing the VMWare Power CLI is something I need to do. However, please forgive my ignorance but one thing that puzzles me is that in my environment the  vcentre is a VMWare Linux Vcenter virtual appliance. What machine do I load the Power CLI onto? Do I need to install it onto the appliance or can I install it on any virtual machine in the vmware environment and connect  that way?

This is the procedure I was thinking of adopting

Copy config from SD card to a USB stick

Swap the SD cards in the server

Restart the Server and load the VMWare ESXi op system?

Copy over the original saved configuration

or is there a simpler method?

Thanks
VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017
Commented:
Okay a few remarks.

1. Stopping at the BOOT sequence is a hardware issue. (*but could be storage issue! - you don't mention storage)

either BOOTABLE media e.g. in this case your SD card,

OR Server HARDWARE - if this is a new server - get this checked. - make sure you are using Dell latest firmware on your server, and ESXi is correct version on on the HCL.

2. Now, and DELL should have spotted this - you must and should use the DELL OEM ESXi version, not vanilla version.


It seems that installing the VMWare Power CLI is something I need to do. However, please forgive my ignorance but one thing that puzzles me is that in my environment the  vcentre is a VMWare Linux Vcenter virtual appliance. What machine do I load the Power CLI onto? Do I need to install it onto the appliance or can I install it on any virtual machine in the vmware environment and connect  that way?

You install vSphere CLI (not Power CLI) on a workstation - Windows!

This is the procedure I was thinking of adopting

Copy config from SD card to a USB stick

Swap the SD cards in the server

Restart the Server and load the VMWare ESXi op system?

Copy over the original saved configuration

or is there a simpler method?




We would not do any of that, waste of time!

1. Download the DELL OEM ESXi version.

2. Obtain a new SD card.

3. Install DELL OEM ESXi version.

4. BOOT server - does it BOOT.

If it does not BOOT correctly, back to DELL as this is a hardware fault with server!!!

The above should take you 11 minutes to prove!

we would do the above procedure to prove hardware and OEM ESXi are working as expected, and then move onto to config... (which could be cause of BOOT!)

Author

Commented:
Thanks Andrew. Set me on the right course.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
no problems, good luck...
Sebastian TalmonSystem Administrator
CERTIFIED EXPERT
Commented:
some months ago DELL had a firmware-Update for the integrated dual SD Module (IDSDM), that is not listed as available firmware upgrade in iDRAC and LifecycleController. This Firmware-Update fixed problems on R630 and other DELL G13/G14 servers with randomly failing SD cards.

Maybe also worth checking the IDSDM Firmare (listed on the overview page of the iDRAC) - the old firmware is version 0.15, the current version is 1.7

https://www.dell.com/support/home/en/us/debsdt1/drivers/driversdetails?driverid=ff4wy

If you are still on version 0.15, it may be worth upgrading.

It is not installable through iDRAC or LifecycleController - you have to build a custom Linux ISO with DELL RepositoryManager, or ask DELL Support, some supporters have a prebuild ISO on dell.box.com for update to 1.7

Author

Commented:
Thanks Sebastian. That was really helpful.