HA failing on two VMware hosts

We have a 4-host vSphere cluster. (ESXi at 6.0. vCenter at 6.5.)
To expand vSAN, we had to reboot all hosts. Now HA is failing on 2 of them. (vSAN is healthy after initial congestion.)
- Have tried "Reconfigure for vSphere HA" many times.
- After putting in Maintenance mode, removed a problem host from inventory. Later added back to the cluster. Made no difference.
- Rebooting has not helped.

Message: 'Cannot complete the operation due to an incorrect request to the server'
Event log says: 'vSphere HA agent for this host has an error: vSphere HA agent cannot be installed or configured.'

Have looked at many KB articles including #2056299. It talks about checking fdm-installer.log file but no such file exists on any of the host, HA working or not. Have not changed configuration of the hosts in at least 6 months, other than expanding vSAN.

Please advise. Thanks.
Who is Participating?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
suggests it's copying and failing...

check storage space is FREE on the SD card, common problem, runs out of storage space, and cannot copy the new HA Agent (VIB) archive, and unpack, and run on the SD card, so fails.

you also using DELL OEM ESXi version ?

esxcli software vib remove -n vmware-fdm

Open in new window

you can read it here

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Common issue, and nothing to do with vSAN.

When you Reconfigure for vSphere HA, when does it fail, percentage ?

How have you installed ESXi, on a USB flash drive or SD card ?

the log should be on the server ?

Have you tried uninstalling the HA (fdm) vib agent, and trying again ?
AkulshAuthor Commented:
Dear Andrew,

On both hosts, it fails at 26%.
Hosts are installed on SD card.
FDM.log was no help.
How do I uninstall HA vib agent -- which KB describes the procedure?
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

AkulshAuthor Commented:
Dear Andrew,

The problem is Fixed! So happy I am.
All I had to do is run:
esxcli software vib remove –n vmware-fdm
After that "Reconfigure for vSphere HA" took many minutes but it worked.

Few points
- KB2056299 is very poorly written. It talks about fdm-installer.log file which is nowhere to be found. It also mentions dependency created by a third party VIB, which supposedly had to be removed first. No such need.
- In our case, problem probably happened because we had to reinstall one ESXi (we used Dell's ISO for all hosts) server due to failed SD card controller. This newly installed host became HA master and did not let 2 hosts join HA because their vib had older date, though same version. Strangely and thankfully, one old host with old file had no issue.
- I had found this posting which helped: https://tinyurl.com/y96ao44e

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
your SD card was full! it happens with the ESXi OEM versions.

and the problem is, there is no space to

1. copy new HA Agent from vCenter Server to host /tmp
2. Extract it to /tmp
3. Execute it and install it to bootbank!

very common issue, and can occurs everytime you update your vCenter Server in the future....

so write a document, so next time it happens you know what to do!
AkulshAuthor Commented:
Andrew knows more than most VMware Support engineers...
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Thanks for your kind words!
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.