VMs This virtual machine failed to become vSphere HA Protected and HA may not attempt to restart it after a failure

Getting the error on a series of VMs This virtual machine failed to become vSphere HA Protected and HA may not attempt to restart it after a failure.  after performing maintenance on my ESXi 6.5 hosts.. Now..  I booted the VMs before booting up vSphere appliance.  Is that the problem?  What should I do?
gopher_49Asked:
Who is Participating?
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
The host which is having difficulty, deploying the FDM agent, needs uninstalling....

and then re-enable HA, it will then deploy the new FDM/HA Agent to the Host
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Reconfigure HA, or disable and enable HA
0
 
gopher_49Author Commented:
I tried disable and enable.. That didn't work..  I also see an error say it can't install the agent to the ESXi host.
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Well that is your issue! (That host is not going to do any HA!)

before you go any further investigating WHY, the FDM Agent (HA agent) cannot be installed on this Host...

Is this host using OEM ESXi edition ?

Is it using SD flash card, or USB flash card ?

Try Re-configuring a few times, try enable and disable...

At what percentage does it fail ?

You are going to have to start a deep dive on the host via console or SSH
0
 
gopher_49Author Commented:
It's a Dell image..  v6.5..  I'm using SD Flash...  I've tried over 4 x times to reconfigure and it fails consistently..  I have the fdm.log open and reviewing now.
0
 
gopher_49Author Commented:
I have no idea to what I'm reading and/or looking for in the fdm.log file..
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, all the signs are there.....

1. OEM ESXi version
2. SD flash.

Your storage space is FULL..... - very common issues!

The HA Agent is pushed out from vCenter Server, when a new host is added, of vCenter Server is upgraded....

It pushes the new VIB to the server, as a tgz, then extracts it, and then installs it...

BUT, it does not check space available before it does this, so likely you do not have enough space for the copy or extraction, so it fails.

check space.... you may have to remove current HA VIB from bootbanks, before proceeding.
0
 
gopher_49Author Commented:
I ran df - h .  I see 3 x vfat volumes.  93 MBs, 101 MBs, and 80MBs Free.  Each original size is approx 250 MBs.  Which do I purge from?  What do I purge?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
can you screenshot the results
0
 
gopher_49Author Commented:
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
can i have a df -h
0
 
gopher_49Author Commented:
That is a df-h output
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
vdf
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
remove the old vib manually...

esxcli software vib remove -n vmware-fdm

We have seen ESXi installations which are corrupted, did you upgrade this ?

also see here

https://kb.vmware.com/s/article/2056299
0
 
gopher_49Author Commented:
Below is the output.. What's odd is all 3 x ESXi hosts have the same symptoms..

[root@fx2host1-ha-esxi1:~] vdf
Tardisk              1k-blocks      Used
sb.v00                  174036    174033
s.v00                   318444    318443
bnxtnet.v00                416       412
ima_be2i.v00              2448      2444
scsi_be2.v00               804       801
net_i40e.v00               424       420
net_igb.v00                320       319
net_ixgb.v00               532       530
nimble_n.v00               572       571
nimble_p.v00               304       301
ima_qla4.v00              1356      1352
misc_cni.v00                24        21
net_bnx2.v00               316       315
net_bnx2.v01              2116      2112
net_cnic.v00               148       147
net_qlcn.v00              1148      1147
qlnative.v00              2680      2678
scsi_bnx.v00               352       349
scsi_bnx.v01               268       267
scsi_qla.v00               532       531
ata_liba.v00               296       294
ata_pata.v00                48        45
ata_pata.v01                32        28
ata_pata.v02                36        32
ata_pata.v03                36        32
ata_pata.v04                40        36
ata_pata.v05                36        32
ata_pata.v06                32        28
ata_pata.v07                36        32
block_cc.v00                88        85
char_ran.v00                52        48
ehci_ehc.v00                92        91
elxnet.v00                 596       594
hid_hid.v00                 64        60
i40en.v00                  444       440
igbn.v00                   328       327
ipmi_ipm.v00                44        40
ipmi_ipm.v01                84        80
ipmi_ipm.v02               104       100
ixgben.v00                 412       411
lpfc.v00                  2320      2317
lsi_mr3.v00                328       325
lsi_msgp.v00               488       485
lsi_msgp.v01               496       493
misc_dri.v00              1312      1309
mtip32xx.v00               252       248
ne1000.v00                 632       629
nenic.v00                  208       204
net_cdc_.v00                28        25
net_e100.v00               312       310
net_e100.v01               352       351
net_enic.v00               172       168
net_fcoe.v00                80        79
net_forc.v00               128       125
net_libf.v00                76        75
net_mlx4.v00               356       353
net_mlx4.v01               240       236
net_nx_n.v00              1952      1949
net_tg3.v00                316       315
net_usbn.v00                56        53
net_vmxn.v00               108       104
nhpsa.v00                  560       557
nmlx4_co.v00               668       665
nmlx4_en.v00               692       688
nmlx4_rd.v00               256       252
nmlx5_co.v00               556       552
ntg3.v00                   112       108
nvme.v00                   256       254
nvmxnet3.v00               160       156
ohci_usb.v00                64        60
pvscsi.v00                  72        68
qedentv.v00               4184      4180
qfle3.v00                 1936      1933
qflge.v00                  496       493
sata_ahc.v00                84        80
sata_ata.v00                60        59
sata_sat.v00                68        65
sata_sat.v01                48        44
sata_sat.v02                48        44
sata_sat.v03                36        32
sata_sat.v04                32        28
scsi_aac.v00               176       174
scsi_adp.v00               448       444
scsi_aic.v00               296       292
scsi_fni.v00               240       236
scsi_hps.v00               212       209
scsi_ips.v00               104       100
scsi_isc.v00                44        42
scsi_lib.v00               212       211
scsi_meg.v00                96        92
scsi_meg.v01               176       175
scsi_meg.v02                96        93
scsi_mpt.v00               460       457
scsi_mpt.v01               504       500
scsi_mpt.v02               428       424
shim_isc.v00                16        15
shim_isc.v01                16        15
shim_lib.v00                44        43
shim_lib.v01                44        43
shim_lib.v02                24        20
shim_lib.v03                24        20
shim_lib.v04                12         9
shim_lib.v05                12         9
shim_vmk.v00               220       217
shim_vmk.v01               232       228
shim_vmk.v02               236       232
uhci_usb.v00                64        60
usb_stor.v00               164       160
usbcore_.v00               316       312
vmkata.v00                 772       771
vmkplexe.v00                48        46
vmkusb.v00                1464      1463
vmw_ahci.v00               264       262
xhci_xhc.v00               236       232
emulex_e.v00                28        25
weaselin.t00              4944      4941
esx_dvfi.v00               456       452
esx_ui.v00               12384     12381
lsu_hp_h.v00               104       101
lsu_lsi_.v00               264       261
lsu_lsi_.v01               480       477
lsu_lsi_.v02               264       261
lsu_lsi_.v03               560       557
native_m.v00               748       744
rste.v00                   824       820
vmware_e.v00               112       108
vmware_f.v00             25404     25400
vsan.v00                 39240     39237
vsanheal.v00              7392      7388
vsanmgmt.v00             12704     12701
dell_con.v00                 4         3
xorg.v00                  3092      3091
imgdb.tgz                 1572      1570
state.tgz                   32        30
-----
Ramdisk              1k-blocks      Used Available Use% Mounted on
root                     32768        24     32744   0% --
etc                      28672       292     28380   1% --
opt                      32768         0     32768   0% --
var                      49152       480     48672   0% --
tmp                     262144         8    262136   0% --
iofilters                32768         0     32768   0% --
hostdstats             1334272      6680   1327592   0% --
[root@fx2host1-ha-esxi1:~]
0
 
gopher_49Author Commented:
In regards to the 'remove the old vib manually...' post... Here's when it messed up.  We bought all ESXi server down.. Maintenance mode first..  The powered off..  When they where powered off we stacked our switches which results in something we didn't know would happen.  All switch ports changed their VLAN assignment.  So..  We powered on all ESXi servers..  The servers came up really slow for they couldn't connect to iSCSI and other network properly.  We noticed what happens..  Fixed all vlans.. Then shut off the servers properly via their https GUI..  Then I brought them up.. They booted in a normal speed and everything seemed good except this.. Now one more thing.  the vcenter server has two instances of it when I powered it on.  I tried to power it on via the HTTPS GUI.  It asked if I copied it.  I said yes..  It boots normally from what I can tell.
0
 
gopher_49Author Commented:
So.. Do I run 'esxcli software vib remove -n vmware-fdm' and then try to disable/enable HA?
0
 
gopher_49Author Commented:
Nevermind.. I see the steps in the KB.  I'll try to schedule maintenance tomorrow morning and handle.  I'll update you then.
0
 
gopher_49Author Commented:
I guess I'll just power down all VMs and put all hosts in maintenance mode...  And then follow theses steps.. But..  I find it hard to believe ALL agents are corrupt?  If one is corrupt will that cause all to mess up since they are part of the same cluster?
0
 
gopher_49Author Commented:
I'm unable to power on two of my VMs it seems.. Luckily these aren't important.  I get the below message. Do I need to disable HA for these VMS? Or just disable HA for now?

The host is reporting errors in its attempts to provide vSphere HA support.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Turn off VMware HA, as it's currently broken!
0
 
gopher_49Author Commented:
I had a feeling that would be easier...  It's so odd that all three hosts are messed up.  I have seen in the past where host hang and / or have issues booting when they can't properly see the iSCSI network. Didn't think it would actually break a module and / or all module instances on each host. The whole fact that there were two different vcenter servers was really odd. I noticed it available in two different host. I double clicked one and it asked if it was copied.  I selected that option.  It seemed the boot just fine. I have a ticket open with VMWare also. I think I'm going to hold off making any changes until I talk to them.  I schedule the changes Friday evening.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I had a feeling that would be easier...  It's so odd that all three hosts are messed up.  I have seen in the past where host hang and / or have issues booting when they can't properly see the iSCSI network.

thats normal

Didn't think it would actually break a module and / or all module instances on each host.

that doesn't but deploying a new HA Agent will!
0
 
gopher_49Author Commented:
It booted in a state of being broken.  This is why I disabled and reenabled HA.  Or attempted to.  I almost wondered if there where two different vcenter instances.  It was very odd that I saw an instance on two different hosts
0
 
gopher_49Author Commented:
I had VMWare look at it..  A reboot of the vcenter appliance fixed it?!  Hahahahaha...  I guess I got lucky?
0
 
gopher_49Author Commented:
Normally this what you do, however, in my instance I simply had to reboot the vcenter appliance.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.