Reconfigure HA: operation timed out

Hi, I am using ESXi 6.0
I just upgraded VMware vCenter Server Applianceto 6.0.0.30300
My cluster has 7 nodes, 2 of them are reporting issue with HA.
I tried to put one host in "maintenance mode", reboot, and "Reconfigure for HA", but operation fails with "operation timed out".
What can I do?
Thank you very much
Armitage318Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Check logs on host.

Check space on host OS installation, is this flash drive ? is this full ?

you may need to un-install ha vib, and re-install
0
Armitage318Author Commented:
Hi Andrew,

I have no space issue. I tried to install upgrade on affected host, and I encountered several problems, solved one by one by VMware KBs.
Now I am stuck with:

"The host returns esxupdate error code:15"

I already did steps suggested in :

https://kb.vmware.com/s/article/2030665


Those are my error:

2018-01-09T16:14:40Z esxupdate: 36685: HostImage: DEBUG: Total of 5504 bytes were written.
2018-01-09T16:14:40Z esxupdate: 36685: HostImage: INFO: Constructing payloads from tardisks for vib VMware_bootbank_cpu-microcode_6.0.0-0.0.2494585
2018-01-09T16:14:40Z esxupdate: 36685: HostImage: ERROR: Failed to send vob install.stage.error: [Errno 27] File too large
2018-01-09T16:14:40Z esxupdate: 36685: esxupdate: ERROR: An esxupdate error exception was caught:
2018-01-09T16:14:40Z esxupdate: 36685: esxupdate: ERROR: Traceback (most recent call last):
2018-01-09T16:14:40Z esxupdate: 36685: esxupdate: ERROR:   File "/usr/sbin/esxupdate", line 238, in main
2018-01-09T16:14:40Z esxupdate: 36685: esxupdate: ERROR:     cmd.Run()
2018-01-09T16:14:40Z esxupdate: 36685: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esx5update/Cmdline.py", line 148, in Run
2018-01-09T16:14:40Z esxupdate: 36685: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 250, in InstallVibsFromSources
2018-01-09T16:14:40Z esxupdate: 36685: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 356, in _installVibs
2018-01-09T16:14:40Z esxupdate: 36685: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 399, in _validateAndInstallProfile
2018-01-09T16:14:40Z esxupdate: 36685: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/HostImage.py", line 703, in Stage
2018-01-09T16:14:40Z esxupdate: 36685: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/HostImage.py", line 534, in _get_tardisk_payload
2018-01-09T16:14:40Z esxupdate: 36685: esxupdate: ERROR: InstallationError: ("set(['VMware_bootbank_sata-sata-sil_2.3-4vmw.600.0.0.2494585', 'VMware_bootbank_scsi-ips_7.12.05-4vmw.600.0.0.2494585', 'VMware_bootbank_ehci-ehci-hcd_1.0-4vmw.600.3.69.5572656', 'VMware_bootbank_esx-xserver_6.0.0-3.76.6856897', 'BRCM_bootbank_net-tg3_3.137l.v60.1-1OEM.600.0.0.2494585', 'VMware_bootbank_sata-sata-promise_2.12-3vmw.600.0.0.2494585', 'VMware_bootbank_ipmi-ipmi-devintf_39.1-5vmw.600.3.79.6921384', 'VMware_bootbank_sata-sata-nv_3.5-4vmw.600.0.0.2494585', 'VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-1vmw.600.0.0.2494585', 'VMware_bootbank_scsi-mptsas_4.23.01.00-9vmw.600.0.0.2494585', 'EMU_bootbank_lpfc_10.6.126.0-1OEM.600.0.0.2159203', 'VMware_bootbank_esx-ui_1.22.0-6282878', 'Avago_bootbank_lsi-msgpt3_10.00.03.00-1OEM.600.0.0.2768847', 'VMware_bootbank_esx-dvfilter-generic-fastpath_6.0.0-3.79.6921384', 'VMware_bootbank_
2018-01-09T16:14:40Z esxupdate: vsan_6.0.0-3.79.6766495', 'VMware_bootbank_scsi-adp94xx_1.0.8.12-6vmw.600.0.0.2494585', 'QLogic_bootbank_scsi-bnx2i_2.712.10.v55.4-1OEM.550.0.0.1331820', 'VMware_bootbank_uhci-usb-uhci_1.0-3vmw.600.0.0.2494585', 'QLogic_bootbank_misc-cnic-register_1.712.10.v55.2-1OEM.550.0.0.1331820', 'VMware_bootbank_cpu-microcode_6.0.0-0.0.2494585', 'QLogic_bootbank_net-bnx2x_2.712.10.v55.4-1OEM.550.0.0.1331820', 'Avago_bootbank_lsi-mr3_6.903.82.00-1OEM.600.0.0.2768847', 'VMware_bootbank_scsi-fnic_1.5.0.45-3vmw.600.0.0.2494585', 'VMware_bootbank_sata-ahci_3.0-26vmw.600.3.57.5050593', 'EMU_bootbank_scsi-be2iscsi_10.6.150.3-1OEM.600.0.0.2159203', 'VMware_bootbank_vsanhealth_6.0.0-3000000.3.0.3.79.6769065', 'VMware_bootbank_misc-drivers_6.0.0-3.79.6921384', 'VMware_bootbank_rste_2.0.2.0088-4vmw.600.0.0.2494585', 'VMware_bootbank_block-cciss_3.6.14-10vmw.600.0.0.2494585', 'VMware_bootbank_xhci-xhci_1.0-3vmw.600.3.6
2018-01-09T16:14:40Z esxupdate: 9.5572656', 'MEL_bootbank_nmlx4-en_3.1.0.0-1OEM.600.0.0.2348722', 'Intel_bootbank_net-i40e_1.3.38-1OEM.550.0.0.1331820', 'VMware_bootbank_esx-base_6.0.0-3.79.6921384', 'VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-3vmw.600.3.57.5050593', 'VMware_bootbank_ata-pata-sil680_0.4.8-3vmw.600.0.0.2494585', 'VMware_bootbank_net-forcedeth_0.61-2vmw.600.0.0.2494585', 'VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-3vmw.600.3.57.5050593', 'VMware_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.600.0.0.2494585', 'VMware_bootbank_ata-pata-via_0.3.3-2vmw.600.0.0.2494585', 'VMware_bootbank_emulex-esx-elxnetcli_10.2.309.6v-0.0.2494585', 'VMWARE_bootbank_mtip32xx-native_3.8.5-1vmw.600.0.0.2494585', 'VMware_bootbank_sata-ata-piix_2.12-10vmw.600.0.0.2494585', 'VMware_bootbank_net-mlx4-core_1.9.7.0-1vmw.600.0.0.2494585', 'VMware_bootbank_net-enic_2.1.2.38-2vmw.600.0.0.2494585', 'VMware_bootbank_net-mlx4-en_1.9.7.0-1vmw.600.0
2018-01-09T16:14:40Z esxupdate: .0.2494585', 'VMware_bootbank_ata-pata-cmd64x_0.2.5-3vmw.600.0.0.2494585', 'QLogic_bootbank_net-cnic_2.712.10.v55.4-1OEM.550.0.0.1331820', 'VMware_bootbank_net-e1000e_3.2.2.1-2vmw.600.3.57.5050593', 'QLogic_bootbank_net-bnx2_2.2.5g.v55.2-1OEM.550.0.0.1331820', 'VMware_bootbank_lsu-lsi-mpt2sas-plugin_1.0.0-4vmw.600.1.17.3029758', 'QLogic_bootbank_scsi-qla4xxx_644.55.36.0-1OEM.550.0.0.1331820', 'VMware_bootbank_ata-pata-atiixp_0.4.6-4vmw.600.0.0.2494585', 'VMware_bootbank_lsu-lsi-mptsas-plugin_1.0.0-1vmw.600.0.0.2494585', 'VMware_bootbank_ohci-usb-ohci_1.0-3vmw.600.0.0.2494585', 'VMware_bootbank_ipmi-ipmi-msghandler_39.1-5vmw.600.3.79.6921384', 'VMware_bootbank_ata-pata-serverworks_0.4.3-3vmw.600.0.0.2494585', 'VMware_bootbank_net-nx-nic_5.0.621-5vmw.600.0.0.2494585', 'VMware_bootbank_sata-sata-sil24_1.1-1vmw.600.0.0.2494585', 'VMware_bootbank_ata-pata-amd_0.3.10-3vmw.600.0.0.2494585', 'VMware_boot
2018-01-09T16:14:40Z esxupdate: bank_scsi-megaraid2_2.00.4-9vmw.600.0.0.2494585', 'MEL_bootbank_nmlx4-core_3.1.0.0-1OEM.600.0.0.2348722', 'Dell_bootbank_dell-configuration-vib_6.0-2A00', 'VMware_bootbank_ata-pata-pdc2027x_1.0-3vmw.600.0.0.2494585', 'EMU_bootbank_elxnet_10.6.126.0-1OEM.600.0.0.2159203', 'Intel_bootbank_net-igb_5.3.1-1OEM.550.0.0.1331820', 'VMware_bootbank_net-e1000_8.0.3.1-5vmw.600.0.0.2494585', 'VMware_bootbank_nmlx4-rdma_3.0.0.0-1vmw.600.0.0.2494585', 'VMware_bootbank_qlnativefc_2.1.50.0-1vmw.600.3.57.5050593', 'VMware_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.600.0.0.2494585', 'QLogic_bootbank_net-qlge_3.0.0.59-1OEM.550.0.0.1331820', 'VMware_bootbank_scsi-aacraid_1.1.5.1-9vmw.600.0.0.2494585', 'VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-4vmw.600.3.57.5050593', 'VMware_bootbank_sata-sata-svw_2.3-3vmw.600.0.0.2494585', 'VMware_bootbank_scsi-mptspi_4.23.01.00-9vmw.600.0.0.2494585', 'QLogic_bootbank_ima-qla4xxx_500.2.01.
2018-01-09T16:14:40Z esxupdate: 31-1vmw.0.3.100400', 'VMware_bootbank_net-vmxnet3_1.1.3.0-3vmw.600.2.43.4192238', 'VMware_bootbank_nvme_1.2.0.27-4vmw.550.0.0.1331820', 'VMware_bootbank_esx-tboot_6.0.0-3.57.5050593', 'VMware_bootbank_scsi-mpt2sas_19.00.00.00-1vmw.600.0.0.2494585', 'VMware_bootbank_scsi-aic79xx_3.1-5vmw.600.0.0.2494585', 'QLogic_bootbank_net-qlcnic_6.1.191-1OEM.600.0.0.2494585', 'VMW_bootbank_scsi-megaraid-sas_06.805.56.00-1OEM.600.0.0.2494585', 'QLogic_bootbank_scsi-bnx2fc_1.712.10.v55.2-1OEM.550.0.0.1331820', 'Intel_bootbank_net-ixgbe_4.1.1.1-1OEM.550.0.0.1331820', 'VMware_bootbank_vmware-fdm_6.0.0-7037393', 'VMware_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.600.0.0.2494585', 'VMware_bootbank_scsi-hpsa_6.0.0.44-4vmw.600.0.0.2494585', 'VMware_locker_tools-light_6.0.0-3.76.6856897', 'EMU_bootbank_ima-be2iscsi_10.6.150.3-1OEM.600.0.0.2159203'])", 'Cannot locate source for payload uc_amd of VIB VMware_bootbank_cpu-m
2018-01-09T16:14:40Z esxupdate: icrocode_6.0.0-0.0.2494585 ')
2018-01-09T16:14:40Z esxupdate: 36685: esxupdate: DEBUG: <<<
2018-01-09T16:14:41Z esxupdate: 36703: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetTimeout']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2018-01-09T16:14:41Z esxupdate: 36703: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetRetries']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2018-01-09T16:14:41Z esxupdate: 36703: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetRateLimit']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2018-01-09T16:14:41Z esxupdate: 36703: esxupdate: INFO: ---
Command: scan

Open in new window

0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
okay, so I'm a little confused, how did you get on with HA Agent  ?

so this has occurred since a host upgrade ?
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

Armitage318Author Commented:
Hi, after VCSA upgrade, I got HA issues on 2 hosts (total cluster: 7).
I tried to "reconfigure HA" without success.
So, I just tried to put in maintenance mode 1 host and tried installing updates through VUM.
I am unable to complete this step..

Further info:

 esxcli software profile get
(Updated) Dell-ESXi-6.0U2-3620759-A00
   Name: (Updated) Dell-ESXi-6.0U2-3620759-A00
   Vendor: Dell
   Creation Time: 2018-01-09T10:21:19
   Modification Time: 2018-01-09T10:21:19
   Stateless Ready: False
   Description: 
      
      2018-01-09T10:21:19.601278+00:00: The following VIBs are
      installed:
        vmware-fdm    6.0.0-7037393
      ----------
      (Original Vendor):Dell
      2017-04-04T16:18:24.806437+00:00: The following VIBs are
      installed:
        ehci-ehci-hcd 1.0-3vmw.600.2.52.4600944
        esx-ui        1.14.0-4940836
        xhci-xhci     1.0-3vmw.600.2.52.4600944
        sata-ahci     3.0-26vmw.600.3.57.5050593
        tools-light   6.0.0-3.57.5050593
        vsanhealth    6.0.0-3000000.3.0.3.58.5224738
        net-vmxnet3   1.1.3.0-3vmw.600.2.43.4192238
        vsan  6.0.0-3.58.5224737
        net-e1000e    3.2.2.1-2vmw.600.3.57.5050593
        qlnativefc    2.1.50.0-1vmw.600.3.57.5050593
        lsu-lsi-lsi-mr3-plugin        1.0.0-3vmw.600.3.57.5050593
        misc-drivers  6.0.0-3.57.5050593
        esx-tboot     6.0.0-3.57.5050593
        lsu-hp-hpsa-plugin    2.0.0-4vmw.600.3.57.5050593
        lsu-lsi-megaraid-sas-plugin   1.0.0-3vmw.600.3.57.5050593
        esx-base      6.0.0-3.58.5224934
      ----------
      (Original Vendor):Dell
      2016-12-16T15:46:15.739469+00:00: The following VIBs are
      installed:
        vmware-fdm    6.0.0-4541948
      ----------
      (Original Vendor):Dell
      2016-10-28T14:02:12.849124+00:00: The following VIBs are
      installed:
        vmware-fdm    6.0.0-3634794
      ----------
      For more information, see http://kb.vmware.com/kb/2142190.

   VIBs: ata-pata-amd 0.3.10-3vmw.600.0.0.2494585, ata-pata-atiixp 0.4.6-4vmw.600.0.0.2494585, ata-pata-cmd64x 0.2.5-3vmw.600.0.0.2494585, ata-pata-hpt3x2n 0.3.4-3vmw.600.0.0.2494585, ata-pata-pdc2027x 1.0-3vmw.600.0.0.2494585, ata-pata-serverworks 0.4.3-3vmw.600.0.0.2494585, ata-pata-sil680 0.4.8-3vmw.600.0.0.2494585, ata-pata-via 0.3.3-2vmw.600.0.0.2494585, block-cciss 3.6.14-10vmw.600.0.0.2494585, cpu-microcode 6.0.0-0.0.2494585, dell-configuration-vib 6.0-2A00, ehci-ehci-hcd 1.0-3vmw.600.2.52.4600944, elxnet 10.6.126.0-1OEM.600.0.0.2159203, emulex-esx-elxnetcli 10.2.309.6v-0.0.2494585, esx-base 6.0.0-3.58.5224934, esx-dvfilter-generic-fastpath 6.0.0-0.0.2494585, esx-tboot 6.0.0-3.57.5050593, esx-ui 1.14.0-4940836, esx-xserver 6.0.0-0.0.2494585, ima-be2iscsi 10.6.150.3-1OEM.600.0.0.2159203, ima-qla4xxx 500.2.01.31-1vmw.0.3.100400, ipmi-ipmi-devintf 39.1-4vmw.600.0.0.2494585, ipmi-ipmi-msghandler 39.1-4vmw.600.0.0.2494585, ipmi-ipmi-si-drv 39.1-4vmw.600.0.0.2494585, lpfc 10.6.126.0-1OEM.600.0.0.2159203, lsi-mr3 6.903.82.00-1OEM.600.0.0.2768847, lsi-msgpt3 10.00.03.00-1OEM.600.0.0.2768847, lsu-hp-hpsa-plugin 2.0.0-4vmw.600.3.57.5050593, lsu-lsi-lsi-mr3-plugin 1.0.0-3vmw.600.3.57.5050593, lsu-lsi-lsi-msgpt3-plugin 1.0.0-1vmw.600.0.0.2494585, lsu-lsi-megaraid-sas-plugin 1.0.0-3vmw.600.3.57.5050593, lsu-lsi-mpt2sas-plugin 1.0.0-4vmw.600.1.17.3029758, lsu-lsi-mptsas-plugin 1.0.0-1vmw.600.0.0.2494585, misc-cnic-register 1.712.10.v55.2-1OEM.550.0.0.1331820, misc-drivers 6.0.0-3.57.5050593, mtip32xx-native 3.8.5-1vmw.600.0.0.2494585, net-bnx2 2.2.5g.v55.2-1OEM.550.0.0.1331820, net-bnx2x 2.712.10.v55.4-1OEM.550.0.0.1331820, net-cnic 2.712.10.v55.4-1OEM.550.0.0.1331820, net-e1000 8.0.3.1-5vmw.600.0.0.2494585, net-e1000e 3.2.2.1-2vmw.600.3.57.5050593, net-enic 2.1.2.38-2vmw.600.0.0.2494585, net-forcedeth 0.61-2vmw.600.0.0.2494585, net-i40e 1.3.38-1OEM.550.0.0.1331820, net-igb 5.3.1-1OEM.550.0.0.1331820, net-ixgbe 4.1.1.1-1OEM.550.0.0.1331820, net-mlx4-core 1.9.7.0-1vmw.600.0.0.2494585, net-mlx4-en 1.9.7.0-1vmw.600.0.0.2494585, net-nx-nic 5.0.621-5vmw.600.0.0.2494585, net-qlcnic 6.1.191-1OEM.600.0.0.2494585, net-qlge 3.0.0.59-1OEM.550.0.0.1331820, net-tg3 3.137l.v60.1-1OEM.600.0.0.2494585, net-vmxnet3 1.1.3.0-3vmw.600.2.43.4192238, nmlx4-core 3.1.0.0-1OEM.600.0.0.2348722, nmlx4-en 3.1.0.0-1OEM.600.0.0.2348722, nmlx4-rdma 3.0.0.0-1vmw.600.0.0.2494585, nvme 1.2.0.27-4vmw.550.0.0.1331820, ohci-usb-ohci 1.0-3vmw.600.0.0.2494585, qlnativefc 2.1.50.0-1vmw.600.3.57.5050593, rste 2.0.2.0088-4vmw.600.0.0.2494585, sata-ahci 3.0-26vmw.600.3.57.5050593, sata-ata-piix 2.12-10vmw.600.0.0.2494585, sata-sata-nv 3.5-4vmw.600.0.0.2494585, sata-sata-promise 2.12-3vmw.600.0.0.2494585, sata-sata-sil 2.3-4vmw.600.0.0.2494585, sata-sata-sil24 1.1-1vmw.600.0.0.2494585, sata-sata-svw 2.3-3vmw.600.0.0.2494585, scsi-aacraid 1.1.5.1-9vmw.600.0.0.2494585, scsi-adp94xx 1.0.8.12-6vmw.600.0.0.2494585, scsi-aic79xx 3.1-5vmw.600.0.0.2494585, scsi-be2iscsi 10.6.150.3-1OEM.600.0.0.2159203, scsi-bnx2fc 1.712.10.v55.2-1OEM.550.0.0.1331820, scsi-bnx2i 2.712.10.v55.4-1OEM.550.0.0.1331820, scsi-fnic 1.5.0.45-3vmw.600.0.0.2494585, scsi-hpsa 6.0.0.44-4vmw.600.0.0.2494585, scsi-ips 7.12.05-4vmw.600.0.0.2494585, scsi-megaraid-mbox 2.20.5.1-6vmw.600.0.0.2494585, scsi-megaraid-sas 06.805.56.00-1OEM.600.0.0.2494585, scsi-megaraid2 2.00.4-9vmw.600.0.0.2494585, scsi-mpt2sas 19.00.00.00-1vmw.600.0.0.2494585, scsi-mptsas 4.23.01.00-9vmw.600.0.0.2494585, scsi-mptspi 4.23.01.00-9vmw.600.0.0.2494585, scsi-qla4xxx 644.55.36.0-1OEM.550.0.0.1331820, tools-light 6.0.0-3.57.5050593, uhci-usb-uhci 1.0-3vmw.600.0.0.2494585, vmware-fdm 6.0.0-7037393, vsan 6.0.0-3.58.5224737, vsanhealth 6.0.0-3000000.3.0.3.58.5224738, xhci-xhci 1.0-3vmw.600.2.52.4600944

Open in new window




esxcli software vib list
Name                           Version                              Vendor  Acceptance Level  Install Date
-----------------------------  -----------------------------------  ------  ----------------  ------------
lsi-mr3                        6.903.82.00-1OEM.600.0.0.2768847     Avago   VMwareCertified   2016-10-28  
lsi-msgpt3                     10.00.03.00-1OEM.600.0.0.2768847     Avago   VMwareCertified   2016-10-28  
net-tg3                        3.137l.v60.1-1OEM.600.0.0.2494585    BRCM    VMwareCertified   2016-10-28  
dell-configuration-vib         6.0-2A00                             Dell    PartnerSupported  2016-10-28  
elxnet                         10.6.126.0-1OEM.600.0.0.2159203      EMU     VMwareCertified   2016-10-28  
ima-be2iscsi                   10.6.150.3-1OEM.600.0.0.2159203      EMU     VMwareCertified   2016-10-28  
lpfc                           10.6.126.0-1OEM.600.0.0.2159203      EMU     VMwareCertified   2016-10-28  
scsi-be2iscsi                  10.6.150.3-1OEM.600.0.0.2159203      EMU     VMwareCertified   2016-10-28  
net-i40e                       1.3.38-1OEM.550.0.0.1331820          Intel   VMwareCertified   2016-10-28  
net-igb                        5.3.1-1OEM.550.0.0.1331820           Intel   VMwareCertified   2016-10-28  
net-ixgbe                      4.1.1.1-1OEM.550.0.0.1331820         Intel   VMwareCertified   2016-10-28  
nmlx4-core                     3.1.0.0-1OEM.600.0.0.2348722         MEL     VMwareCertified   2016-10-28  
nmlx4-en                       3.1.0.0-1OEM.600.0.0.2348722         MEL     VMwareCertified   2016-10-28  
ima-qla4xxx                    500.2.01.31-1vmw.0.3.100400          QLogic  VMwareCertified   2016-10-28  
misc-cnic-register             1.712.10.v55.2-1OEM.550.0.0.1331820  QLogic  VMwareCertified   2016-10-28  
net-bnx2                       2.2.5g.v55.2-1OEM.550.0.0.1331820    QLogic  VMwareCertified   2016-10-28  
net-bnx2x                      2.712.10.v55.4-1OEM.550.0.0.1331820  QLogic  VMwareCertified   2016-10-28  
net-cnic                       2.712.10.v55.4-1OEM.550.0.0.1331820  QLogic  VMwareCertified   2016-10-28  
net-qlcnic                     6.1.191-1OEM.600.0.0.2494585         QLogic  VMwareCertified   2016-10-28  
net-qlge                       3.0.0.59-1OEM.550.0.0.1331820        QLogic  VMwareCertified   2016-10-28  
scsi-bnx2fc                    1.712.10.v55.2-1OEM.550.0.0.1331820  QLogic  VMwareCertified   2016-10-28  
scsi-bnx2i                     2.712.10.v55.4-1OEM.550.0.0.1331820  QLogic  VMwareCertified   2016-10-28  
scsi-qla4xxx                   644.55.36.0-1OEM.550.0.0.1331820     QLogic  VMwareCertified   2016-10-28  
mtip32xx-native                3.8.5-1vmw.600.0.0.2494585           VMWARE  VMwareCertified   2016-10-28  
scsi-megaraid-sas              06.805.56.00-1OEM.600.0.0.2494585    VMW     VMwareCertified   2016-10-28  
ata-pata-amd                   0.3.10-3vmw.600.0.0.2494585          VMware  VMwareCertified   2016-10-28  
ata-pata-atiixp                0.4.6-4vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
ata-pata-cmd64x                0.2.5-3vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
ata-pata-hpt3x2n               0.3.4-3vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
ata-pata-pdc2027x              1.0-3vmw.600.0.0.2494585             VMware  VMwareCertified   2016-10-28  
ata-pata-serverworks           0.4.3-3vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
ata-pata-sil680                0.4.8-3vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
ata-pata-via                   0.3.3-2vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
block-cciss                    3.6.14-10vmw.600.0.0.2494585         VMware  VMwareCertified   2016-10-28  
cpu-microcode                  6.0.0-0.0.2494585                    VMware  VMwareCertified   2016-10-28  
ehci-ehci-hcd                  1.0-3vmw.600.2.52.4600944            VMware  VMwareCertified   2017-04-04  
emulex-esx-elxnetcli           10.2.309.6v-0.0.2494585              VMware  VMwareCertified   2016-10-28  
esx-base                       6.0.0-3.58.5224934                   VMware  VMwareCertified   2017-04-04  
esx-dvfilter-generic-fastpath  6.0.0-0.0.2494585                    VMware  VMwareCertified   2016-10-28  
esx-tboot                      6.0.0-3.57.5050593                   VMware  VMwareCertified   2017-04-04  
esx-ui                         1.14.0-4940836                       VMware  VMwareCertified   2017-04-04  
esx-xserver                    6.0.0-0.0.2494585                    VMware  VMwareCertified   2016-10-28  
ipmi-ipmi-devintf              39.1-4vmw.600.0.0.2494585            VMware  VMwareCertified   2016-10-28  
ipmi-ipmi-msghandler           39.1-4vmw.600.0.0.2494585            VMware  VMwareCertified   2016-10-28  
ipmi-ipmi-si-drv               39.1-4vmw.600.0.0.2494585            VMware  VMwareCertified   2016-10-28  
lsu-hp-hpsa-plugin             2.0.0-4vmw.600.3.57.5050593          VMware  VMwareCertified   2017-04-04  
lsu-lsi-lsi-mr3-plugin         1.0.0-3vmw.600.3.57.5050593          VMware  VMwareCertified   2017-04-04  
lsu-lsi-lsi-msgpt3-plugin      1.0.0-1vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
lsu-lsi-megaraid-sas-plugin    1.0.0-3vmw.600.3.57.5050593          VMware  VMwareCertified   2017-04-04  
lsu-lsi-mpt2sas-plugin         1.0.0-4vmw.600.1.17.3029758          VMware  VMwareCertified   2016-10-28  
lsu-lsi-mptsas-plugin          1.0.0-1vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
misc-drivers                   6.0.0-3.57.5050593                   VMware  VMwareCertified   2017-04-04  
net-e1000                      8.0.3.1-5vmw.600.0.0.2494585         VMware  VMwareCertified   2016-10-28  
net-e1000e                     3.2.2.1-2vmw.600.3.57.5050593        VMware  VMwareCertified   2017-04-04  
net-enic                       2.1.2.38-2vmw.600.0.0.2494585        VMware  VMwareCertified   2016-10-28  
net-forcedeth                  0.61-2vmw.600.0.0.2494585            VMware  VMwareCertified   2016-10-28  
net-mlx4-core                  1.9.7.0-1vmw.600.0.0.2494585         VMware  VMwareCertified   2016-10-28  
net-mlx4-en                    1.9.7.0-1vmw.600.0.0.2494585         VMware  VMwareCertified   2016-10-28  
net-nx-nic                     5.0.621-5vmw.600.0.0.2494585         VMware  VMwareCertified   2016-10-28  
net-vmxnet3                    1.1.3.0-3vmw.600.2.43.4192238        VMware  VMwareCertified   2017-04-04  
nmlx4-rdma                     3.0.0.0-1vmw.600.0.0.2494585         VMware  VMwareCertified   2016-10-28  
nvme                           1.2.0.27-4vmw.550.0.0.1331820        VMware  VMwareCertified   2016-10-28  
ohci-usb-ohci                  1.0-3vmw.600.0.0.2494585             VMware  VMwareCertified   2016-10-28  
qlnativefc                     2.1.50.0-1vmw.600.3.57.5050593       VMware  VMwareCertified   2017-04-04  
rste                           2.0.2.0088-4vmw.600.0.0.2494585      VMware  VMwareCertified   2016-10-28  
sata-ahci                      3.0-26vmw.600.3.57.5050593           VMware  VMwareCertified   2017-04-04  
sata-ata-piix                  2.12-10vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
sata-sata-nv                   3.5-4vmw.600.0.0.2494585             VMware  VMwareCertified   2016-10-28  
sata-sata-promise              2.12-3vmw.600.0.0.2494585            VMware  VMwareCertified   2016-10-28  
sata-sata-sil24                1.1-1vmw.600.0.0.2494585             VMware  VMwareCertified   2016-10-28  
sata-sata-sil                  2.3-4vmw.600.0.0.2494585             VMware  VMwareCertified   2016-10-28  
sata-sata-svw                  2.3-3vmw.600.0.0.2494585             VMware  VMwareCertified   2016-10-28  
scsi-aacraid                   1.1.5.1-9vmw.600.0.0.2494585         VMware  VMwareCertified   2016-10-28  
scsi-adp94xx                   1.0.8.12-6vmw.600.0.0.2494585        VMware  VMwareCertified   2016-10-28  
scsi-aic79xx                   3.1-5vmw.600.0.0.2494585             VMware  VMwareCertified   2016-10-28  
scsi-fnic                      1.5.0.45-3vmw.600.0.0.2494585        VMware  VMwareCertified   2016-10-28  
scsi-hpsa                      6.0.0.44-4vmw.600.0.0.2494585        VMware  VMwareCertified   2016-10-28  
scsi-ips                       7.12.05-4vmw.600.0.0.2494585         VMware  VMwareCertified   2016-10-28  
scsi-megaraid-mbox             2.20.5.1-6vmw.600.0.0.2494585        VMware  VMwareCertified   2016-10-28  
scsi-megaraid2                 2.00.4-9vmw.600.0.0.2494585          VMware  VMwareCertified   2016-10-28  
scsi-mpt2sas                   19.00.00.00-1vmw.600.0.0.2494585     VMware  VMwareCertified   2016-10-28  
scsi-mptsas                    4.23.01.00-9vmw.600.0.0.2494585      VMware  VMwareCertified   2016-10-28  
scsi-mptspi                    4.23.01.00-9vmw.600.0.0.2494585      VMware  VMwareCertified   2016-10-28  
uhci-usb-uhci                  1.0-3vmw.600.0.0.2494585             VMware  VMwareCertified   2016-10-28  
vmware-fdm                     6.0.0-7037393                        VMware  VMwareCertified   2018-01-09  
vsan                           6.0.0-3.58.5224737                   VMware  VMwareCertified   2017-04-04  
vsanhealth                     6.0.0-3000000.3.0.3.58.5224738       VMware  VMwareCertified   2017-04-04  
xhci-xhci                      1.0-3vmw.600.2.52.4600944            VMware  VMwareCertified   2017-04-04  
tools-light                    6.0.0-3.57.5050593                   VMware  VMwareCertified   2017-04-04  

Open in new window


Thank you very much
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I have no space issue. I tried to install upgrade on affected host, and I encountered several problems,

Have you checked and logged into host at console or SSH and checked ? The upgrade states file too large ?

Okay, yes this is normal because the VCSA wants to push a new HA Agent onto the hosts ?

and if using OEM ESXi, this can be the issue because the ESXi OS partition is out of space.

Are you hosts flash , e.g. SD or USB flash drives?

have you checked the space on the ESXi OS partitions, using df or vdf ?
0
Armitage318Author Commented:
Hi Andrew,

this is command's output:

 df -h
Filesystem    Size   Used Available Use% Mounted on
VMFS-5      499.8G 368.2G    131.6G  74% /vmfs/volumes/cubestorage2_lun6
VMFS-5     1023.8G 935.8G     87.9G  91% /vmfs/volumes/cubestorage3_lun3
VMFS-5     1023.8G 962.1G     61.6G  94% /vmfs/volumes/cubestorage3_lun4
VMFS-5      499.8G 412.2G     87.5G  82% /vmfs/volumes/cubestorage2_lun8
VMFS-5      499.8G 397.2G    102.5G  79% /vmfs/volumes/cubestorage1_lun2
VMFS-5      499.8G 352.5G    147.3G  71% /vmfs/volumes/cubestorage2_lun3
VMFS-5      499.8G 121.9G    377.8G  24% /vmfs/volumes/cubestorage1_lun3
VMFS-5      499.8G 406.7G     93.0G  81% /vmfs/volumes/cubestorage1_lun9
VMFS-5      499.8G 401.3G     98.5G  80% /vmfs/volumes/cubestorage2_lun9
VMFS-5      499.8G 429.8G     70.0G  86% /vmfs/volumes/cubestorage1_lun4
VMFS-5      499.8G 409.4G     90.4G  82% /vmfs/volumes/cubestorage1_lun6
VMFS-5      499.8G 381.2G    118.5G  76% /vmfs/volumes/cubestorage2_lun7
VMFS-5     1023.8G 924.4G     99.4G  90% /vmfs/volumes/cubestorage3_lun0
VMFS-5      499.8G 333.2G    166.6G  67% /vmfs/volumes/cubestorage3_lun6
VMFS-5      499.8G 299.0G    200.8G  60% /vmfs/volumes/cubestorage2_lun0
VMFS-5      499.8G 411.4G     88.4G  82% /vmfs/volumes/cubestorage1_lun8
VMFS-5      465.5G 222.3G    243.2G  48% /vmfs/volumes/Localdisk (host1)
VMFS-5      483.8G 414.5G     69.2G  86% /vmfs/volumes/cubestorage1_lun10
VMFS-5     1023.8G 890.0G    133.7G  87% /vmfs/volumes/cubestorage3_lun5
VMFS-5      326.5G 180.9G    145.6G  55% /vmfs/volumes/cubestorage2_lun10
VMFS-5      599.8G 518.5G     81.3G  86% /vmfs/volumes/cubestorage1_lun5
VMFS-5      395.8G 366.7G     29.0G  93% /vmfs/volumes/cubestorage3_lun7
VMFS-5      499.8G 446.3G     53.5G  89% /vmfs/volumes/cubestorage2_lun2
VMFS-5      465.2G 376.2G     89.1G  81% /vmfs/volumes/cubestorage2_lun5
VMFS-5     1023.8G 959.4G     64.3G  94% /vmfs/volumes/cubestorage3_lun1
VMFS-5      499.8G 449.1G     50.7G  90% /vmfs/volumes/cubestorage2_lun1
VMFS-5      513.5G 477.7G     35.8G  93% /vmfs/volumes/cubestorage2_lun4
VMFS-5      499.8G 418.2G     81.5G  84% /vmfs/volumes/cubestorage1_lun1
VMFS-5      499.8G 454.3G     45.5G  91% /vmfs/volumes/cubestorage1_lun0
VMFS-5      499.8G 322.2G    177.5G  64% /vmfs/volumes/cubestorage1_lun7
VMFS-5      499.8G 334.1G    165.6G  67% /vmfs/volumes/cubestorage3_lun2
vfat        249.7M 169.4M     80.3M  68% /vmfs/volumes/64600de9-8fa9e120-e5e9-61a105546360
vfat        249.7M 184.9M     64.8M  74% /vmfs/volumes/a8d8beea-0bd31a36-daf7-2238ad159346
vfat        285.8M 203.6M     82.2M  71% /vmfs/volumes/5829e530-34d64d98-a0bd-d4ae52bc6b95

Open in new window



vdf -h
Tardisk                  Space      Used
sb.v00                    140M      140M
s.v00                     317M      317M
lsi_mr3.v00               280K      277K
lsi_msgp.v00              460K      456K
net_tg3.v00               300K      298K
elxnet.v00                508K      505K
ima_be2i.v00                2M        2M
lpfc.v00                    2M        2M
scsi_be2.v00              800K      796K
net_i40e.v00              404K      402K
net_igb.v00               324K      320K
net_ixgb.v00              524K      523K
nmlx4_co.v00              584K      581K
nmlx4_en.v00              612K      609K
ima_qla4.v00                1M        1M
misc_cni.v00               24K       21K
net_bnx2.v00              304K      303K
net_bnx2.v01                1M        1M
net_cnic.v00              140K      138K
net_qlcn.v00                1M        1M
net_qlge.v00              340K      339K
scsi_bnx.v00              288K      287K
scsi_bnx.v01              244K      241K
scsi_qla.v00              532K      530K
mtip32xx.v00              244K      241K
scsi_meg.v00              176K      175K
ata_pata.v00               40K       37K
ata_pata.v01               28K       26K
ata_pata.v02               32K       28K
ata_pata.v03               32K       29K
ata_pata.v04               36K       33K
ata_pata.v05               32K       30K
ata_pata.v06               28K       26K
ata_pata.v07               32K       30K
block_cc.v00               80K       76K
ehci_ehc.v00               92K       89K
emulex_e.v00               24K       22K
weaselin.t00                5M        5M
esx_dvfi.v00              416K      414K
esx_ui.v00                 11M       11M
ipmi_ipm.v00               36K       34K
ipmi_ipm.v01               80K       77K
ipmi_ipm.v02              100K       96K
lsu_hp_h.v00               64K       61K
lsu_lsi_.v00              240K      237K
lsu_lsi_.v01              420K      417K
lsu_lsi_.v02              240K      237K
lsu_lsi_.v03              508K      504K
lsu_lsi_.v04              304K      302K
misc_dri.v00                5M        5M
net_e100.v00              308K      305K
net_e100.v01              344K      342K
net_enic.v00              140K      139K
net_forc.v00              120K      117K
net_mlx4.v00              340K      337K
net_mlx4.v01              228K      227K
net_nx_n.v00                1M        1M
net_vmxn.v00              100K       99K
nmlx4_rd.v00              172K      171K
nvme.v00                  232K      231K
ohci_usb.v00               60K       57K
qlnative.v00                2M        2M
rste.v00                  796K      794K
sata_ahc.v00               80K       79K
sata_ata.v00               52K       51K
sata_sat.v00               60K       59K
sata_sat.v01               40K       38K
sata_sat.v02               40K       39K
sata_sat.v03               32K       30K
sata_sat.v04               28K       27K
scsi_aac.v00              172K      169K
scsi_adp.v00              428K      425K
scsi_aic.v00              284K      282K
scsi_fni.v00              228K      226K
scsi_hps.v00              172K      169K
scsi_ips.v00              100K       98K
scsi_meg.v01               92K       91K
scsi_meg.v02               88K       87K
scsi_mpt.v00              448K      445K
scsi_mpt.v01              492K      489K
scsi_mpt.v02              420K      416K
uhci_usb.v00               60K       57K
vmware_f.v00               47M       47M
vsan.v00                   23M       23M
vsanheal.v00                2M        2M
vsanmgmt.v00                6M        6M
xhci_xhc.v00              228K      226K
dell_con.v00                4K        3K
xorg.v00                    3M        3M
imgdb.tgz                  12K       10K
state.tgz                  36K       36K
-----
Ramdisk                   Size      Used Available Use% Mounted on
root                       32M      244K       31M   0% --
etc                        28M      328K       27M   1% --
opt                        32M        0B       32M   0% --
var                        48M        1M       46M   2% --
tmp                       256M       20K      255M   0% --
iofilters                  32M        0B       32M   0% --
hostdstats                803M        6M      796M   0% --
stagebootbank             250M       13M      236M   5% --

Open in new window

0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
what does the FDM log state ?

can you see the vib being pushed to the host from the server, you can see it with WinSCP or SSH

have you tried removing the existing HA vib if present ? (vmware_f.v00)
0
Armitage318Author Commented:
Hi, my fdm.log file is empty.
I tried "Remediate" action from vSphere client, and I saw in realtime log through ssh.
I attach to complete esxupdate.log file.

How can I remove HA vib, as you suggested?
esxupdate.txt
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Please don't update your host, this will not cause the issue to go away.....

if you cannot push the HA Agent out to your host, something is WRONG with the host file system, or space.

when you push the HA Agent (e.g. reconfigure HA from vSphere) do you see it appear on host ?
0
Armitage318Author Commented:
Hi Andrew,
the affected host is in maintenance mode. It seems I have no option to "Reconfigure HA" through vSphere.
Do you think I can leave maintenance mode or is it risky?
Thank you
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Exit Maintenance Mode, and HA should reconfigure itself again, and elect itself to Master or Slave of FAIL.

Currently all that is not working is HA, if all is well with Host and Networking, it should still function, and if you have DRS enabled, VMs will automatically move VMs to it.

OR remove it from the Cluster!
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Armitage318Author Commented:
Hi, I removed host from maintenance mode, and I noticed this:

2018-01-10T08:09:16Z esxupdate: 78268: HostImage: ERROR: Failed to send vob install.stage.error: [Errno 27] File too large^@
2018-01-10T08:09:16Z esxupdate: 78268: esxupdate: ERROR: An esxupdate error exception was caught:^@
2018-01-10T08:09:16Z esxupdate: 78268: esxupdate: ERROR: Traceback (most recent call last):^@
2018-01-10T08:09:16Z esxupdate: 78268: esxupdate: ERROR:   File "/usr/sbin/esxupdate", line 238, in main^@
2018-01-10T08:09:16Z esxupdate: 78268: esxupdate: ERROR:     cmd.Run()^@
2018-01-10T08:09:16Z esxupdate: 78268: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-pack
ages/vmware/esx5update/Cmdline.py", line 148, in Run^@
2018-01-10T08:09:16Z esxupdate: 78268: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-pack
ages/vmware/esximage/Transaction.py", line 250, in InstallVibsFromSources^@
2018-01-10T08:09:16Z esxupdate: 78268: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 356, in _installVibs^@
2018-01-10T08:09:16Z esxupdate: 78268: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 399, in _validateAndInstallProfile^@
2018-01-10T08:09:16Z esxupdate: 78268: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/HostImage.py", line 703, in Stage^@
2018-01-10T08:09:16Z esxupdate: 78268: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/HostImage.py", line 534, in _get_tardisk_payload^@

Open in new window


Strange, it seems I have no space issue..
0
Armitage318Author Commented:
According to "file too large" error, I reconfigured ScratchConfig path (and I am sure that there is enough space on it).
After reboot, I saw this in logs:

2018-01-10T09:30:42Z esxupdate: root: INFO: Command = profile.setacceptance^@
2018-01-10T09:30:42Z esxupdate: root: INFO: Options = {}^@
2018-01-10T09:30:42Z esxupdate: BootBankInstaller.pyc: DEBUG: Creating an empty ImageProfile for bootbank /bootbank^@
2018-01-10T09:30:42Z esxupdate: BootBankInstaller.pyc: INFO: Unrecognized value "title=Loading VMware ESXi" in boot.cfg^@
2018-01-10T09:30:42Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/bootOption', '-rp']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.^@
2018-01-10T09:30:42Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/bootOption', '-ro']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.^@
2018-01-10T09:30:42Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-U', 'host-acceptance-level', '-G']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.^@
2018-01-10T09:30:42Z esxupdate: root: ERROR: Traceback (most recent call last):^@
2018-01-10T09:30:42Z esxupdate: root: ERROR:   File "/usr/lib/vmware/esxcli/int/esxcli-softwareinternal", line 104, in <module>^@
2018-01-10T09:30:42Z esxupdate: root: ERROR:     main()^@
2018-01-10T09:30:42Z esxupdate: root: ERROR:   File "/usr/lib/vmware/esxcli/int/esxcli-softwareinternal", line 96, in main^@
2018-01-10T09:30:42Z esxupdate: root: ERROR:     ret = CMDTABLE[command](options)^@
2018-01-10T09:30:42Z esxupdate: root: ERROR:   File "/usr/lib/vmware/esxcli/int/esxcli-softwareinternal", line 66, in AcceptanceSetCmd^@
2018-01-10T09:30:42Z esxupdate: root: ERROR:     return profile.acceptancelevel^@
2018-01-10T09:30:42Z esxupdate: root: ERROR: AttributeError: 'NoneType' object has no attribute 'acceptancelevel'^@

Open in new window

0
Armitage318Author Commented:
This is main difference between this host and another ore which is running fine:

AFFECTED HOST:
 esxcli software vib list
Name         Version             Vendor  Acceptance Level  Install Date
-----------  ------------------  ------  ----------------  ------------
tools-light  6.0.0-3.57.5050593  VMware  VMwareCertified   2017-04-04  

Open in new window



OTHER (OK) HOST:
esxcli software vib list
Name                           Version                              Vendor  Acceptance Level  Install Date
-----------------------------  -----------------------------------  ------  ----------------  ------------
lsi-mr3                        6.903.82.00-1OEM.600.0.0.2768847     Avago   VMwareCertified   2016-10-28  
lsi-msgpt3                     10.00.03.00-1OEM.600.0.0.2768847     Avago   VMwareCertified   2016-10-28  
net-tg3                        3.137l.v60.1-1OEM.600.0.0.2494585    BRCM    VMwareCertified   2016-10-28  
dell-configuration-vib         6.0-2A00                             Dell    PartnerSupported  2016-10-28  
elxnet                         10.6.126.0-1OEM.600.0.0.2159203      EMU     VMwareCertified   2016-10-28  
ima-be2iscsi                   10.6.150.3-1OEM.600.0.0.2159203      EMU     VMwareCertified   2016-10-28  
lpfc                           10.6.126.0-1OEM.600.0.0.2159203      EMU     VMwareCertified   2016-10-28  
scsi-be2iscsi                  10.6.150.3-1OEM.600.0.0.2159203      EMU     VMwareCertified   2016-10-28  
net-i40e                       1.3.38-1OEM.550.0.0.1331820          Intel   VMwareCertified   2016-10-28  
net-igb                        5.3.1-1OEM.550.0.0.1331820           Intel   VMwareCertified   2016-10-28  
net-ixgbe                      4.1.1.1-1OEM.550.0.0.1331820         Intel   VMwareCertified   2016-10-28  
nmlx4-core                     3.1.0.0-1OEM.600.0.0.2348722         MEL     VMwareCertified   2016-10-28  
nmlx4-en                       3.1.0.0-1OEM.600.0.0.2348722         MEL     VMwareCertified   2016-10-28  
ima-qla4xxx                    500.2.01.31-1vmw.0.3.100400          QLogic  VMwareCertified   2016-10-28  
misc-cnic-register             1.712.10.v55.2-1OEM.550.0.0.1331820  QLogic  VMwareCertified   2016-10-28  
net-bnx2                       2.2.5g.v55.2-1OEM.550.0.0.1331820    QLogic  VMwareCertified   2016-10-28  
net-bnx2x                      2.712.10.v55.4-1OEM.550.0.0.1331820  QLogic  VMwareCertified   2016-10-28  
net-cnic                       2.712.10.v55.4-1OEM.550.0.0.1331820  QLogic  VMwareCertified   2016-10-28  
net-qlcnic                     6.1.191-1OEM.600.0.0.2494585         QLogic  VMwareCertified   2016-10-28  
net-qlge                       3.0.0.59-1OEM.550.0.0.1331820        QLogic  VMwareCertified   2016-10-28  
scsi-bnx2fc                    1.712.10.v55.2-1OEM.550.0.0.1331820  QLogic  VMwareCertified   2016-10-28  
scsi-bnx2i                     2.712.10.v55.4-1OEM.550.0.0.1331820  QLogic  VMwareCertified   2016-10-28  
scsi-qla4xxx                   644.55.36.0-1OEM.550.0.0.1331820     QLogic  VMwareCertified   2016-10-28  
mtip32xx-native                3.8.5-1vmw.600.0.0.2494585           VMWARE  VMwareCertified   2016-10-28  
scsi-megaraid-sas              06.805.56.00-1OEM.600.0.0.2494585    VMW     VMwareCertified   2016-10-28  
ata-pata-amd                   0.3.10-3vmw.600.0.0.2494585          VMware  VMwareCertified   2016-10-28  
ata-pata-atiixp                0.4.6-4vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
ata-pata-cmd64x                0.2.5-3vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
ata-pata-hpt3x2n               0.3.4-3vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
ata-pata-pdc2027x              1.0-3vmw.600.0.0.2494585             VMware  VMwareCertified   2016-10-28  
ata-pata-serverworks           0.4.3-3vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
ata-pata-sil680                0.4.8-3vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
ata-pata-via                   0.3.3-2vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
block-cciss                    3.6.14-10vmw.600.0.0.2494585         VMware  VMwareCertified   2016-10-28  
cpu-microcode                  6.0.0-0.0.2494585                    VMware  VMwareCertified   2016-10-28  
ehci-ehci-hcd                  1.0-3vmw.600.2.52.4600944            VMware  VMwareCertified   2017-04-04  
emulex-esx-elxnetcli           10.2.309.6v-0.0.2494585              VMware  VMwareCertified   2016-10-28  
esx-base                       6.0.0-3.58.5224934                   VMware  VMwareCertified   2017-04-04  
esx-dvfilter-generic-fastpath  6.0.0-0.0.2494585                    VMware  VMwareCertified   2016-10-28  
esx-tboot                      6.0.0-3.57.5050593                   VMware  VMwareCertified   2017-04-04  
esx-ui                         1.14.0-4940836                       VMware  VMwareCertified   2017-04-04  
esx-xserver                    6.0.0-0.0.2494585                    VMware  VMwareCertified   2016-10-28  
ipmi-ipmi-devintf              39.1-4vmw.600.0.0.2494585            VMware  VMwareCertified   2016-10-28  
ipmi-ipmi-msghandler           39.1-4vmw.600.0.0.2494585            VMware  VMwareCertified   2016-10-28  
ipmi-ipmi-si-drv               39.1-4vmw.600.0.0.2494585            VMware  VMwareCertified   2016-10-28  
lsu-hp-hpsa-plugin             2.0.0-4vmw.600.3.57.5050593          VMware  VMwareCertified   2017-04-04  
lsu-lsi-lsi-mr3-plugin         1.0.0-3vmw.600.3.57.5050593          VMware  VMwareCertified   2017-04-04  
lsu-lsi-lsi-msgpt3-plugin      1.0.0-1vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
lsu-lsi-megaraid-sas-plugin    1.0.0-3vmw.600.3.57.5050593          VMware  VMwareCertified   2017-04-04  
lsu-lsi-mpt2sas-plugin         1.0.0-4vmw.600.1.17.3029758          VMware  VMwareCertified   2016-10-28  
lsu-lsi-mptsas-plugin          1.0.0-1vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
misc-drivers                   6.0.0-3.57.5050593                   VMware  VMwareCertified   2017-04-04  
net-e1000                      8.0.3.1-5vmw.600.0.0.2494585         VMware  VMwareCertified   2016-10-28  
net-e1000e                     3.2.2.1-2vmw.600.3.57.5050593        VMware  VMwareCertified   2017-04-04  
net-enic                       2.1.2.38-2vmw.600.0.0.2494585        VMware  VMwareCertified   2016-10-28  
net-forcedeth                  0.61-2vmw.600.0.0.2494585            VMware  VMwareCertified   2016-10-28  
net-mlx4-core                  1.9.7.0-1vmw.600.0.0.2494585         VMware  VMwareCertified   2016-10-28  
net-mlx4-en                    1.9.7.0-1vmw.600.0.0.2494585         VMware  VMwareCertified   2016-10-28  
net-nx-nic                     5.0.621-5vmw.600.0.0.2494585         VMware  VMwareCertified   2016-10-28  
net-vmxnet3                    1.1.3.0-3vmw.600.2.43.4192238        VMware  VMwareCertified   2017-04-04  
nmlx4-rdma                     3.0.0.0-1vmw.600.0.0.2494585         VMware  VMwareCertified   2016-10-28  
nvme                           1.2.0.27-4vmw.550.0.0.1331820        VMware  VMwareCertified   2016-10-28  
ohci-usb-ohci                  1.0-3vmw.600.0.0.2494585             VMware  VMwareCertified   2016-10-28  
qlnativefc                     2.1.50.0-1vmw.600.3.57.5050593       VMware  VMwareCertified   2017-04-04  
rste                           2.0.2.0088-4vmw.600.0.0.2494585      VMware  VMwareCertified   2016-10-28  
sata-ahci                      3.0-26vmw.600.3.57.5050593           VMware  VMwareCertified   2017-04-04  
sata-ata-piix                  2.12-10vmw.600.0.0.2494585           VMware  VMwareCertified   2016-10-28  
sata-sata-nv                   3.5-4vmw.600.0.0.2494585             VMware  VMwareCertified   2016-10-28  
sata-sata-promise              2.12-3vmw.600.0.0.2494585            VMware  VMwareCertified   2016-10-28  
sata-sata-sil24                1.1-1vmw.600.0.0.2494585             VMware  VMwareCertified   2016-10-28  
sata-sata-sil                  2.3-4vmw.600.0.0.2494585             VMware  VMwareCertified   2016-10-28  
sata-sata-svw                  2.3-3vmw.600.0.0.2494585             VMware  VMwareCertified   2016-10-28  
scsi-aacraid                   1.1.5.1-9vmw.600.0.0.2494585         VMware  VMwareCertified   2016-10-28  
scsi-adp94xx                   1.0.8.12-6vmw.600.0.0.2494585        VMware  VMwareCertified   2016-10-28  
scsi-aic79xx                   3.1-5vmw.600.0.0.2494585             VMware  VMwareCertified   2016-10-28  
scsi-fnic                      1.5.0.45-3vmw.600.0.0.2494585        VMware  VMwareCertified   2016-10-28  
scsi-hpsa                      6.0.0.44-4vmw.600.0.0.2494585        VMware  VMwareCertified   2016-10-28  
scsi-ips                       7.12.05-4vmw.600.0.0.2494585         VMware  VMwareCertified   2016-10-28  
scsi-megaraid-mbox             2.20.5.1-6vmw.600.0.0.2494585        VMware  VMwareCertified   2016-10-28  
scsi-megaraid2                 2.00.4-9vmw.600.0.0.2494585          VMware  VMwareCertified   2016-10-28  
scsi-mpt2sas                   19.00.00.00-1vmw.600.0.0.2494585     VMware  VMwareCertified   2016-10-28  
scsi-mptsas                    4.23.01.00-9vmw.600.0.0.2494585      VMware  VMwareCertified   2016-10-28  
scsi-mptspi                    4.23.01.00-9vmw.600.0.0.2494585      VMware  VMwareCertified   2016-10-28  
uhci-usb-uhci                  1.0-3vmw.600.0.0.2494585             VMware  VMwareCertified   2016-10-28  
vmware-fdm                     6.0.0-7037393                        VMware  VMwareCertified   2018-01-09  
vsan                           6.0.0-3.58.5224737                   VMware  VMwareCertified   2017-04-04  
vsanhealth                     6.0.0-3000000.3.0.3.58.5224738       VMware  VMwareCertified   2017-04-04  
xhci-xhci                      1.0-3vmw.600.2.52.4600944            VMware  VMwareCertified   2017-04-04  
tools-light                    6.0.0-3.57.5050593                   VMware  VMwareCertified   2017-04-04 

Open in new window

0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
do you have VMware Tools on other hosts ?
0
Armitage318Author Commented:
Hi Andrew,
yes.
I tried this way:

https://communities.vmware.com/thread/545379

since I noticed that, in fact, vmware_f.v00 file was missing on affected host.
Anyway, same error:

2018-01-10T10:28:03Z esxupdate: 38404: HostImage: ERROR: Failed to send vob install.stage.error: [Errno 27] File too large
2018-01-10T10:28:03Z esxupdate: 38404: esxupdate: ERROR: An esxupdate error exception was caught:
2018-01-10T10:28:03Z esxupdate: 38404: esxupdate: ERROR: Traceback (most recent call last):
2018-01-10T10:28:03Z esxupdate: 38404: esxupdate: ERROR:   File "/usr/sbin/esxupdate", line 238, in main
2018-01-10T10:28:03Z esxupdate: 38404: esxupdate: ERROR:     cmd.Run()
2018-01-10T10:28:03Z esxupdate: 38404: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esx5update/Cmdline.py", line 148, in Run
2018-01-10T10:28:03Z esxupdate: 38404: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 250, in InstallVibsFromSources
2018-01-10T10:28:03Z esxupdate: 38404: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 356, in _installVibs
2018-01-10T10:28:03Z esxupdate: 38404: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 399, in _validateAndInstallProfile
2018-01-10T10:28:03Z esxupdate: 38404: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/HostImage.py", line 703, in Stage
2018-01-10T10:28:03Z esxupdate: 38404: esxupdate: ERROR:   File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/HostImage.py", line 534, in _get_tardisk_payload

Open in new window

0
Armitage318Author Commented:
I noticed that "imgdb.tgz" file in /bootbank had wrong size:

[ANOTHER HOST]
ls -l 3b1636ea-d50f443d-2136-4a8b1aa042e5/imgdb.tgz
-rwx------    1 root     root         59390 Jan  9 10:22 3b1636ea-d50f443d-2136-4a8b1aa042e5/imgdb.tgz

ls -l a560c0ec-2b02d7bc-4c14-fccee4e9fcdb/imgdb.tgz
-rwx------    1 root     root         59193 Apr  4  2017 a560c0ec-2b02d7bc-4c14-fccee4e9fcdb/imgdb.tgz

Open in new window



[AFFECTED HOST]
ls -l 64600de9-8fa9e120-e5e9-61a105546360/imgdb.tgz
-rwx------    1 root     root         58183 Nov 14  2016 64600de9-8fa9e120-e5e9-61a105546360/imgdb.tgz
ls -l a8d8beea-0bd31a36-daf7-2238ad159346/imgdb.tgz
-rwx------    1 root     root           184 Apr  4  2017 a8d8beea-0bd31a36-daf7-2238ad159346/imgdb.tgz

Open in new window


I copied it from another host, and I reboot.. I will update you shortly
0
Armitage318Author Commented:
Ok, finally remediation went fine.
I think the "corrupted" imgdb.tgz was the root cause.
Anyway, I tried "reconfigure with HA" and I still get "operation timeout" error!
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
can you see the package deployed to host ?

or rebuild host, and /or remove unused OEM drivers ESXi is not using.
0
Armitage318Author Commented:
Basically I just upgraded all hosts to same ESXi version 6.0.0.7504637, and turn on again HA: this time, configuration went fine.
I think HA failed previously due to issues with imgdb.tgz and /var/db.
This URL was very very useful:

http://www.provirtualzone.com/esxi-6-0-fix-corrupted-host-imageprofile/

I would like to thank Andrew for his help
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
high availability

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.