vCenter Troubleshooting TIPS: ESXi 6.0 reports “error code 99” during Stage patches to entity VUM operation

Luciano PatrãoICT Senior Infraestructure  Engineer  
vExpert vSAN, NSX, Cloud Provider, Veeam Vanguard, Virtual Backups, and Storage design, and an active blogger.
This is similar to the error described in my previous Article, but with a different source problem and a different solution. When trying to scan and stage the ESXi 6.0 updates using VMware Update Manager, we can get "error code 90"
The configuration is an Hewlett-Packard DL360 G7 with the ESXi 6.0 build 3620759.
When trying to use the VMware update manager in this host, the scan runs without any problem, but when trying to stage the updates, we can get:

Stage patches to entity esxi01.localdomain. The host returns esxupdate error code:99.
An unhandled exception was encountered. Check the Update Manager log files and esxupdate log files for more details.
Looking at the esxpudate.log (in ESXi var/log/) I see lot of failed tries to upload VIB files to the repository, along with this:

Open in new window

So we need to check the Host Software profile(not the same as Host Profile) and check the issue.

Then I notice that somehow the ESXi lost its software profile, so it was not possible to stage any update, or even install a VIB manually. When troubleshooting the issue, I found out that there was no software profile in this host.

root@esxi01:~] esxcli software profile get
                      No host image profile defined
                      Please refer to the log file for more details.

Open in new window

Also checking the VIB list in this host I get this:

[root@esxi01:~] esxcli software vib list
                      Name         Version             Vendor  Acceptance Level  Install Date
                      -----------  ------------------  ------  ----------------  ------------
                      tools-light  6.0.0-2.34.3620759  VMware  VMwareCertified   2016-04-23

Open in new window

So we need to check the VIB folder and profiles folder in /var/db/esximg and I noticed the two folders were empty.

[root@esx01:~] ls -l /var/db/esximg/*
                      total 0
                      total 0

Open in new window

When checking the same information from a working ESXi host I get this:

[root@esxi02:~] ls -l /var/db/esximg/profiles/
                      total 24
                      -r--r--r--    1 root     root         23998 Apr 23 12:05 %28Updated%29%20HP-ESXi-6.0.0-iso-600.9.1.39-687127855
                      [root@esxi02:~] esxcli software profile get
                      (Updated) HP-ESXi-6.0.0-iso-600.9.1.39
                         Name: (Updated) HP-ESXi-6.0.0-iso-600.9.1.39
                         Vendor: esxi02.localdomain
                         Creation Time: 2016-04-23T12:05:31
                         Modification Time: 2016-04-23T12:05:31
                         Stateless Ready: True
                            (Original Vendor):Hewlett-Packard
                            2016-04-23T12:05:30.688280+00:00: The following VIBs are

Open in new window

And this:

[root@esxi02:~] esxcli software vib list
                      Name                           Version                         Vendor           Acceptance Level  Install Date
                      ----------------------  -------------------------------------  ---------------  ----------------  ------------
                      net-tg3                 3.137l.v60.1-1OEM.600.0.0.2494585      BRCM             VMwareCertified   2016-01-23
                      elxnet               EMU              VMwareCertified   2016-04-23
                      ima-be2iscsi         EMU              VMwareCertified   2016-04-23
                      lpfc                  EMU              VMwareCertified   2016-04-23
                      scsi-be2iscsi        EMU              VMwareCertified   2016-04-23
                      amsHelper               600.10.4.0-22.2494585                  HPE              PartnerSupported  2016-04-23
                      scsi-megaraid-mbox          VMware           VMwareCertified   2015-08-23
                      scsi-megaraid-sas       6.603.55.00-2vmw.600.0.0.2494585       VMware           VMwareCertified   2015-08-23
                      scsi-megaraid2          2.00.4-9vmw.600.0.0.2494585            VMware           VMwareCertified   2015-08-23
                      scsi-mptsas           VMware           VMwareCertified   2015-08-23
                      scsi-mptspi           VMware           VMwareCertified   2015-08-23
                      uhci-usb-uhci           1.0-3vmw.600.0.0.2494585               VMware           VMwareCertified   2015-08-23
                      vsan                    6.0.0-2.34.3563498                     VMware           VMwareCertified   2016-04-23
                      vsanhealth              6.0.0-3000000.         VMware           VMwareCertified   2016-04-23
                      xhci-xhci               1.0-3vmw.600.2.34.3620759              VMware           VMwareCertified   2016-04-23
                      tools-light             6.0.0-2.34.3620759                     VMware           VMwareCertified   2016-04-23
                      scsi-qla2xxx            911.k1.1-26OEM.500.0.0.472560          qlogic           VMwareCertified   2015-08-23

Open in new window

Searching in the Internet for the same type of issue, I did not find anything similar. The VMware KnowledgeBase has a solution for the “error code 99” by Initializing VUM without reinstall, but since this is a problem in one host, this could not be the solution for this particular issue.

I then found an article with a similar issue in Patrick Terlisten blog. Since I didn't want to reinstall the ESXi host, I needed to try this solution, even if was not 100% identical, it was almost the same.

I started by following Patrick's recommendation to copy the esximg files/folders from a working host (in this case esxi-02) to the corrupted ESXi host (esxi-01).

First if  you don't have your SSH Client enabled in the host firewall, you need to set it enabled to do the next task using the SCP command. To enable SSH Client in the source ESXi host:

[root@esx02:~] esxcli network firewall ruleset set --enabled true --ruleset-id=sshClient

Open in new window

Note: Don't forget to disable SSH Client after do this tasks.

After you run the SCP command you will be prompted for the root password of the remote host and once you have successfully authenticated the file will copy.

[root@esxi02:~] scp -r /var/db/esximg/ root@esxi02:/var/db
                      The authenticity of host 'esxi02 (esxi02)' can't be established.
                      RSA key fingerprint is SHA256:isiF8md0Q6GDazZ97fbJ/4ZiqxOrf9tE4mHv0XN64kM.
                      Are you sure you want to continue connecting (yes/no)? yes
                      Warning: Permanently added 'esxi02' (RSA) to the list of known hosts.
                      sata-sata-sil-1748273158.xml                                   100% 1717     1.7KB/s   00:00
                      lsi-mr3-989864457.xml                                          100% 1504     1.5KB/s   00:00
                      scsi-ips--1979861494.xml                                       100% 1619     1.6KB/s   00:00
                      char-hpcru--1874046437.xml                                     100% 1638     1.6KB/s   00:00
                      net-tg3--917722591.xml                                         100% 1461     1.4KB/s   00:00
                      ipmi-ipmi-devintf-1862766627.xml                               100% 1719     1.7KB/s   00:00
                      sata-sata-promise--860290007.xml                               100% 1742     1.7KB/s   00:00
                      esx-ui--629271967.xml                                          100% 53KB  53.2KB/s   00:00
                      cpu-microcode-1845960806.xml                                   100% 2381     2.3KB/s   00:00
                      scsi-bnx2i--630528917.xml                                      100% 1407     1.4KB/s   00:00
                      scsi-fnic-248026735.xml                                        100% 1784     1.7KB/s   00:00
                      scsi-hpdsa--570772364.xml                                      100% 1412     1.4KB/s   00:00
                      qlnativefc--775726474.xml                                      100% 1408     1.4KB/s   00:00
                      hpe-ilo--120862598.xml                                         100% 1550     1.5KB/s   00:00
                      net-cnic-1577337473.xml                                        100% 1389     1.4KB/s   00:00
                      scsi-qla4xxx-1755174022.xml                                    100% 1517     1.5KB/s   00:00
                      hpe-smx-provider-1463103112.xml                                100% 5426     5.3KB/s   00:00
                      net-bnx2x-272405923.xml                                        100% 1534     1.5KB/s   00:00
                      net-bnx2-984265687.xml                                         100% 1507     1.5KB/s   00:00
                      net-ixgbe--494567451.xml                                       100% 1415     1.4KB/s   00:00
                      scsi-mpt2sas--1969732120.xml                                   100% 1679     1.6KB/s   00:00
                      scsi-megaraid-mbox--26860043.xml                               100% 1775     1.7KB/s   00:00
                      scsi-aic79xx-757558775.xml                                     100% 1643     1.6KB/s   00:00
                      %28Updated%29%20HP-ESXi-6.0.0-iso-600.9.1.39-687127855         100% 23KB  23.4KB/s   00:00

Open in new window

After copying the files, just check if the files are in the ESXi host (esxi01).

Host Software profile:

[root@esx01:~] esxcli software profile get
                      (Updated) HP-ESXi-6.0.0-iso-600.9.1.39
                         Name: (Updated) HP-ESXi-6.0.0-iso-600.9.1.39
                         Vendor: deacvm-dsc-002
                         Creation Time: 2016-04-23T12:05:29
                         Modification Time: 2016-04-23T12:05:29
                         Stateless Ready: False
                            (Original Vendor):Hewlett-Packard
                            2016-04-23T12:05:29.506598+00:00: The following VIBs are
                      [root@esx01:~] ls -l /var/db/esximg/profiles/
                      total 24
                      -r--r--r--    1 root     root         23998 Apr 23 12:05 %28Updated%29%20HP-ESXi-6.0.0-iso-600.9.1.39-687127855

Open in new window

After checking that the files are copied into the destination, run the VUM Scan again on the host before and after staging to ensure no errors are found. After that I was able to remediate the ESXi host and fully update it.

Again, thanks to Patrick for his solution and for putting me on the right track.

Hope this article can you help fixing this error that you may encounter in your ESXi upgrade.

This is the part of my "TIP Articles". So, please vote "Helpful" on this Article. And I encourage your comments and feedback.
Luciano PatrãoICT Senior Infraestructure  Engineer  
vExpert vSAN, NSX, Cloud Provider, Veeam Vanguard, Virtual Backups, and Storage design, and an active blogger.

Comments (2)

Does this copy affect the running VMs ?
Worked perfectly.

Have a question about something in this article? You can receive help directly from the article author. Sign up for a free trial to get started.