vCenter Troubleshooting TIPS: ESXi 6.0 reports “error code 99” during Stage patches to entity VUM operation

Published on
7,783 Points
Last Modified:
Luciano Patrão
My main areas of expertise are VMware, Virtual Backups and Storage design.
I am a vExpert and a Technical Blogger in the same areas.
This is similar to the error described in my previous Article, but with a different source problem and a different solution. When trying to scan and stage the ESXi 6.0 updates using VMware Update Manager, we can get "error code 90"
The configuration is an Hewlett-Packard DL360 G7 with the ESXi 6.0 build 3620759.
When trying to use the VMware update manager in this host, the scan runs without any problem, but when trying to stage the updates, we can get:

Stage patches to entity esxi01.localdomain. The host returns esxupdate error code:99.
An unhandled exception was encountered. Check the Update Manager log files and esxupdate log files for more details.
Looking at the esxpudate.log (in ESXi var/log/) I see lot of failed tries to upload VIB files to the repository, along with this:

Open in new window

So we need to check the Host Software profile(not the same as Host Profile) and check the issue.

Then I notice that somehow the ESXi lost its software profile, so it was not possible to stage any update, or even install a VIB manually. When troubleshooting the issue, I found out that there was no software profile in this host.

root@esxi01:~] esxcli software profile get
No host image profile defined
Please refer to the log file for more details.

Open in new window

Also checking the VIB list in this host I get this:

[root@esxi01:~] esxcli software vib list

Name         Version             Vendor  Acceptance Level  Install Date
-----------  ------------------  ------  ----------------  ------------
tools-light  6.0.0-2.34.3620759  VMware  VMwareCertified   2016-04-23

Open in new window

So we need to check the VIB folder and profiles folder in /var/db/esximg and I noticed the two folders were empty.

[root@esx01:~] ls -l /var/db/esximg/*
total 0

total 0

Open in new window

When checking the same information from a working ESXi host I get this:

[root@esxi02:~] ls -l /var/db/esximg/profiles/
total 24
-r--r--r--    1 root     root         23998 Apr 23 12:05 %28Updated%29%20HP-ESXi-6.0.0-iso-600.9.1.39-687127855

[root@esxi02:~] esxcli software profile get
(Updated) HP-ESXi-6.0.0-iso-600.9.1.39
   Name: (Updated) HP-ESXi-6.0.0-iso-600.9.1.39
   Vendor: esxi02.localdomain
   Creation Time: 2016-04-23T12:05:31
   Modification Time: 2016-04-23T12:05:31
   Stateless Ready: True

      (Original Vendor):Hewlett-Packard
      2016-04-23T12:05:30.688280+00:00: The following VIBs are

Open in new window

And this:

[root@esxi02:~] esxcli software vib list
Name                           Version                         Vendor           Acceptance Level  Install Date
----------------------  -------------------------------------  ---------------  ----------------  ------------
net-tg3                 3.137l.v60.1-1OEM.600.0.0.2494585      BRCM             VMwareCertified   2016-01-23
elxnet               EMU              VMwareCertified   2016-04-23
ima-be2iscsi         EMU              VMwareCertified   2016-04-23
lpfc                  EMU              VMwareCertified   2016-04-23
scsi-be2iscsi        EMU              VMwareCertified   2016-04-23
amsHelper               600.10.4.0-22.2494585                  HPE              PartnerSupported  2016-04-23
scsi-megaraid-mbox          VMware           VMwareCertified   2015-08-23
scsi-megaraid-sas       6.603.55.00-2vmw.600.0.0.2494585       VMware           VMwareCertified   2015-08-23
scsi-megaraid2          2.00.4-9vmw.600.0.0.2494585            VMware           VMwareCertified   2015-08-23
scsi-mptsas           VMware           VMwareCertified   2015-08-23
scsi-mptspi           VMware           VMwareCertified   2015-08-23
uhci-usb-uhci           1.0-3vmw.600.0.0.2494585               VMware           VMwareCertified   2015-08-23
vsan                    6.0.0-2.34.3563498                     VMware           VMwareCertified   2016-04-23
vsanhealth              6.0.0-3000000.         VMware           VMwareCertified   2016-04-23
xhci-xhci               1.0-3vmw.600.2.34.3620759              VMware           VMwareCertified   2016-04-23
tools-light             6.0.0-2.34.3620759                     VMware           VMwareCertified   2016-04-23
scsi-qla2xxx            911.k1.1-26OEM.500.0.0.472560          qlogic           VMwareCertified   2015-08-23

Open in new window

Searching in the Internet for the same type of issue, I did not find anything similar. The VMware KnowledgeBase has a solution for the “error code 99” by Initializing VUM without reinstall, but since this is a problem in one host, this could not be the solution for this particular issue.

I then found an article with a similar issue in Patrick Terlisten blog. Since I didn't want to reinstall the ESXi host, I needed to try this solution, even if was not 100% identical, it was almost the same.

I started by following Patrick's recommendation to copy the esximg files/folders from a working host (in this case esxi-02) to the corrupted ESXi host (esxi-01).

First if  you don't have your SSH Client enabled in the host firewall, you need to set it enabled to do the next task using the SCP command. To enable SSH Client in the source ESXi host:

[root@esx02:~] esxcli network firewall ruleset set --enabled true --ruleset-id=sshClient

Open in new window

Note: Don't forget to disable SSH Client after do this tasks.

After you run the SCP command you will be prompted for the root password of the remote host and once you have successfully authenticated the file will copy.

[root@esxi02:~] scp -r /var/db/esximg/ root@esxi02:/var/db
The authenticity of host 'esxi02 (esxi02)' can't be established.
RSA key fingerprint is SHA256:isiF8md0Q6GDazZ97fbJ/4ZiqxOrf9tE4mHv0XN64kM.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'esxi02' (RSA) to the list of known hosts.
sata-sata-sil-1748273158.xml                                   100% 1717     1.7KB/s   00:00
lsi-mr3-989864457.xml                                          100% 1504     1.5KB/s   00:00
scsi-ips--1979861494.xml                                       100% 1619     1.6KB/s   00:00
char-hpcru--1874046437.xml                                     100% 1638     1.6KB/s   00:00
net-tg3--917722591.xml                                         100% 1461     1.4KB/s   00:00
ipmi-ipmi-devintf-1862766627.xml                               100% 1719     1.7KB/s   00:00
sata-sata-promise--860290007.xml                               100% 1742     1.7KB/s   00:00
esx-ui--629271967.xml                                          100% 53KB  53.2KB/s   00:00
cpu-microcode-1845960806.xml                                   100% 2381     2.3KB/s   00:00
scsi-bnx2i--630528917.xml                                      100% 1407     1.4KB/s   00:00
scsi-fnic-248026735.xml                                        100% 1784     1.7KB/s   00:00
scsi-hpdsa--570772364.xml                                      100% 1412     1.4KB/s   00:00
qlnativefc--775726474.xml                                      100% 1408     1.4KB/s   00:00
hpe-ilo--120862598.xml                                         100% 1550     1.5KB/s   00:00
net-cnic-1577337473.xml                                        100% 1389     1.4KB/s   00:00
scsi-qla4xxx-1755174022.xml                                    100% 1517     1.5KB/s   00:00
hpe-smx-provider-1463103112.xml                                100% 5426     5.3KB/s   00:00
net-bnx2x-272405923.xml                                        100% 1534     1.5KB/s   00:00
net-bnx2-984265687.xml                                         100% 1507     1.5KB/s   00:00
net-ixgbe--494567451.xml                                       100% 1415     1.4KB/s   00:00
scsi-mpt2sas--1969732120.xml                                   100% 1679     1.6KB/s   00:00
scsi-megaraid-mbox--26860043.xml                               100% 1775     1.7KB/s   00:00
scsi-aic79xx-757558775.xml                                     100% 1643     1.6KB/s   00:00
%28Updated%29%20HP-ESXi-6.0.0-iso-600.9.1.39-687127855         100% 23KB  23.4KB/s   00:00

Open in new window

After copying the files, just check if the files are in the ESXi host (esxi01).

Host Software profile:

[root@esx01:~] esxcli software profile get
(Updated) HP-ESXi-6.0.0-iso-600.9.1.39
   Name: (Updated) HP-ESXi-6.0.0-iso-600.9.1.39
   Vendor: deacvm-dsc-002
   Creation Time: 2016-04-23T12:05:29
   Modification Time: 2016-04-23T12:05:29
   Stateless Ready: False

      (Original Vendor):Hewlett-Packard
      2016-04-23T12:05:29.506598+00:00: The following VIBs are

[root@esx01:~] ls -l /var/db/esximg/profiles/
total 24
-r--r--r--    1 root     root         23998 Apr 23 12:05 %28Updated%29%20HP-ESXi-6.0.0-iso-600.9.1.39-687127855

Open in new window

After checking that the files are copied into the destination, run the VUM Scan again on the host before and after staging to ensure no errors are found. After that I was able to remediate the ESXi host and fully update it.

Again, thanks to Patrick for his solution and for putting me on the right track.

Hope this article can you help fixing this error that you may encounter in your ESXi upgrade.

This is the part of my "TIP Articles". So, please vote "Helpful" on this Article. And I encourage your comments and feedback.
1 Comment

Expert Comment

by:Kenneth Bolt
Does this copy affect the running VMs ?

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Join & Write a Comment

Teach the user how to edit .vmx files to add advanced configuration options Open vSphere Web Client: Edit Settings for a VM: Choose VM Options -> Advanced: Add Configuration Parameters:
This Micro Tutorial walks you through using a remote console to access a server and install ESXi 5.1. This example is showing remote access and installation using a Dell server. The hypervisor is the very first component of your virtual infrastructu…

Keep in touch with Experts Exchange

Tech news and trends delivered to your inbox every month