vCenter Troubleshooting TIPS: ESXi 6.0 reports “error code 99” during Stage patches to entity VUM operation
vExpert vSAN, NSX, Cloud Provider, Veeam Vanguard, Virtual Backups, and Storage design, and an active blogger.
Published:
Browse All Articles > vCenter Troubleshooting TIPS: ESXi 6.0 reports “error code 99” during Stage patches to entity VUM operation
This is similar to the error described in my previous Article, but with a different source problem and a different solution. When trying to scan and stage the ESXi 6.0 updates using VMware Update Manager, we can get "error code 90"
The configuration is an Hewlett-Packard DL360 G7 with the ESXi 6.0 build 3620759.
When trying to use the VMware update manager in this host, the scan runs without any problem, but when trying to stage the updates, we can get:
![image.gif]()
![image.gif]()
So we need to check the Host Software profile(not the same as Host Profile) and check the issue.
Then I notice that somehow the ESXi lost its software profile, so it was not possible to stage any update, or even install a VIB manually. When troubleshooting the issue, I found out that there was no software profile in this host.
Also checking the VIB list in this host I get this:
So we need to check the VIB folder and profiles folder in /var/db/esximg and I noticed the two folders were empty.
When checking the same information from a working ESXi host I get this:
And this:
Searching in the Internet for the same type of issue, I did not find anything similar. The VMware KnowledgeBase has a solution for the “error code 99” by Initializing VUM without reinstall, but since this is a problem in one host, this could not be the solution for this particular issue.
I then found an article with a similar issue in Patrick Terlisten blog. Since I didn't want to reinstall the ESXi host, I needed to try this solution, even if was not 100% identical, it was almost the same.
I started by following Patrick's recommendation to copy the esximg files/folders from a working host (in this case esxi-02) to the corrupted ESXi host (esxi-01).
First if you don't have your SSH Client enabled in the host firewall, you need to set it enabled to do the next task using the SCP command. To enable SSH Client in the source ESXi host:
Note: Don't forget to disable SSH Client after do this tasks.
After you run the SCP command you will be prompted for the root password of the remote host and once you have successfully authenticated the file will copy.
After copying the files, just check if the files are in the ESXi host (esxi01).
Host Software profile:
After checking that the files are copied into the destination, run the VUM Scan again on the host before and after staging to ensure no errors are found. After that I was able to remediate the ESXi host and fully update it.
Again, thanks to Patrick for his solution and for putting me on the right track.
Hope this article can you help fixing this error that you may encounter in your ESXi upgrade.
This is the part of my "TIP Articles". So, please vote "Helpful" on this Article. And I encourage your comments and feedback.
When trying to use the VMware update manager in this host, the scan runs without any problem, but when trying to stage the updates, we can get:
Looking at the esxpudate.log (in ESXi var/log/) I see lot of failed tries to upload VIB files to the repository, along with this:
Stage patches to entity esxi01.localdomain. The host returns esxupdate error code:99.
An unhandled exception was encountered. Check the Update Manager log files and esxupdate log files for more details.
So we need to check the Host Software profile(not the same as Host Profile) and check the issue.
Then I notice that somehow the ESXi lost its software profile, so it was not possible to stage any update, or even install a VIB manually. When troubleshooting the issue, I found out that there was no software profile in this host.
root@esxi01:~] esxcli software profile get
Exception]
No host image profile defined
Please refer to the log file for more details.Also checking the VIB list in this host I get this:
[root@esxi01:~] esxcli software vib list
Name Version Vendor Acceptance Level Install Date
----------- ------------------ ------ ---------------- ------------
tools-light 6.0.0-2.34.3620759 VMware VMwareCertified 2016-04-23So we need to check the VIB folder and profiles folder in /var/db/esximg and I noticed the two folders were empty.
[root@esx01:~] ls -l /var/db/esximg/*
/var/db/esximg/profiles:
total 0
/var/db/esximg/vibs:
total 0
[root@esx01:~]When checking the same information from a working ESXi host I get this:
[root@esxi02:~] ls -l /var/db/esximg/profiles/
total 24
-r--r--r-- 1 root root 23998 Apr 23 12:05 %28Updated%29%20HP-ESXi-6.0.0-iso-600.9.1.39-687127855
[root@esxi02:~] esxcli software profile get
(Updated) HP-ESXi-6.0.0-iso-600.9.1.39
Name: (Updated) HP-ESXi-6.0.0-iso-600.9.1.39
Vendor: esxi02.localdomain
Creation Time: 2016-04-23T12:05:31
Modification Time: 2016-04-23T12:05:31
Stateless Ready: True
Description:
(Original Vendor):Hewlett-Packard
2016-04-23T12:05:30.688280+00:00: The following VIBs are
installed:
hpssacli 2.40.13.0-6.0.0.1854445
hponcfg 6.0.0.04-00.14.4.2494585
conrep 6.0.0.01-01.00.7.2494585
hptestevent 6.0.0.01-01.00.5.2494585
hpbootcfg 6.0.0.02-02.00.6.2494585And this:
[root@esxi02:~] esxcli software vib list
Name Version Vendor Acceptance Level Install Date
---------------------- ------------------------------------- --------------- ---------------- ------------
net-tg3 3.137l.v60.1-1OEM.600.0.0.2494585 BRCM VMwareCertified 2016-01-23
elxnet 10.7.110.13-1OEM.600.0.0.2768847 EMU VMwareCertified 2016-04-23
ima-be2iscsi 10.7.110.10-1OEM.600.0.0.2159203 EMU VMwareCertified 2016-04-23
lpfc 10.7.110.4-1OEM.600.0.0.2768847 EMU VMwareCertified 2016-04-23
scsi-be2iscsi 10.7.110.10-1OEM.600.0.0.2159203 EMU VMwareCertified 2016-04-23
amsHelper 600.10.4.0-22.2494585 HPE PartnerSupported 2016-04-23
....
....
scsi-megaraid-mbox 2.20.5.1-6vmw.600.0.0.2494585 VMware VMwareCertified 2015-08-23
scsi-megaraid-sas 6.603.55.00-2vmw.600.0.0.2494585 VMware VMwareCertified 2015-08-23
scsi-megaraid2 2.00.4-9vmw.600.0.0.2494585 VMware VMwareCertified 2015-08-23
scsi-mptsas 4.23.01.00-9vmw.600.0.0.2494585 VMware VMwareCertified 2015-08-23
scsi-mptspi 4.23.01.00-9vmw.600.0.0.2494585 VMware VMwareCertified 2015-08-23
uhci-usb-uhci 1.0-3vmw.600.0.0.2494585 VMware VMwareCertified 2015-08-23
vsan 6.0.0-2.34.3563498 VMware VMwareCertified 2016-04-23
vsanhealth 6.0.0-3000000.3.0.2.34.3544323 VMware VMwareCertified 2016-04-23
xhci-xhci 1.0-3vmw.600.2.34.3620759 VMware VMwareCertified 2016-04-23
tools-light 6.0.0-2.34.3620759 VMware VMwareCertified 2016-04-23
scsi-qla2xxx 911.k1.1-26OEM.500.0.0.472560 qlogic VMwareCertified 2015-08-23Searching in the Internet for the same type of issue, I did not find anything similar. The VMware KnowledgeBase has a solution for the “error code 99” by Initializing VUM without reinstall, but since this is a problem in one host, this could not be the solution for this particular issue.
I then found an article with a similar issue in Patrick Terlisten blog. Since I didn't want to reinstall the ESXi host, I needed to try this solution, even if was not 100% identical, it was almost the same.
I started by following Patrick's recommendation to copy the esximg files/folders from a working host (in this case esxi-02) to the corrupted ESXi host (esxi-01).
First if you don't have your SSH Client enabled in the host firewall, you need to set it enabled to do the next task using the SCP command. To enable SSH Client in the source ESXi host:
[root@esx02:~] esxcli network firewall ruleset set --enabled true --ruleset-id=sshClientNote: Don't forget to disable SSH Client after do this tasks.
After you run the SCP command you will be prompted for the root password of the remote host and once you have successfully authenticated the file will copy.
[root@esxi02:~] scp -r /var/db/esximg/ root@esxi02:/var/db
The authenticity of host 'esxi02 (esxi02)' can't be established.
RSA key fingerprint is SHA256:isiF8md0Q6GDazZ97fbJ/4ZiqxOrf9tE4mHv0XN64kM.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'esxi02' (RSA) to the list of known hosts.
Password:
sata-sata-sil-1748273158.xml 100% 1717 1.7KB/s 00:00
lsi-mr3-989864457.xml 100% 1504 1.5KB/s 00:00
scsi-ips--1979861494.xml 100% 1619 1.6KB/s 00:00
char-hpcru--1874046437.xml 100% 1638 1.6KB/s 00:00
net-tg3--917722591.xml 100% 1461 1.4KB/s 00:00
ipmi-ipmi-devintf-1862766627.xml 100% 1719 1.7KB/s 00:00
sata-sata-promise--860290007.xml 100% 1742 1.7KB/s 00:00
esx-ui--629271967.xml 100% 53KB 53.2KB/s 00:00
cpu-microcode-1845960806.xml 100% 2381 2.3KB/s 00:00
scsi-bnx2i--630528917.xml 100% 1407 1.4KB/s 00:00
scsi-fnic-248026735.xml 100% 1784 1.7KB/s 00:00
scsi-hpdsa--570772364.xml 100% 1412 1.4KB/s 00:00
qlnativefc--775726474.xml 100% 1408 1.4KB/s 00:00
hpe-ilo--120862598.xml 100% 1550 1.5KB/s 00:00
net-cnic-1577337473.xml 100% 1389 1.4KB/s 00:00
scsi-qla4xxx-1755174022.xml 100% 1517 1.5KB/s 00:00
hpe-smx-provider-1463103112.xml 100% 5426 5.3KB/s 00:00
...
...
net-bnx2x-272405923.xml 100% 1534 1.5KB/s 00:00
net-bnx2-984265687.xml 100% 1507 1.5KB/s 00:00
net-ixgbe--494567451.xml 100% 1415 1.4KB/s 00:00
scsi-mpt2sas--1969732120.xml 100% 1679 1.6KB/s 00:00
scsi-megaraid-mbox--26860043.xml 100% 1775 1.7KB/s 00:00
scsi-aic79xx-757558775.xml 100% 1643 1.6KB/s 00:00
%28Updated%29%20HP-ESXi-6.0.0-iso-600.9.1.39-687127855 100% 23KB 23.4KB/s 00:00
[root@esxi02:~]After copying the files, just check if the files are in the ESXi host (esxi01).
Host Software profile:
[root@esx01:~] esxcli software profile get
(Updated) HP-ESXi-6.0.0-iso-600.9.1.39
Name: (Updated) HP-ESXi-6.0.0-iso-600.9.1.39
Vendor: deacvm-dsc-002
Creation Time: 2016-04-23T12:05:29
Modification Time: 2016-04-23T12:05:29
Stateless Ready: False
Description:
(Original Vendor):Hewlett-Packard
2016-04-23T12:05:29.506598+00:00: The following VIBs are
installed:
hpssacli 2.40.13.0-6.0.0.1854445
hponcfg 6.0.0.04-00.14.4.2494585
conrep 6.0.0.01-01.00.7.2494585
hptestevent 6.0.0.01-01.00.5.2494585
hpbootcfg 6.0.0.02-02.00.6.2494585
[root@esx01:~] ls -l /var/db/esximg/profiles/
total 24
-r--r--r-- 1 root root 23998 Apr 23 12:05 %28Updated%29%20HP-ESXi-6.0.0-iso-600.9.1.39-687127855
[root@esx01:~]After checking that the files are copied into the destination, run the VUM Scan again on the host before and after staging to ensure no errors are found. After that I was able to remediate the ESXi host and fully update it.
Again, thanks to Patrick for his solution and for putting me on the right track.
Hope this article can you help fixing this error that you may encounter in your ESXi upgrade.
This is the part of my "TIP Articles". So, please vote "Helpful" on this Article. And I encourage your comments and feedback.
vExpert vSAN, NSX, Cloud Provider, Veeam Vanguard, Virtual Backups, and Storage design, and an active blogger.
Have a question about something in this article? You can receive help directly from the article author. Sign up for a free trial to get started.
Comments (2)
Commented:
Commented: