VMWare ESXi 5.1 Host upgrade to 5.5 Error: Ramdisk "root" is full!

Hi all;

I having an issue with an ESXi host upgrade to 5.5. The error is: The ramdisk 'root' is full. As a result the file /usr/share/vua/vua could not be written.

I checked the "vmware-vum-server-log4cpp.log" log file, and found the following:

[2015-07-21 08:52:13:211 'SingleHostUpgradeScanTask.SingleHostUpgradeScanTask{28}' 4116 ERROR] [singleHostUpgradeScanTask, 277] Error while scanning: Error connecting. Error code: 11, message: 514 Error connecting to vua service instance.

[2015-07-21 08:52:13:220 'ChainedTaskContainer.ChainedTaskContainer{27}' 4116 INFO] [vciTaskBase, 587] A subTask finished: SingleHostUpgradeScanTask{28}

Open in new window



I connected to the host via SSH, and performed the following commands:

1. vdf -h

The output looks fine:

Ramdisk                   Size      Used Available Use% Mounted on
root                       32M       18M       13M  58% --
etc                        28M      944K       27M   3% --
tmp                       192M       20K      191M   0% --
hostdstats                346M        5M      340M   1% --

Open in new window


2. df /var/log

Non of the log files is on 100% usage:

~ # df /var/log
Filesystem         Bytes         Used     Available Use% Mounted on
VMFS-5     1099243192320  96721698816 1002521493504   9% /vmfs/volumes/ESX_TFI_Replica_DS3
VMFS-5      141465485312  20209205248  121256280064  14% /vmfs/volumes/datastore1
VMFS-5      107105746944  15900606464   91205140480  15% /vmfs/volumes/TEMPDATASTORE-TEST
VMFS-5     2198754820096  65953333248 2132801486848   3% /vmfs/volumes/ESX_TFI_Replica_DS2
VMFS-5     2198754820096 205563887616 1993190932480   9% /vmfs/volumes/ESX_TFI_Replica_DS1
VMFS-5     1169304846336 522926948352  646377897984  45% /vmfs/volumes/UDS_VMAX4457_SK_405E_01_BC
VMFS-5     1169304846336 276016660480  893288185856  24% /vmfs/volumes/UDS_VMAX4457_SK_4066_02_BC
VMFS-5     1169304846336 763790098432  405514747904  65% /vmfs/volumes/UDS_VMAX4457_SK_406E_03_BC
VMFS-5     1169304846336 428430327808  740874518528  37% /vmfs/volumes/UDS_VMAX4457_SK_4076_04_BC
VMFS-5     1169304846336 237407043584  931897802752  20% /vmfs/volumes/UDS_VMAX4457_SK_407E_05_BC
VMFS-5     1169304846336 460148703232  709156143104  39% /vmfs/volumes/UDS_VMAX4457_SK_4086_06_BC
VMFS-5     1169304846336 291003957248  878300889088  25% /vmfs/volumes/UDS_VMAX4457_SK_4164_07_BC
VMFS-5     1169304846336 661584347136  507720499200  57% /vmfs/volumes/UDS_VMAX4457_SK_416C_08_BC
VMFS-5     1169304846336  86991962112 1082312884224   7% /vmfs/volumes/UDS_VMAX4457_SK_4174_09_BC
VMFS-5     1169304846336 216023433216  953281413120  18% /vmfs/volumes/UDS_VMAX4457_SK_417C_10_BC
VMFS-5     1169304846336 209093394432  960211451904  18% /vmfs/volumes/UDS_VMAX4457_SK_4184_11_BC
VMFS-5     1169304846336 652572884992  516731961344  56% /vmfs/volumes/UDS_VMAX4457_SK_418C_12_BC
VMFS-5     1169304846336 222999609344  946305236992  19% /vmfs/volumes/UDS_VMAX4457_SK_4194_13_BC
VMFS-5     1169304846336 780012617728  389292228608  67% /vmfs/volumes/UDS_VMAX4457_SK_3EBE_14_BC
VMFS-5     1169304846336 989813800960  179491045376  85% /vmfs/volumes/UDS_VMAX4457_SK_3EC6_15_BC
VMFS-5     1169304846336 628352876544  540951969792  54% /vmfs/volumes/UDS_VMAX4457_SK_3ECE_16_BC
VMFS-5     1169304846336 826799030272  342505816064  71% /vmfs/volumes/UDS_VMAX4457_SK_3ED6_17_BC
VMFS-5     1169304846336 813572292608  355732553728  70% /vmfs/volumes/UDS_VMAX4457_SK_3EE6_19_BC
VMFS-5     1169304846336 458150117376  711154728960  39% /vmfs/volumes/UDS_VMAX4457_SK_3EF6_21_BC
VMFS-5     1169304846336 188982755328  980322091008  16% /vmfs/volumes/UDS_VMAX4457_SK_3EFE_22_BC
VMFS-5     1169304846336 277681799168  891623047168  24% /vmfs/volumes/UDS_VMAX4457_SK_1E78_23_BC
VMFS-5     1169304846336 221261070336  948043776000  19% /vmfs/volumes/UDS_VMAX4457_SK_1E80_24_BC
VMFS-5     1169304846336 845222510592  324082335744  72% /vmfs/volumes/UDS_VMAX4457_SK_1E88_25_BC
VMFS-5     2338878128128   1028653056 2337849475072   0% /vmfs/volumes/UDS_VMAX4457_SK_35a6_26_BC
VMFS-5     2338878128128 484265951232 1854612176896  21% /vmfs/volumes/UDS_VMAX4457_SK_35F6_27_BC
VMFS-5     1169304846336 121338068992 1047966777344  10% /vmfs/volumes/UDS_VMAX4457_SK_2948_28_BC
vfat          4293591040    155779072    4137811968   4% /vmfs/volumes/545cceb3-89ec2368-7ddf-2c768a4e54f8
vfat           261853184    156082176     105771008  60% /vmfs/volumes/43006d70-4e7d55cd-e9de-00383ca16b34
vfat           261853184    182595584      79257600  70% /vmfs/volumes/ce0a8079-47efd824-ed08-d74680150f8b
vfat           299712512    215588864      84123648  72% /vmfs/volumes/545cceac-029dc45d-d013-2c768a4e54f8
~ #

Open in new window


3. ls -l /

 # ls -l /
lrwxrwxrwx    1 root     root            49 Jul 17 16:56 altbootbank -> /vmfs/volumes/43006d70-4e7d55cd-e9de-00383ca16b34
drwxr-xr-x    1 root     root           512 Jul 17 16:57 bin
lrwxrwxrwx    1 root     root            49 Jul 17 16:56 bootbank -> /vmfs/volumes/ce0a8079-47efd824-ed08-d74680150f8b
-r--r--r--    1 root     root        300060 Mar 23  2013 bootpart.gz
drwxr-xr-x    1 root     root           512 Jul 21 08:13 dev
drwxr-xr-x    1 root     root           512 Jul 20 15:20 etc
drwxr-xr-x    1 root     root           512 Jul 17 16:52 kdriver
drwxr-xr-x    1 root     root           512 Jul 17 16:52 lib
drwxr-xr-x    1 root     root           512 Jul 17 16:52 lib64
-r-x------    1 root     root        561125 Jul 17 16:49 local.tgz
lrwxrwxrwx    1 root     root             6 Jul 17 16:56 locker -> /store
drwxr-xr-x    1 root     root           512 Jul 17 16:52 mbr
drwxr-xr-x    1 root     root           512 Jul 17 16:52 opt
drwxr-xr-x    1 root     root        131072 Jul 21 08:13 proc
lrwxrwxrwx    1 root     root            22 Jul 17 16:56 productLocker -> /locker/packages/5.1.0
drwxr-xr-x    1 root     root           512 Jul 17 16:56 sbin
lrwxrwxrwx    1 root     root            49 Jul 17 16:56 scratch -> /vmfs/volumes/545cceb3-89ec2368-7ddf-2c768a4e54f8
lrwxrwxrwx    1 root     root            49 Jul 17 16:56 store -> /vmfs/volumes/545cceac-029dc45d-d013-2c768a4e54f8
drwxr-xr-x    1 root     root           512 Jul 17 16:52 tardisks
drwxr-xr-x    1 root     root           512 Jul 17 16:52 tardisks.noauto
drwxrwxrwt    1 root     root           512 Jul 21 08:13 tmp
drwxr-xr-x    1 root     root           512 Jul 17 16:52 usr
drwxr-xr-x    1 root     root           512 Jul 17 16:56 var
drwxr-xr-x    1 root     root           512 Jul 17 16:52 vmfs
drwxr-xr-x    1 root     root           512 Jul 17 16:52 vmimages
lrwxrwxrwx    1 root     root            17 Mar 23  2013 vmupgrade -> /locker/vmupgrade 

Open in new window


4. du -ch /vmimages /var/core /root /var/log

~ # du -ch /vmimages /var/core /root /var/log
12.0K   /vmimages
4.0K    /var/core
du: /root: No such file or directory
4.0K    /var/log/vmware/journal
8.0K    /var/log/vmware
4.0K    /var/log/ipmi/0
8.0K    /var/log/ipmi
220.0K  /var/log
236.0K  total 

Open in new window


5. ls -l /var/log

 # ls -l /var/log
lrwxrwxrwx    1 root     root            21 Jul 17 16:56 Xorg.log -> /scratch/log/Xorg.log
lrwxrwxrwx    1 root     root            21 Jul 17 16:56 auth.log -> /scratch/log/auth.log
-rw-rw-rw-    1 root     root         46709 Jul 17 16:56 boot.gz
-rw-r--r--    1 root     root         13917 Jul 17 16:57 configRP.log
lrwxrwxrwx    1 root     root            25 Jul 17 16:56 dhclient.log -> /scratch/log/dhclient.log
-rw-r--r--    1 root     root           649 Jul 17 16:57 esxcli.log
lrwxrwxrwx    1 root     root            26 Jul 17 16:56 esxupdate.log -> /scratch/log/esxupdate.log
lrwxrwxrwx    1 root     root            20 Jul 17 16:56 fdm.log -> /scratch/log/fdm.log
lrwxrwxrwx    1 root     root            28 Jul 17 16:56 hostd-probe.log -> /scratch/log/hostd-probe.log
lrwxrwxrwx    1 root     root            22 Jul 17 16:56 hostd.log -> /scratch/log/hostd.log
lrwxrwxrwx    1 root     root            33 Jul 17 16:56 hostprofiletrace.log -> /scratch/log/hostprofiletrace.log
lrwxrwxrwx    1 root     root            25 Jul 20 15:37 hpHelper.log -> /scratch/log/hpHelper.log
drwxr-xr-x    1 root     root           512 Jul 17 16:57 ipmi
lrwxrwxrwx    1 root     root            27 Jun 29 12:33 kdriver.log -> /scratch/log/kdriver.log.00
lrwxrwxrwx    1 root     root            21 Jul 17 16:56 lacp.log -> /scratch/log/lacp.log
lrwxrwxrwx    1 root     root            27 Jul 17 16:56 rhttpproxy.log -> /scratch/log/rhttpproxy.log
lrwxrwxrwx    1 root     root            22 Jul 17 16:56 shell.log -> /scratch/log/shell.log
-rw-------    1 root     root          5389 Jul 17 16:57 smbios.bin
lrwxrwxrwx    1 root     root            26 Jul 17 16:56 storagerm.log -> /scratch/log/storagerm.log
-rw-------    1 root     root          4179 Jul 17 16:56 sysboot.log
lrwxrwxrwx    1 root     root            23 Jul 17 16:56 syslog.log -> /scratch/log/syslog.log
lrwxrwxrwx    1 root     root            20 Jul 17 16:56 usb.log -> /scratch/log/usb.log
lrwxrwxrwx    1 root     root            24 Jul 17 16:56 vmamqpd.log -> /scratch/log/vmamqpd.log
lrwxrwxrwx    1 root     root            24 Jul 17 16:56 vmauthd.log -> /scratch/log/vmauthd.log
-rw-r--r--    1 root     root         10573 Jul 17 16:52 vmkdevmgr.log
lrwxrwxrwx    1 root     root            25 Jul 17 16:56 vmkernel.log -> /scratch/log/vmkernel.log
lrwxrwxrwx    1 root     root            26 Jul 17 16:56 vmkeventd.log -> /scratch/log/vmkeventd.log
lrwxrwxrwx    1 root     root            27 Jul 17 16:56 vmksummary.log -> /scratch/log/vmksummary.log
lrwxrwxrwx    1 root     root            27 Jul 17 16:56 vmkwarning.log -> /scratch/log/vmkwarning.log
drwxr-xr-x    1 root     root           512 Jul 17 16:57 vmware
lrwxrwxrwx    1 root     root            21 Jul 17 16:56 vobd.log -> /scratch/log/vobd.log
lrwxrwxrwx    1 root     root            24 Jul 17 16:56 vprobed.log -> /scratch/log/vprobed.log
lrwxrwxrwx    1 root     root            21 Jul 17 16:56 vpxa.log -> /scratch/log/vpxa.log 

Open in new window


And I still don't see anything wrong!

Can you please help...

Thanks...
Kinan Al-HaffarSenior Systems EngineerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Are you using an OEM version of any additional VIBs (drivers).

We would recommend FULL installations, rather than upgrades.
Kinan Al-HaffarSenior Systems EngineerAuthor Commented:
Hi Andrew;

Thank you for your answer.

The used image for the upgrade contains all the OEM derivers "HP Gen8" in this case. And we have upgraded several similar hosts with no issues. Therefore; I don't think that the drivers are causing the issue.

Thanks.
Andreas PeetzCommented:
Hi,

can you please check KB2055924 if it describes your issue?
Try to delete /var/log/hpHelper.log as described in the article and try the upgrade again.

Andreas
Kinan Al-HaffarSenior Systems EngineerAuthor Commented:
Hi Andreas;

Thank you for your reply. However; I already tried that, and nothing changed.

Thank you.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
see this issue with the Gen8, seems to be a common issues, with a workaround

http://h20564.www2.hp.com/hpsc/doc/public/display?docId=mmr_kc-0111843&sp4ts.oid=4010459

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.