After upgrading to 5.1U2 from 5.0 I am getting bootbank errors and slow ssh connections

We upgraded our vcenter and esxi hosts yesterday to 5.1U2 from 5.0 U1.

evertyhing looked to go smooth but this morning while trying to upgrade the vmware tools on a VM the it gave out these warning:

Unable to install VMware Tools. An error occurred
while trying to access image file "/usr/lib/vmware
 /isoimages/linux.iso" needed to install VMware
Tools: 0 (Input/output error). If your product
shipped with the VMware Tools package, reinstall
VMware ESX, then try again to install the VMware
Tools package in the virtual machine

Reseatching i found that it may be because the esxi host doesnt have the image. When i tried to SSH to the host, the commands were very slow. when i type it would freeze for about 5-6 minutes. i looked at the console and when i hit ALT-F11 i would get this:

Bootbank cannot be found at path '/bootbank'
(2014-04-27T14:26:57.583Z cpu5:70427)
 warning
 

When i hit ALT-F12 i get this:
When hitting ALT-F12
I noticed at least two of the hosts updated are exibiting this behavior. i do not notice it in the other 10 hosts. however i am home troubleshooting and dont have direct access to them. I used the offline bundle, update-from-esxi5.1-5.1_update02.zip, to update from 5.0 U1.

I do have a ticket to VMWare tech support but i was hoping someone could shed some light on this.

As a side note, the VM's seem to be working fine. Although it is a sunday and no heavy loads are on it right now.
LVL 2
msidnamAsked:
Who is Participating?
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
Is your ESXi installation on flash, check through the /var/log/vmkernel.log

Check your Bootbank/ESXi installation OS media, is NOT FULL!

vdf or df at the console

We've noticed recently the ESXi installations, are getting larger or log more, and you really need to move the /var/log (scratch) to persistent storage.

if it's upgraded an existing installation, using inplace offline bundle, although this completes, it does use more storage because of additional drivers and code.

This is where upgrades get you by the short and curlies, and sometimes, a fresh new virgin installation is better, because it re-creates all new partitions and symlinks, rather than relying on older ESXi 5.0 partition tables which are smaller!
0
 
msidnamAuthor Commented:
The install is on the local datastore. i made a folder called 5.1U2 and used esxcli to update from 5.0U1 to 5.1U2. I am trying to view the vmkernel.log file but the SSH connection times out for several minutes and its making it difficult

I do have the scratch log going to a datastore on our SAN. Should i move that back to local drives? i moved it to the SAN a while back since we were having issues of hosts locking up at one point and when i rebooted the logs were cleared.

I had to remove our DVD drives a while back since they were the TEAC drives causing issues: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1017297.

If i need to install virgin can i do so from a USB? I will need to remember how to do a host profile since we have a ton of NICS and vswitches.

we may be in for a long haul. :)
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
No, keep the logs on the SAN!

Yes, you can install from USB, a new installation.

You can backup the ESXi configuratiion, and re-install to a virgin installation!
0
Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

 
msidnamAuthor Commented:
Thank you for the info. strange things are happening. I rebotted the server and it told me i had multibit ecc errors on the RAID controller memory. when it rebooted i didnt get the bootbank errors and the ALT-F12 kernel logs seemed to have calmed down.[newlog
then i rebooted again and no raid controller errors or bootbank errors. this is very strange. im going to try and reinstall asap but hopefully this gives me a little time.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Check your storage comntroller?
0
 
msidnamAuthor Commented:
it looks like i need new memory or for the RAID controller or a new controller. I'm thinking this is the reason for all the craziness
0
 
msidnamAuthor Commented:
A quick update in case anyone else has a similar issue. I needed to update my RAID controller firmware. I found this post: https://communities.vmware.com/message/2249566?z=PNeQaQ and updated the firmware.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.