Solved

After upgrading to 5.1U2 from 5.0 I am getting bootbank errors and slow ssh connections

Posted on 2014-04-27
7
3,329 Views
Last Modified: 2014-05-01
We upgraded our vcenter and esxi hosts yesterday to 5.1U2 from 5.0 U1.

evertyhing looked to go smooth but this morning while trying to upgrade the vmware tools on a VM the it gave out these warning:

Unable to install VMware Tools. An error occurred
while trying to access image file "/usr/lib/vmware
 /isoimages/linux.iso" needed to install VMware
Tools: 0 (Input/output error). If your product
shipped with the VMware Tools package, reinstall
VMware ESX, then try again to install the VMware
Tools package in the virtual machine

Reseatching i found that it may be because the esxi host doesnt have the image. When i tried to SSH to the host, the commands were very slow. when i type it would freeze for about 5-6 minutes. i looked at the console and when i hit ALT-F11 i would get this:

Bootbank cannot be found at path '/bootbank'
(2014-04-27T14:26:57.583Z cpu5:70427)
 warning
 

When i hit ALT-F12 i get this:
When hitting ALT-F12
I noticed at least two of the hosts updated are exibiting this behavior. i do not notice it in the other 10 hosts. however i am home troubleshooting and dont have direct access to them. I used the offline bundle, update-from-esxi5.1-5.1_update02.zip, to update from 5.0 U1.

I do have a ticket to VMWare tech support but i was hoping someone could shed some light on this.

As a side note, the VM's seem to be working fine. Although it is a sunday and no heavy loads are on it right now.
0
Comment
Question by:msidnam
  • 4
  • 3
7 Comments
 
LVL 117

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
Comment Utility
Is your ESXi installation on flash, check through the /var/log/vmkernel.log

Check your Bootbank/ESXi installation OS media, is NOT FULL!

vdf or df at the console

We've noticed recently the ESXi installations, are getting larger or log more, and you really need to move the /var/log (scratch) to persistent storage.

if it's upgraded an existing installation, using inplace offline bundle, although this completes, it does use more storage because of additional drivers and code.

This is where upgrades get you by the short and curlies, and sometimes, a fresh new virgin installation is better, because it re-creates all new partitions and symlinks, rather than relying on older ESXi 5.0 partition tables which are smaller!
0
 
LVL 2

Author Comment

by:msidnam
Comment Utility
The install is on the local datastore. i made a folder called 5.1U2 and used esxcli to update from 5.0U1 to 5.1U2. I am trying to view the vmkernel.log file but the SSH connection times out for several minutes and its making it difficult

I do have the scratch log going to a datastore on our SAN. Should i move that back to local drives? i moved it to the SAN a while back since we were having issues of hosts locking up at one point and when i rebooted the logs were cleared.

I had to remove our DVD drives a while back since they were the TEAC drives causing issues: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1017297.

If i need to install virgin can i do so from a USB? I will need to remember how to do a host profile since we have a ton of NICS and vswitches.

we may be in for a long haul. :)
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
No, keep the logs on the SAN!

Yes, you can install from USB, a new installation.

You can backup the ESXi configuratiion, and re-install to a virgin installation!
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 2

Author Comment

by:msidnam
Comment Utility
Thank you for the info. strange things are happening. I rebotted the server and it told me i had multibit ecc errors on the RAID controller memory. when it rebooted i didnt get the bootbank errors and the ALT-F12 kernel logs seemed to have calmed down.[newlog
then i rebooted again and no raid controller errors or bootbank errors. this is very strange. im going to try and reinstall asap but hopefully this gives me a little time.
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
Check your storage comntroller?
0
 
LVL 2

Author Comment

by:msidnam
Comment Utility
it looks like i need new memory or for the RAID controller or a new controller. I'm thinking this is the reason for all the craziness
0
 
LVL 2

Author Comment

by:msidnam
Comment Utility
A quick update in case anyone else has a similar issue. I needed to update my RAID controller firmware. I found this post: https://communities.vmware.com/message/2249566?z=PNeQaQ and updated the firmware.
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

VM backup deduplication is a method of reducing the amount of storage space needed to save VM backups. In most organizations, VMs contain many duplicate copies of data, such as VMs deployed from the same template, VMs with the same OS, or VMs that h…
Last article we focus in how to VMware: How to create and use VMs TAGs – Part 1 so before follow this article and perform the next tasks, you should read the first article how to create the TAG before using them in Veeam Backup Jobs.
Teach the user how to install log collectors and how to configure ESXi 5.5 for remote logging Open console session and mount vCenter Server installer: Install vSphere Core Dump Collector: Install vSphere Syslog Collector: Open vSphere Client: Config…
This Micro Tutorial steps you through the configuration steps to configure your ESXi host Management Network settings and test the management network, ensure the host is recognized by the DNS Server, configure a new password, and the troubleshooting…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

7 Experts available now in Live!

Get 1:1 Help Now