Solved

After upgrading to 5.1U2 from 5.0 I am getting bootbank errors and slow ssh connections

Posted on 2014-04-27
7
3,614 Views
Last Modified: 2014-05-01
We upgraded our vcenter and esxi hosts yesterday to 5.1U2 from 5.0 U1.

evertyhing looked to go smooth but this morning while trying to upgrade the vmware tools on a VM the it gave out these warning:

Unable to install VMware Tools. An error occurred
while trying to access image file "/usr/lib/vmware
 /isoimages/linux.iso" needed to install VMware
Tools: 0 (Input/output error). If your product
shipped with the VMware Tools package, reinstall
VMware ESX, then try again to install the VMware
Tools package in the virtual machine

Reseatching i found that it may be because the esxi host doesnt have the image. When i tried to SSH to the host, the commands were very slow. when i type it would freeze for about 5-6 minutes. i looked at the console and when i hit ALT-F11 i would get this:

Bootbank cannot be found at path '/bootbank'
(2014-04-27T14:26:57.583Z cpu5:70427)
 warning
 

When i hit ALT-F12 i get this:
When hitting ALT-F12
I noticed at least two of the hosts updated are exibiting this behavior. i do not notice it in the other 10 hosts. however i am home troubleshooting and dont have direct access to them. I used the offline bundle, update-from-esxi5.1-5.1_update02.zip, to update from 5.0 U1.

I do have a ticket to VMWare tech support but i was hoping someone could shed some light on this.

As a side note, the VM's seem to be working fine. Although it is a sunday and no heavy loads are on it right now.
0
Comment
Question by:msidnam
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
7 Comments
 
LVL 121

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 500 total points
ID: 40025871
Is your ESXi installation on flash, check through the /var/log/vmkernel.log

Check your Bootbank/ESXi installation OS media, is NOT FULL!

vdf or df at the console

We've noticed recently the ESXi installations, are getting larger or log more, and you really need to move the /var/log (scratch) to persistent storage.

if it's upgraded an existing installation, using inplace offline bundle, although this completes, it does use more storage because of additional drivers and code.

This is where upgrades get you by the short and curlies, and sometimes, a fresh new virgin installation is better, because it re-creates all new partitions and symlinks, rather than relying on older ESXi 5.0 partition tables which are smaller!
0
 
LVL 2

Author Comment

by:msidnam
ID: 40025884
The install is on the local datastore. i made a folder called 5.1U2 and used esxcli to update from 5.0U1 to 5.1U2. I am trying to view the vmkernel.log file but the SSH connection times out for several minutes and its making it difficult

I do have the scratch log going to a datastore on our SAN. Should i move that back to local drives? i moved it to the SAN a while back since we were having issues of hosts locking up at one point and when i rebooted the logs were cleared.

I had to remove our DVD drives a while back since they were the TEAC drives causing issues: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1017297.

If i need to install virgin can i do so from a USB? I will need to remember how to do a host profile since we have a ton of NICS and vswitches.

we may be in for a long haul. :)
0
 
LVL 121
ID: 40026013
No, keep the logs on the SAN!

Yes, you can install from USB, a new installation.

You can backup the ESXi configuratiion, and re-install to a virgin installation!
0
Free NetCrunch network monitor licenses!

Only on Experts-Exchange: Sign-up for a free-trial and we'll send you your permanent license!

Here is what you get: 30 Nodes | Unlimited Sensors | No Time Restrictions | Absolutely FREE!

Act now. This offer ends July 14, 2017.

 
LVL 2

Author Comment

by:msidnam
ID: 40026520
Thank you for the info. strange things are happening. I rebotted the server and it told me i had multibit ecc errors on the RAID controller memory. when it rebooted i didnt get the bootbank errors and the ALT-F12 kernel logs seemed to have calmed down.[newlog
then i rebooted again and no raid controller errors or bootbank errors. this is very strange. im going to try and reinstall asap but hopefully this gives me a little time.
0
 
LVL 121
ID: 40026687
Check your storage comntroller?
0
 
LVL 2

Author Comment

by:msidnam
ID: 40027021
it looks like i need new memory or for the RAID controller or a new controller. I'm thinking this is the reason for all the craziness
0
 
LVL 2

Author Comment

by:msidnam
ID: 40034422
A quick update in case anyone else has a similar issue. I needed to update my RAID controller firmware. I found this post: https://communities.vmware.com/message/2249566?z=PNeQaQ and updated the firmware.
0

Featured Post

Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this article, I will show you HOW TO: Create your first Windows Virtual Machine on a VMware vSphere Hypervisor 6.5 (ESXi 6.5) Host Server, the Windows OS we will install is Windows Server 2016.
Ransomware is a malware that is again in the list of security  concerns. Not only for companies, but also for Government security and  even at personal use. IT departments should be aware and have the right  knowledge to how to fight it.
Advanced tutorial on how to run the esxtop command to capture a batch file in csv format in order to export the file and use it for performance analysis. He demonstrates how to download the file using a vSphere web client (or vSphere client) and exp…
How to install and configure Citrix XenApp 6.5 - Part 1. In this video tutorial we have explained step by step installation of Citrix XenApp 6.5 Server on Windows Server 2008 R2 is explained in this video. We have explained the difference between…

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question