Solved

After upgrading to 5.1U2 from 5.0 I am getting bootbank errors and slow ssh connections

Posted on 2014-04-27
7
3,537 Views
Last Modified: 2014-05-01
We upgraded our vcenter and esxi hosts yesterday to 5.1U2 from 5.0 U1.

evertyhing looked to go smooth but this morning while trying to upgrade the vmware tools on a VM the it gave out these warning:

Unable to install VMware Tools. An error occurred
while trying to access image file "/usr/lib/vmware
 /isoimages/linux.iso" needed to install VMware
Tools: 0 (Input/output error). If your product
shipped with the VMware Tools package, reinstall
VMware ESX, then try again to install the VMware
Tools package in the virtual machine

Reseatching i found that it may be because the esxi host doesnt have the image. When i tried to SSH to the host, the commands were very slow. when i type it would freeze for about 5-6 minutes. i looked at the console and when i hit ALT-F11 i would get this:

Bootbank cannot be found at path '/bootbank'
(2014-04-27T14:26:57.583Z cpu5:70427)
 warning
 

When i hit ALT-F12 i get this:
When hitting ALT-F12
I noticed at least two of the hosts updated are exibiting this behavior. i do not notice it in the other 10 hosts. however i am home troubleshooting and dont have direct access to them. I used the offline bundle, update-from-esxi5.1-5.1_update02.zip, to update from 5.0 U1.

I do have a ticket to VMWare tech support but i was hoping someone could shed some light on this.

As a side note, the VM's seem to be working fine. Although it is a sunday and no heavy loads are on it right now.
0
Comment
Question by:msidnam
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
7 Comments
 
LVL 120

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 500 total points
ID: 40025871
Is your ESXi installation on flash, check through the /var/log/vmkernel.log

Check your Bootbank/ESXi installation OS media, is NOT FULL!

vdf or df at the console

We've noticed recently the ESXi installations, are getting larger or log more, and you really need to move the /var/log (scratch) to persistent storage.

if it's upgraded an existing installation, using inplace offline bundle, although this completes, it does use more storage because of additional drivers and code.

This is where upgrades get you by the short and curlies, and sometimes, a fresh new virgin installation is better, because it re-creates all new partitions and symlinks, rather than relying on older ESXi 5.0 partition tables which are smaller!
0
 
LVL 2

Author Comment

by:msidnam
ID: 40025884
The install is on the local datastore. i made a folder called 5.1U2 and used esxcli to update from 5.0U1 to 5.1U2. I am trying to view the vmkernel.log file but the SSH connection times out for several minutes and its making it difficult

I do have the scratch log going to a datastore on our SAN. Should i move that back to local drives? i moved it to the SAN a while back since we were having issues of hosts locking up at one point and when i rebooted the logs were cleared.

I had to remove our DVD drives a while back since they were the TEAC drives causing issues: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1017297.

If i need to install virgin can i do so from a USB? I will need to remember how to do a host profile since we have a ton of NICS and vswitches.

we may be in for a long haul. :)
0
 
LVL 120
ID: 40026013
No, keep the logs on the SAN!

Yes, you can install from USB, a new installation.

You can backup the ESXi configuratiion, and re-install to a virgin installation!
0
Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

 
LVL 2

Author Comment

by:msidnam
ID: 40026520
Thank you for the info. strange things are happening. I rebotted the server and it told me i had multibit ecc errors on the RAID controller memory. when it rebooted i didnt get the bootbank errors and the ALT-F12 kernel logs seemed to have calmed down.[newlog
then i rebooted again and no raid controller errors or bootbank errors. this is very strange. im going to try and reinstall asap but hopefully this gives me a little time.
0
 
LVL 120
ID: 40026687
Check your storage comntroller?
0
 
LVL 2

Author Comment

by:msidnam
ID: 40027021
it looks like i need new memory or for the RAID controller or a new controller. I'm thinking this is the reason for all the craziness
0
 
LVL 2

Author Comment

by:msidnam
ID: 40034422
A quick update in case anyone else has a similar issue. I needed to update my RAID controller firmware. I found this post: https://communities.vmware.com/message/2249566?z=PNeQaQ and updated the firmware.
0

Featured Post

Simplifying Server Workload Migrations

This use case outlines the migration challenges that organizations face and how the Acronis AnyData Engine supports physical-to-physical (P2P), physical-to-virtual (P2V), virtual to physical (V2P), and cross-virtual (V2V) migration scenarios to address these challenges.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

HOW TO: Connect to the VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere (HTML5 Web) Host Client 6.5, and perform a simple configuration task of adding a new VMFS 6 datastore.
In this article, I am going to show you how to simulate a multi-site Lab environment on a single Hyper-V host. I use this method successfully in my own lab to simulate three fully routed global AD Sites on a Windows 10 Hyper-V host.
Teach the user how to configure vSphere Replication and how to protect and recover VMs Open vSphere Web Client: Verify vsphere Replication is enabled: Enable vSphere Replication for a virtual machine: Verify replicated VM is created: Recover replica…
Teach the user how to use vSphere Update Manager to update the VMware Tools and virtual machine hardware version Open vSphere Client: Review manual processes for updating VMware Tools and virtual hardware versions: Create a new baseline group in vSp…

733 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question