VMware tools upgrade to 10.5 causing major outage ?

Senior IT System Engineer
Senior IT System Engineer used Ask the Experts™
on
Hi People,

When performing the previous VMware tools upgrade, I usually don't expect to do any major outage at all, normally only 1-2 ping packet loss. But today it has caused my some big issue where the production terminal server patch network adapter lost (not appearing in Network adapter folder) ?

The VMware tools was upgraded from earlier version to the latest v10.5 by double clicking on the setup64.exe file and it was taking so long time stuck in VMXNet3 driver update. eventually it finished successfully with the below error logged:

Level:Log Name:      System
Source:        vmxnet3ndis6
Date:          8/01/2016 1:20:57 PM
Event ID:      5001
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      PRODTS47-VM.domain.com
Description:
vmxnet3 Ethernet Adapter #2 : Could not allocate the resources necessary for operation.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="vmxnet3ndis6" />
    <EventID Qualifiers="49152">5001</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2016-01-08T02:20:57.105730500Z" />
    <EventRecordID>139469</EventRecordID>
    <Channel>System</Channel>
    <Computer>PRODTS47-VM.domain.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>\Device\NDMP14</Data>
    <Data>vmxnet3 Ethernet Adapter #2</Data>
    <Binary>00001C0002004C0000000000891300C00000000000000000000000000000000000000000000000009A0000C0436D23007265637671756575652E6300CD00000006000000</Binary>
  </EventData>
</Event>

Therefore I have to perform reboot which cause some grief and frustration from the users !

Can anyone please assist me in troubleshooting what the root cause of the issue as the management and the change management team now demands some explanation from me ?

Please help me.
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017
Commented:
1. We always test any changes to Production VMs, this includes the upgrading of VMware Tools.

2. We always perform any changes to schedule Production VMs, out of core hours.

3. We always ensure we have full backups before we many any changes to production machines.

All of the above are agreed with our Change Boards and Change Management Teams before we commence any work.

So did you test it ?

You did this change with users logged on ?

The error message looks like, running out of memory or resources, do you regularly reboot your terminal servers every 24 hours ?

Author

Commented:
Hi Andrew,

No I don't regularly reboot it.
The only test that I did was the other IT Management server which doesn't cause any outage.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017
Commented:
We've always regularly rebooted our terminal servers after 24 hours, every day at 3AM, to clear all the sessions out, and clears out all the memory leaks and resource issues. Which seems to be with many users logging on and off, and running apps.
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Author

Commented:
Hi Andrew,

Yes that does make sense.

But my case here is different. I got 55 VMs that server as Win2008R2 Terminal Servers RDSH for multiple different office in across several continents, so it is 24/7 used by the users.

The load balancer here is Riverbed Steel App

Restarting every 24 hours randomly is just going to cause some problem when there is some users logged on.
VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017
Commented:
Well you need to plan it then!

We have a farm, and take one out, then another restarts etc.
Top Expert 2015
Commented:
But you have to install windows updates.
We schedule vm tools update at host reboot that happens sunday after patch tuesday.

Author

Commented:
Thanks !

Author

Commented:
Thanks all, I guess it is a bug or something that is unfortunate happens in my environment.

This is the first time i encounter this problem in my life as VMware admin.
Top Expert 2015

Commented:
There are some bad seeds in smooth update promise with mayor vmtools update.
Out of ~1000 windows VMs we got to reinstall vmware tools in about 20 when moving from ESXi 4.1 to 5.1

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial