VMware tools upgrade to 10.5 causing major outage ?

Hi People,

When performing the previous VMware tools upgrade, I usually don't expect to do any major outage at all, normally only 1-2 ping packet loss. But today it has caused my some big issue where the production terminal server patch network adapter lost (not appearing in Network adapter folder) ?

The VMware tools was upgraded from earlier version to the latest v10.5 by double clicking on the setup64.exe file and it was taking so long time stuck in VMXNet3 driver update. eventually it finished successfully with the below error logged:

Level:Log Name:      System
Source:        vmxnet3ndis6
Date:          8/01/2016 1:20:57 PM
Event ID:      5001
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      PRODTS47-VM.domain.com
vmxnet3 Ethernet Adapter #2 : Could not allocate the resources necessary for operation.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <Provider Name="vmxnet3ndis6" />
    <EventID Qualifiers="49152">5001</EventID>
    <TimeCreated SystemTime="2016-01-08T02:20:57.105730500Z" />
    <Security />
    <Data>vmxnet3 Ethernet Adapter #2</Data>

Therefore I have to perform reboot which cause some grief and frustration from the users !

Can anyone please assist me in troubleshooting what the root cause of the issue as the management and the change management team now demands some explanation from me ?

Please help me.
LVL 12
Senior IT System EngineerSenior Systems EngineerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
1. We always test any changes to Production VMs, this includes the upgrading of VMware Tools.

2. We always perform any changes to schedule Production VMs, out of core hours.

3. We always ensure we have full backups before we many any changes to production machines.

All of the above are agreed with our Change Boards and Change Management Teams before we commence any work.

So did you test it ?

You did this change with users logged on ?

The error message looks like, running out of memory or resources, do you regularly reboot your terminal servers every 24 hours ?
Senior IT System EngineerSenior Systems EngineerAuthor Commented:
Hi Andrew,

No I don't regularly reboot it.
The only test that I did was the other IT Management server which doesn't cause any outage.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
We've always regularly rebooted our terminal servers after 24 hours, every day at 3AM, to clear all the sessions out, and clears out all the memory leaks and resource issues. Which seems to be with many users logging on and off, and running apps.
Your Guide to Achieving IT Business Success

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

Senior IT System EngineerSenior Systems EngineerAuthor Commented:
Hi Andrew,

Yes that does make sense.

But my case here is different. I got 55 VMs that server as Win2008R2 Terminal Servers RDSH for multiple different office in across several continents, so it is 24/7 used by the users.

The load balancer here is Riverbed Steel App

Restarting every 24 hours randomly is just going to cause some problem when there is some users logged on.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Well you need to plan it then!

We have a farm, and take one out, then another restarts etc.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
But you have to install windows updates.
We schedule vm tools update at host reboot that happens sunday after patch tuesday.
Senior IT System EngineerSenior Systems EngineerAuthor Commented:
Thanks !
Senior IT System EngineerSenior Systems EngineerAuthor Commented:
Thanks all, I guess it is a bug or something that is unfortunate happens in my environment.

This is the first time i encounter this problem in my life as VMware admin.
There are some bad seeds in smooth update promise with mayor vmtools update.
Out of ~1000 windows VMs we got to reinstall vmware tools in about 20 when moving from ESXi 4.1 to 5.1
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.