Windows Server 2012 Exchange 2013 Powershell spawns new process every hour and consumes lots of CPU

This only appears to be happening on the Windows Server 2012 (non-R2) Exchange 2013 servers we manage.

Every hour a new Powershell instance starts and begins hanging the CPU. Over the course of time dozens of these Powershell.exe instances get created, and there is always one at the top consuming a ton of CPU. If I kill that process, another one comes to life and begins consuming a ton of CPU. The only way to get it to stop is to
taskkill /IM powershell.exe /F

Open in new window

and it kills all the Powershell instances.

But with time they will come creeping back. If I look in Process Explorer, each Powershell instance is hosted by conhost.exe (which might be normal)

Also in Process Explorer, I see that the command it is trying to run is
powershell.exe "Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAcount SilentlyContinue;get-spfarm | select Products;

Open in new window

2015-10-19_12-40-19.jpgWe use a monitoring tool called LabTech and I notice that this also happens when I deploy scripts to these machines via LabTech.

Powershell works fine when you run it at the console of the machine, so it appears that this is only happening when Powershell is executed in a non-interactive mode.
Darren KattanIT ConsultantAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Neil RussellTechnical Development LeadCommented:
The powershell command "get-spfarm | select products" is for a sharepoint server.  Do have SharePoint installed on the same machine as exchange?  Bad Bad move if you have!
Jeremy WeisingerSenior Network Consultant / EngineerCommented:
Something is running those commands. Take a look at the task scheduler,... Does the labtech software have tasks it runs using powershell? Either of those might reveal what is launching the scripts/commands and causing the issue.
Seth SimmonsSr. Systems AdministratorCommented:
looks like a powershell script is running as a scheduled task and not terminating properly so it continues with additional processes

i would locate that scheduled task and check to see if it is supposed to stop after x hours and/or change how often it runs
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

Neil RussellTechnical Development LeadCommented:
Regardless of what is starting and why, the more important thing is that that is a Powershell command for SharePoint servers and NOT exchange servers.  
You not only need to find whats starting it and stop it , more importantly you need to understand why a powershell command for Sharepoint is running on your exchange servers.  It is not natural and not good.
Darren KattanIT ConsultantAuthor Commented:
To answer the question, no this is not a SharePoint server, nor does it have SharePoint installed on it.

I did a ton of digging and it turns out that it is LabTech doing a discovery on the server every hour trying to determine if any roles have changed. But ultimately I think the fact that it is a SharePoint cmdlet is a red-herring, as any command I deploy to it via LabTech causes Powershell to hang and consume a ton of CPU. I've been able to disable the hourly scan so at least I don't have 24 new hung Powershell processes every day, and I've got a ticket open with LabTech and they are going to take a look at it today.

I'd close the question out and leave it LabTech support but at this point I'm not convinced that this isn't a deeper Windows problem that LabTech unintentionally uncovered as this is ONLY a problem with Server 2012 with Exchange 2013 installed. I've witnessed this problem on every server we manage with this combination of software, which puts us at about 6 total for completely separate clients. There is a hotfix available (KB2985459) that acknowledges a Powershell race condition stemming from Exchange Powershell cmdlets being executed against IIS on Server 2012/Exchange 2013. But, I installed it on 3 of the servers and it hasn't helped :-/

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Jeremy WeisingerSenior Network Consultant / EngineerCommented:
I think the Labtech software has issues. It shouldn't be running commands for products that are not installed. It must be misidentifying what is on the system. That is most likely the cause of the problem.
Darren KattanIT ConsultantAuthor Commented:
Jeremy, that begs the question, how else is it supposed to figure out what is on the system?

That command, when run in normal context, does not hang the system. It just errors out. LabTech interprets that (correctly) as to mean that Sharepoint is not installed.
Jeremy WeisingerSenior Network Consultant / EngineerCommented:
I'm sorry but that is a terrible way to determine what is installed and I doubt they intended to do it that way. You can simply enumerate the software installed on the system and then run appropriate commands to gather information.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2012

From novice to tech pro — start learning today.