Windows Server 2008 r2 becoming unstable requiring daily reboot

Also noticed there are ton of PowerShell processes running literally hundreds wonder if this may be a cause.  Not sure where these are coming from possible back end GPO maybe... 

Some additional text

Here is some additional info

In context: 2022-01-11 19:36:36.957 2022-01-11 14:36:30 3 2004 EVENTLOG_WARNING_TYPE #288572 Microsoft-Windows-Resource-Exhaustion-Detector Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: sqlservr.exe (2176) consumed 1520644096 bytes, LMIGuardianSvc.exe (88432) consumed 1088512000 bytes, and AgentService.exe (604) consumed 442245120 bytes. 2022-01-11 19:41:38.354 2022-01-11 14:41:34  3 2004 EVENTLOG_WARNING_TYPE #288573 Microsoft-Windows-Resource-Exhaustion-Detector Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: sqlservr.exe (2176) consumed 1520640000 bytes, LMIGuardianSvc.exe (88432) consumed 1088512000 bytes, and AgentService.exe (604) consumed 442245120 bytes. 2022-01-11 19:46:44.762 2022-01-11 14:46:38  3 2004 EVENTLOG_WARNING_TYPE #288574 Microsoft-Windows-Resource-Exhaustion-Detector Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: sqlservr.exe (2176) consumed 1520640000 bytes, LMIGuardianSvc.exe (88432) consumed 1088536576 bytes, and AgentService.exe (604) consumed 442118144 bytes. 2022-01-11 19:52:01.222 2022-01-11 14:51:54  3 2004 EVENTLOG_WARNING_TYPE #288575 Microsoft-Windows-Resource-Exhaustion-Detector Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: sqlservr.exe (2176) consumed 1520644096 bytes, LMIGuardianSvc.exe (88432) consumed 1088512000 bytes, and AgentService.exe (604) consumed 442105856 bytes. 2022-01-11 19:57:02.622 2022-01-11 14:56:58  3 2004 EVENTLOG_WARNING_TYPE #288576 Microsoft-Windows-Resource-Exhaustion-Detector Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: sqlservr.exe (2176) consumed 1520640000 bytes, LMIGuardianSvc.exe (88432) consumed 1088512000 bytes, and AgentService.exe (604) consumed 442736640 bytes. 2022-01-11 20:02:09.053 2022-01-11 15:02:02  3 2004 EVENTLOG_WARNING_TYPE #288577 Microsoft-Windows-Resource-Exhaustion-Detector Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: sqlservr.exe (2176) consumed 1520644096 bytes, LMIGuardianSvc.exe (88432) consumed 1088536576 bytes, and AgentService.exe (604) consumed 442245120 bytes. 2022-01-11 20:07:15.463 2022-01-11 15:07:08  3 2004 EVENTLOG_WARNING_TYPE #288578 Microsoft-Windows-Resource-Exhaustion-Detector Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: sqlservr.exe (2176) consumed 1520640000 bytes, LMIGuardianSvc.exe (88432) consumed 1088536576 bytes, and AgentService.exe (604) consumed 442245120 bytes. 2022-01-11 20:12:21.881 2022-01-11 15:12:14  3 2004 EVENTLOG_WARNING_TYPE #288579 Microsoft-Windows-Resource-Exhaustion-Detector Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: sqlservr.exe (2176) consumed 1520586752 bytes, LMIGuardianSvc.exe (88432) consumed 1088512000 bytes, and AgentService.exe (604) consumed 442138624 bytes. 2022-01-11 20:17:18.251 2022-01-11 15:17:14  3 2004 EVENTLOG_WARNING_TYPE #288580 Microsoft-Windows-Resource-Exhaustion-Detector Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: sqlservr.exe (2176) consumed 1520586752 bytes, LMIGuardianSvc.exe (88432) consumed 1088512000 bytes, and AgentService.exe (604) consumed 442224640 bytes. 2022-01-11 20:22:24.676 2022-01-11 15:22:20  3 2004 EVENTLOG_WARNING_TYPE #288581 Microsoft-Windows-Resource-Exhaustion-Detector Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: sqlservr.exe (2176) consumed 1520640000 bytes, LMIGuardianSvc.exe (88432) consumed 1088536576 bytes, and AgentService.exe (604) consumed 442224640 bytes. 2022-01-11 20:27:26.069 2022-01-11 15:27:20  3 2004 EVENTLOG_WARNING_TYPE #288582 Microsoft-Windows-Resource-Exhaustion-Detector Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: sqlservr.exe (2176) consumed 1520640000 bytes, LMIGuardianSvc.exe (88432) consumed 1088512000 bytes, and AgentService.exe (604) consumed 442843136 bytes.

Just found a ton of these running back end.

I am starting a MWB scan to see I am starting to think infection.  -_-

Good news is it found nothing related.

Conhost Svchost and powershell are still running multiple times on all of these servers I know we have SSRS in the background but not sure if that would be part of the cause.  

We have RMM running perhaps a policy may be related we have disabled and scheduled a reboot tomorrow.   It may be possible something GPO related may be the cause as well we are unsure.   Hope this is all enough info.  

Since 2008 R2 is off support it will only become more and more vulnerable to "evil nasties"...
You really should push for an upgrade budget...
Also try multiple scanners do double check for infections...
There have been several major attacks that have come out since 2008 EOL...

Hi Scott I did run a scan it only found our proprietary app executable nothing else I was thinking along the same lines that this may be a victim of an attack but nothing has shown proven yet as a culprit. 

have you checked task scheduler?
something could be configured to not check if another instance is running and launches another one and has an issue causing it to not terminate

i would change the columns in task manager to show the command line; this could help to see exactly what is running

Hi Seth yes task scheduler has been checked there was one possible cause.   We are going to use this server as a Guinee pig and use process of elimination to see if we can determine the cause.   We started with RMM we can do tasks next.   

Disabling RMM did not do the trick this may still be possibly something from GPO.  

System Idle Process.txt

Here is the system idle process on 1 of the servers rebooting only temp fixes it... It has to be something in GPO but where would or possibly what policy?  

Here is also a dump file of the processes.   conhost.dmp 

Can anyone recommend a good DMP file reader I am having trouble trying to look at these.  Or if someone is versed in dmp files can possibly tell me what the cause of these to repeat in memory.  

The power shell dump is almost 200 megs I cant upload it here.   

Conhost Debug

Microsoft (R) Windows Debugger Version 10.0.22473.1005 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.



Loading Dump File [C:\Users\andre\Downloads\conhost.dmp]
User Mini Dump File with Full Memory: Only application data is available

Symbol search path is: srv*
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (4 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Machine Name:
Debug session time: Fri Jan 14 10:49:55.000 2022 (UTC - 5:00)
System Uptime: 8 days 17:21:21.582
Process Uptime: 8 days 17:21:06.000
..............
For analysis of this file, run !analyze -v
ntdll!NtReplyWaitReceivePort+0xa:
00000000`777e985a c3 ret


Microsoft (R) Windows Debugger Version 10.0.22473.1005 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Users\andre\Downloads\conhost.dmp]
User Mini Dump File with Full Memory: Only application data is available

Symbol search path is: srv*
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (4 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Machine Name:
Debug session time: Fri Jan 14 10:49:55.000 2022 (UTC - 5:00)
System Uptime: 8 days 17:21:21.582
Process Uptime: 8 days 17:21:06.000
..............
For analysis of this file, run !analyze -v
ntdll!NtReplyWaitReceivePort+0xa:
00000000`777e985a c3 ret
NatVis script unloaded from 'C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2111.9001.0_neutral__8wekyb3d8bbwe\amd64\Visualizers\atlmfc.natvis'
NatVis script unloaded from 'C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2111.9001.0_neutral__8wekyb3d8bbwe\amd64\Visualizers\ObjectiveC.natvis'
NatVis script unloaded from 'C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2111.9001.0_neutral__8wekyb3d8bbwe\amd64\Visualizers\concurrency.natvis'
NatVis script unloaded from 'C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2111.9001.0_neutral__8wekyb3d8bbwe\amd64\Visualizers\cpp_rest.natvis'
NatVis script unloaded from 'C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2111.9001.0_neutral__8wekyb3d8bbwe\amd64\Visualizers\stl.natvis'
NatVis script unloaded from 'C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2111.9001.0_neutral__8wekyb3d8bbwe\amd64\Visualizers\Windows.Data.Json.natvis'
NatVis script unloaded from 'C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2111.9001.0_neutral__8wekyb3d8bbwe\amd64\Visualizers\Windows.Devices.Geolocation.natvis'
NatVis script unloaded from 'C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2111.9001.0_neutral__8wekyb3d8bbwe\amd64\Visualizers\Windows.Devices.Sensors.natvis'
NatVis script unloaded from 'C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2111.9001.0_neutral__8wekyb3d8bbwe\amd64\Visualizers\Windows.Media.natvis'
NatVis script unloaded from 'C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2111.9001.0_neutral__8wekyb3d8bbwe\amd64\Visualizers\windows.natvis'
NatVis script unloaded from 'C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2111.9001.0_neutral__8wekyb3d8bbwe\amd64\Visualizers\winrt.natvis'

Microsoft (R) Windows Debugger Version 10.0.22473.1005 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Users\andre\Downloads\powershell.dmp]
User Mini Dump File with Full Memory: Only application data is available


************* Path validation summary **************
Response Time (ms) Location
Deferred srv*
Symbol search path is: srv*
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (4 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Machine Name:
Debug session time: Fri Jan 14 10:50:15.000 2022 (UTC - 5:00)
System Uptime: 8 days 17:21:41.722
Process Uptime: 7 days 16:50:00.000
................................................................
...................
Loading unloaded module list
..
For analysis of this file, run !analyze -v
ntdll!ZwWaitForSingleObject+0xa:
00000000`777e97ea c3 ret


Powershell debug

Here are thread and stack debugs 

Doing some additional research it looks like all 98 of our servers are having this happen to them.   

Having looked at the properties of one of the powershell commands they are all running in LMI.   We have totally removed LMI from a server and rebooting it tomorrow will see what happens.  

Are you still using LogMeIn? Are they up to date? Or are they old versions you no longer are using?

@Scott Hi Scott I am assuming they may or may not be the latest version which prompted me to remove one on one of the servers and reboot to see if things stabilize in the morning.   It may be possible they were not the latest version. 

@Scott that was my thinking pattern as well, which is why I totally uninstalled it.   However it does not explain control or service host running multiple times unless LMI is triggering that. 

I've successfully located the issue with Powershell, it looks like the issue was being caused by LMI's background process.  I have logged into the server this morning that I removed LMI from and the server is stable and all powershell scripts are removed.   There are still multiple conhost and svchost processes running but the server is no where near low memory usage.  Thoughts?



  

Reinstalled LMI it happened again.   Can anyone verify what was the last supported version of LMI client that ran with 2008r2?  We may be stuck using that until we upgrade our servers.

 

I would contact Them on that issue... They may have better insight into what is needed. It could be a lmi downgrade is required based on other levels on your systems.
Are they fully up to date with the latest patches that were released?

@Scott Its 2008R2 I am not sure which patches they have as this is handled by INF.   I would assume they are it may be a version conflict at this standpoint.   I have LMI already notified and they are contacting development to see what can be done.   

We have located in one of the other processes, that Windows defender has been the culprit causing LMI to spread wildly and cause all the powershell.exe processes to happen.  

Shutting off windows defender has allowed LMI to run with no memory spikes and the servers are now stable.   Now the question is what is causing the 2 to conflict.   That will be for LMI to device.