Solved

SBS2011 connection lost every day at 12 am

Posted on 2014-01-30
12
305 Views
Last Modified: 2014-07-30
Since a few weeks there’s a big issue in the network. There a fysical server (HP ML380gen8) running ESXi 5.0.0 running on it. On this ESXi server a few virtual servers are running. One of them is a SBS2011 server with 18Gb Ram,  2 virtual sockets and 300Gb disk.

Every day on 12:00am. The cpu loads get high to 99% , after that the connection (also ping) is lost for about 2 minutes. I checked every possible task that may be running at that time and checked/did the following the last few weeks on the SBS2011:

-      de-installed antivirus
-      disabled shadow copies
-      disabled backupexec agent
-      checked every possible scheduled task

Nothing works. The strange thing is that this happens every day on 12:00 am.  Other thing is that the system eventlog shows that at 12:00 “The Volume Shadow Copy service entered the running state.” And at 12:04: “The Volume Shadow Copy service entered the stopped state.”  Thats the same period the connection was lost. But at 12am no shadow copies where scheduled. And the issue also came when shadown copies was completly disabled.

See attach performance graphics.
1.png
2.png
0
Comment
Question by:sitpro
  • 5
  • 4
  • 2
  • +1
12 Comments
 
LVL 22

Expert Comment

by:David Atkin
ID: 39820911
Hello,

Have you installed any new printers onto the last few weeks?

I ask because the printisolationhost.exe process is running quite high.  it wouldn't explain why it crashes at 12AM every day though.

Any issue on the other VMs?
0
 

Author Comment

by:sitpro
ID: 39821189
no new printers where installed, other vm's work fine. But as you see, the free fysical memory is 0, but SBS always take much ram isnt it?
0
 
LVL 22

Expert Comment

by:David Atkin
ID: 39821237
SBS is a resource hog for RAM yes.  SQL and Exchange will use as much as it.  You can set limits on these.

If you left a constant ping from the server do you get replied from the PCs?  I.e. is all communication stopped or just the inbound communication?

Any other errors in the Event logs around and before this time?
0
Use Case: Protecting a Hybrid Cloud Infrastructure

Microsoft Azure is rapidly becoming the norm in dynamic IT environments. This document describes the challenges that organizations face when protecting data in a hybrid cloud IT environment and presents a use case to demonstrate how Acronis Backup protects all data.

 
LVL 8

Accepted Solution

by:
Ratnesh Mishra earned 500 total points
ID: 39821580
Don't worry ,its not network issue.
The server gets into soft hang situation for this 2 min.

 In order to omit printer or print driver issue , please disable print spooler service .

You have already uninstalled the anti -virus , you have disabled backup exec , and also checked there is no scheduled task running at that time.

Its a simple case of High CPU utilization , if possible can you take xperf  log for knowing which process is causing this. You may follow the article ,its very concise and to the point.

http://blogs.technet.com/b/sooraj-sec/archive/2011/09/14/collecting-data-using-xperf-for-high-cpu-utilization-of-a-process.aspx

Details on how to use xperf
http://blogs.msdn.com/b/debuggingtoolbox/archive/2010/03/15/xperf-tool-why-can-t-you-live-without-it.aspx
0
 

Author Comment

by:sitpro
ID: 39822119
@Ratnesh Mishra
tomorrow i will start the highcpu measurment

i also think its a soft hang caused by a high load. And i'm very curious wich process is caused this issue.



@David Atkin
also check at the console if all communication is gone.
0
 
LVL 12

Expert Comment

by:ktaczala
ID: 39822216
Do you have branch office's  connected over VPN?

I had one customer that had a similar issue.  Taking the branches out of WSUS fixed it.
0
 
LVL 8

Expert Comment

by:Ratnesh Mishra
ID: 39822264
Xperf log analysis will certainly provide us the clue , however would like to let you know donot run xperf for longer than 5 min as its consumes more disk space. you have to run it just before you start experiencing the issue.
Apart from this you also may use Process Explorer which process is consuming this much of resources, if it shows you svchost do note down the process id.
0
 

Author Comment

by:sitpro
ID: 39822299
@ktaczala
No VPN users are active on that server

@Ratnesh
I will start the trace just before 12 for a few minutes. Also check process explorer.
0
 

Author Comment

by:sitpro
ID: 39824837
Ratnesh,


see attached files for the highcpu result. I cannot judge this results.
1.png
2.png
3.png
4.png
0
 
LVL 8

Expert Comment

by:Ratnesh Mishra
ID: 39829410
Please follow the below mentioned article to segregate the services causing the svchost container to spike the CPU.

http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Windows/A_12862-SVCHOST-EXE-CONSUMING-HIGH-CPU-MEMORY.html 

However based  on  my experience it seems that some configuration regarding event forwarding or any WMI query is executed on the machine to cause it to go in soft hang situation. If possible can you stop COM+ Event system services for the specific time frame or little bit earlier to verify the issue and also check the status of service after the time elapse.
0
 

Author Comment

by:sitpro
ID: 39978442
i'm still searching for a solution using xperf
0
 
LVL 8

Expert Comment

by:Ratnesh Mishra
ID: 40009408
Were you able to collect HighCPU Xperf logs for analysis.
If not , I would suggest you to collect procmon log for the entire duration of issue occurrence.
That will also help us in understanding what might be causing this issue.

One most important thing which I might have missed. Is there anything configured on Host level which may caused to stop the VM for few minutes to take snapshot or backup and there after resumes it.
As in Hyper-V case in order to get VSS aware consistent snapshot ,Hyper-V pause the VM ,takes the snapshot and resumes it back.
0

Featured Post

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The password reset disk is often mentioned as the best solution to deal with the lost Windows password problem. In Windows 2008, 7, Vista and XP, a password reset disk can be easily created. But besides Windows 7/Vista/XP, Windows Server 2008 and ot…
When you upgrade from Windows 8 to 8.1 or to Windows 10 or if you are like me you are on the Insider Program you may find yourself with many 450MB recovery partitions.  With a traditional disk that may not be a problem but with relatively smaller SS…
In this video, we discuss why the need for additional vertical screen space has become more important in recent years, namely, due to the transition in the marketplace of 4x3 computer screens to 16x9 and 16x10 screens (so-called widescreen format). …
Windows 8 came with a dramatically different user interface known as Metro. Notably missing from that interface was a Start button and Start Menu. Microsoft responded to negative user feedback of the Metro interface, bringing back the Start button a…

809 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question