Solved

SBS2011 connection lost every day at 12 am

Posted on 2014-01-30
12
297 Views
Last Modified: 2014-07-30
Since a few weeks there’s a big issue in the network. There a fysical server (HP ML380gen8) running ESXi 5.0.0 running on it. On this ESXi server a few virtual servers are running. One of them is a SBS2011 server with 18Gb Ram,  2 virtual sockets and 300Gb disk.

Every day on 12:00am. The cpu loads get high to 99% , after that the connection (also ping) is lost for about 2 minutes. I checked every possible task that may be running at that time and checked/did the following the last few weeks on the SBS2011:

-      de-installed antivirus
-      disabled shadow copies
-      disabled backupexec agent
-      checked every possible scheduled task

Nothing works. The strange thing is that this happens every day on 12:00 am.  Other thing is that the system eventlog shows that at 12:00 “The Volume Shadow Copy service entered the running state.” And at 12:04: “The Volume Shadow Copy service entered the stopped state.”  Thats the same period the connection was lost. But at 12am no shadow copies where scheduled. And the issue also came when shadown copies was completly disabled.

See attach performance graphics.
1.png
2.png
0
Comment
Question by:sitpro
  • 5
  • 4
  • 2
  • +1
12 Comments
 
LVL 22

Expert Comment

by:David Atkin
ID: 39820911
Hello,

Have you installed any new printers onto the last few weeks?

I ask because the printisolationhost.exe process is running quite high.  it wouldn't explain why it crashes at 12AM every day though.

Any issue on the other VMs?
0
 

Author Comment

by:sitpro
ID: 39821189
no new printers where installed, other vm's work fine. But as you see, the free fysical memory is 0, but SBS always take much ram isnt it?
0
 
LVL 22

Expert Comment

by:David Atkin
ID: 39821237
SBS is a resource hog for RAM yes.  SQL and Exchange will use as much as it.  You can set limits on these.

If you left a constant ping from the server do you get replied from the PCs?  I.e. is all communication stopped or just the inbound communication?

Any other errors in the Event logs around and before this time?
0
 
LVL 8

Accepted Solution

by:
Ratnesh Mishra earned 500 total points
ID: 39821580
Don't worry ,its not network issue.
The server gets into soft hang situation for this 2 min.

 In order to omit printer or print driver issue , please disable print spooler service .

You have already uninstalled the anti -virus , you have disabled backup exec , and also checked there is no scheduled task running at that time.

Its a simple case of High CPU utilization , if possible can you take xperf  log for knowing which process is causing this. You may follow the article ,its very concise and to the point.

http://blogs.technet.com/b/sooraj-sec/archive/2011/09/14/collecting-data-using-xperf-for-high-cpu-utilization-of-a-process.aspx

Details on how to use xperf
http://blogs.msdn.com/b/debuggingtoolbox/archive/2010/03/15/xperf-tool-why-can-t-you-live-without-it.aspx
0
 

Author Comment

by:sitpro
ID: 39822119
@Ratnesh Mishra
tomorrow i will start the highcpu measurment

i also think its a soft hang caused by a high load. And i'm very curious wich process is caused this issue.



@David Atkin
also check at the console if all communication is gone.
0
 
LVL 12

Expert Comment

by:ktaczala
ID: 39822216
Do you have branch office's  connected over VPN?

I had one customer that had a similar issue.  Taking the branches out of WSUS fixed it.
0
Are your corporate email signatures appalling?

Is it scary how unprofessional your email signatures look? Do users create their own terrible designs and give themselves stupid job titles? You can make this a lot easier for yourself by choosing an email signature management solution from Exclaimer today.

 
LVL 8

Expert Comment

by:Ratnesh Mishra
ID: 39822264
Xperf log analysis will certainly provide us the clue , however would like to let you know donot run xperf for longer than 5 min as its consumes more disk space. you have to run it just before you start experiencing the issue.
Apart from this you also may use Process Explorer which process is consuming this much of resources, if it shows you svchost do note down the process id.
0
 

Author Comment

by:sitpro
ID: 39822299
@ktaczala
No VPN users are active on that server

@Ratnesh
I will start the trace just before 12 for a few minutes. Also check process explorer.
0
 

Author Comment

by:sitpro
ID: 39824837
Ratnesh,


see attached files for the highcpu result. I cannot judge this results.
1.png
2.png
3.png
4.png
0
 
LVL 8

Expert Comment

by:Ratnesh Mishra
ID: 39829410
Please follow the below mentioned article to segregate the services causing the svchost container to spike the CPU.

http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Windows/A_12862-SVCHOST-EXE-CONSUMING-HIGH-CPU-MEMORY.html

However based  on  my experience it seems that some configuration regarding event forwarding or any WMI query is executed on the machine to cause it to go in soft hang situation. If possible can you stop COM+ Event system services for the specific time frame or little bit earlier to verify the issue and also check the status of service after the time elapse.
0
 

Author Comment

by:sitpro
ID: 39978442
i'm still searching for a solution using xperf
0
 
LVL 8

Expert Comment

by:Ratnesh Mishra
ID: 40009408
Were you able to collect HighCPU Xperf logs for analysis.
If not , I would suggest you to collect procmon log for the entire duration of issue occurrence.
That will also help us in understanding what might be causing this issue.

One most important thing which I might have missed. Is there anything configured on Host level which may caused to stop the VM for few minutes to take snapshot or backup and there after resumes it.
As in Hyper-V case in order to get VSS aware consistent snapshot ,Hyper-V pause the VM ,takes the snapshot and resumes it back.
0

Featured Post

Want to promote your upcoming event?

Are you going to an event? Are you going to be exhibiting at a tradeshow? Talking at a conference? Using a promotional banner in your email signature ensures that your organization’s most important contacts stay in the know and can potentially spread the word about the event.

Join & Write a Comment

My previous article  (http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/Windows_Server_2008/A_4466-A-beginners-guide-to-installing-SCCM2007-on-Windows-2008-R2-Server.html)detailed one possible method to get SCCM 2007 installed an…
INTRODUCTION The purpose of this document is to demonstrate the Installation and configuration of the Data Protection Manager product. Note that this demonstration was prepared on the basis of Windows OS is 2008 R2 and DPM 2010. DATA PROTECTI…
In this video, we discuss why the need for additional vertical screen space has become more important in recent years, namely, due to the transition in the marketplace of 4x3 computer screens to 16x9 and 16x10 screens (so-called widescreen format). …
With the advent of Windows 10, Microsoft is pushing a Get Windows 10 icon into the notification area (system tray) of qualifying computers. There are many reasons for wanting to remove this icon. This two-part Experts Exchange video Micro Tutorial s…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now