Solved

Windows Server 2003 w/ Exchange 2003 SP2 drops network connection every WED and SAT b/w 3am and 7am EST

Posted on 2013-12-12
6
432 Views
Last Modified: 2014-01-08
We have a virtualized (ESXi 4.1) Windows Server 2003 server running Exchange 2003 R2 that intermittently drops the network connection for a few minutes every Wednesday and Saturday between 3am EST and 7am EST. The network drops out between 5-15 minutes and happens 2 to 5 times during the 3am to 7am period.

We monitor this server remotely and our monitoring software detects that the server is down. In addition, we use an external spam filtering service and they report the email server is not accepting connections when the network drops.

The server also runs an older version of BES 4 with one or two BBS devices. Sometimes in the event log, the SRP connection to Blackberry will fail, indicating that the server is also having network connection problems during the specified times above. However, this event log warning only appears about 10% of the time, since the network doesn’t stay long.

Since this server is virtualized, I also checked our history for other servers running the VMWare Host, no other guest VMs are dropping during that time. The vSwitch that the Exchange server is running on is the same as the other other guest VMs on that host.

Troubleshooting Steps:
Checked Application and System Logs, nothing indicative of causing network connection loss. System Logs will have 0 entries before and after network drops. Application log will have logs, but nothing that indicates a network connection drop
Checked Network Adapter uptime, does not change.
Changed network adapter from VMWare Flexible to VMWare VMXNET3, no change
Increased Exchange logging, still not event logs that indicate
Checked task scheduler, no tasks on the server and no hidden tasks on the server
Exchange maintenance is set to run DAILY from 1am to 5am, I doubt this is the cause if the server only drops on WED and SAT.
Our backup agent runs at 11pm nightly and never lasts more than 2 hours, so it starts/stops prior to network dropping

I’m really frustrated by Windows Server 2003 lack of logging. If it was a Server 2008+, logging appears to be way better with system components.

Any suggestions as to how to troubleshoot the network dropping b/w 3am and 7am EST on WED and SAT?
0
Comment
Question by:brainsurf1
  • 3
  • 2
6 Comments
 
LVL 6

Expert Comment

by:vmagan
ID: 39714320
Have you checked to make sure that updates are not running during this times which is causing the server to restart after the completion of updates.
0
 
LVL 12

Expert Comment

by:piattnd
ID: 39714330
Do any other virtuals do this same thing?  What do the system logs show before and after the connectivity goes down?  Do you see any evidence of services going into a stopping state (before) and starting state (after)?
0
 
LVL 12

Expert Comment

by:piattnd
ID: 39714339
If you still have issues with no luck in the event viewer, try booting the server with all non Microsoft services disabled.  To do this:

start > run > msconfig

Click on the Services tab.  Click the box for "hide all Microsoft services".  Disable all of these services EXCEPT potentially critical services (critical meaning exchange or the operating system won't function without it).  Run the server like this for a while and see if the issue happens again.  If it does, it's definitely a Microsoft based program/feature of the operating system causing the problem, but I think vmagan may be on to something with the reboot.  It may not be Automatic Updates, but it seems odd connectivity would drop the same time on those days repeatedly and not have a scheduled task or something like that causing it.
0
Get up to 2TB FREE CLOUD per backup license!

An exclusive Black Friday offer just for Expert Exchange audience! Buy any of our top-rated backup solutions & get up to 2TB free cloud per system! Perform local & cloud backup in the same step, and restore instantly—anytime, anywhere. Grab this deal now before it disappears!

 

Author Comment

by:brainsurf1
ID: 39714757
@piattnd - No other virtual machine on the same host is dropping connections, there are a few other guest VMs on that host that we monitor and never receive notifications. Monitoring thresholds are not different between the guest VMs.

The System Event Log for the server shows the oldest shadow copy being deleted at 1:10am EST and then the next event log is at 12:46pm about system uptime.

There are no event logs for services stopping at that time in the SYSTEM event log.


In the APPLICATION logs, I am seeing that there is an ActiveSync error right before the event happens. However, this same error message appears frequently throughout the day and the server does not drop in the middle of they.
Unexpected Exchange mailbox Server error: Server: [exchange.domain.com] User: [user@domain.com] HTTP status code: [409]. Verify that the Exchange mailbox Server is working correctly. 

Open in new window

0
 

Author Comment

by:brainsurf1
ID: 39714786
@vmagan - we have windows updates turned off on this server and use a RMM tool to push out updates to the server. This tool runs daily to install updates that haven't been installed. We checked our scheduled tasks for this server in our RMM tool and do not see any tasks for WED and SAT.
0
 
LVL 12

Accepted Solution

by:
piattnd earned 500 total points
ID: 39715060
I would say start your exchange server up with only MS services running then (that's one of the first troubleshooting steps Microsoft Professional Support goes through).  This will help you identify whether the problem is with something in the Microsoft services or perhaps introduced through another third party service.

Does the VM Host Server have any logs referencing virtual NIC issues?
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Resolve DNS query failed errors for Exchange
Following basic email etiquette rules will help you write a professional email and achieve a good, lasting impression with your contacts.
In this video we show how to create a Shared Mailbox in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Recipients >> Sha…
In this video we show how to create an Accepted Domain in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Mail Flow >> Ac…

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now