Windows Server 2003 w/ Exchange 2003 SP2 drops network connection every WED and SAT b/w 3am and 7am EST

Posted on 2013-12-12
Medium Priority
Last Modified: 2014-01-08
We have a virtualized (ESXi 4.1) Windows Server 2003 server running Exchange 2003 R2 that intermittently drops the network connection for a few minutes every Wednesday and Saturday between 3am EST and 7am EST. The network drops out between 5-15 minutes and happens 2 to 5 times during the 3am to 7am period.

We monitor this server remotely and our monitoring software detects that the server is down. In addition, we use an external spam filtering service and they report the email server is not accepting connections when the network drops.

The server also runs an older version of BES 4 with one or two BBS devices. Sometimes in the event log, the SRP connection to Blackberry will fail, indicating that the server is also having network connection problems during the specified times above. However, this event log warning only appears about 10% of the time, since the network doesn’t stay long.

Since this server is virtualized, I also checked our history for other servers running the VMWare Host, no other guest VMs are dropping during that time. The vSwitch that the Exchange server is running on is the same as the other other guest VMs on that host.

Troubleshooting Steps:
Checked Application and System Logs, nothing indicative of causing network connection loss. System Logs will have 0 entries before and after network drops. Application log will have logs, but nothing that indicates a network connection drop
Checked Network Adapter uptime, does not change.
Changed network adapter from VMWare Flexible to VMWare VMXNET3, no change
Increased Exchange logging, still not event logs that indicate
Checked task scheduler, no tasks on the server and no hidden tasks on the server
Exchange maintenance is set to run DAILY from 1am to 5am, I doubt this is the cause if the server only drops on WED and SAT.
Our backup agent runs at 11pm nightly and never lasts more than 2 hours, so it starts/stops prior to network dropping

I’m really frustrated by Windows Server 2003 lack of logging. If it was a Server 2008+, logging appears to be way better with system components.

Any suggestions as to how to troubleshoot the network dropping b/w 3am and 7am EST on WED and SAT?
Question by:brainsurf1
  • 3
  • 2

Expert Comment

ID: 39714320
Have you checked to make sure that updates are not running during this times which is causing the server to restart after the completion of updates.
LVL 12

Expert Comment

ID: 39714330
Do any other virtuals do this same thing?  What do the system logs show before and after the connectivity goes down?  Do you see any evidence of services going into a stopping state (before) and starting state (after)?
LVL 12

Expert Comment

ID: 39714339
If you still have issues with no luck in the event viewer, try booting the server with all non Microsoft services disabled.  To do this:

start > run > msconfig

Click on the Services tab.  Click the box for "hide all Microsoft services".  Disable all of these services EXCEPT potentially critical services (critical meaning exchange or the operating system won't function without it).  Run the server like this for a while and see if the issue happens again.  If it does, it's definitely a Microsoft based program/feature of the operating system causing the problem, but I think vmagan may be on to something with the reboot.  It may not be Automatic Updates, but it seems odd connectivity would drop the same time on those days repeatedly and not have a scheduled task or something like that causing it.
Easily manage email signatures in Office 365

Managing email signatures in Office 365 can be a challenging task if you don't have the right tool. CodeTwo Email Signatures for Office 365 will help you implement a unified email signature look, no matter what email client is used by users. Test it for free!


Author Comment

ID: 39714757
@piattnd - No other virtual machine on the same host is dropping connections, there are a few other guest VMs on that host that we monitor and never receive notifications. Monitoring thresholds are not different between the guest VMs.

The System Event Log for the server shows the oldest shadow copy being deleted at 1:10am EST and then the next event log is at 12:46pm about system uptime.

There are no event logs for services stopping at that time in the SYSTEM event log.

In the APPLICATION logs, I am seeing that there is an ActiveSync error right before the event happens. However, this same error message appears frequently throughout the day and the server does not drop in the middle of they.
Unexpected Exchange mailbox Server error: Server: [exchange.domain.com] User: [user@domain.com] HTTP status code: [409]. Verify that the Exchange mailbox Server is working correctly. 

Open in new window


Author Comment

ID: 39714786
@vmagan - we have windows updates turned off on this server and use a RMM tool to push out updates to the server. This tool runs daily to install updates that haven't been installed. We checked our scheduled tasks for this server in our RMM tool and do not see any tasks for WED and SAT.
LVL 12

Accepted Solution

piattnd earned 1500 total points
ID: 39715060
I would say start your exchange server up with only MS services running then (that's one of the first troubleshooting steps Microsoft Professional Support goes through).  This will help you identify whether the problem is with something in the Microsoft services or perhaps introduced through another third party service.

Does the VM Host Server have any logs referencing virtual NIC issues?

Featured Post

Easily manage email signatures in Office 365

Managing email signatures in Office 365 can be a challenging task if you don't have the right tool. CodeTwo Email Signatures for Office 365 will help you implement a unified email signature look, no matter what email client is used by users. Test it for free!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

You finally migrated Public Folders to Office 365, decommissioned the Public Folder mailbox database and since then, when you send an email from on-premise to mail-enabled Public Folders, you get the following error: "Misconfigured public folder mai…
What is the biggest problem in managing an exchange environment today? It is the lack of backups, disaster recovery (DR) plan, testing of the DR plan or believing that it won’t happen to us.
There are cases when e.g. an IT administrator wants to have full access and view into selected mailboxes on Exchange server, directly from his own email account in Outlook or Outlook Web Access. This proves useful when for example administrator want…
Exchange organizations may use the Journaling Agent of the Transport Service to archive messages going through Exchange. However, if the Transport Service is integrated with some email content management application (such as an anti-spam), the admin…
Suggested Courses
Course of the Month5 days, 3 hours left to enroll

601 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question