Windows Server 2003 w/ Exchange 2003 SP2 drops network connection every WED and SAT b/w 3am and 7am EST

We have a virtualized (ESXi 4.1) Windows Server 2003 server running Exchange 2003 R2 that intermittently drops the network connection for a few minutes every Wednesday and Saturday between 3am EST and 7am EST. The network drops out between 5-15 minutes and happens 2 to 5 times during the 3am to 7am period.

We monitor this server remotely and our monitoring software detects that the server is down. In addition, we use an external spam filtering service and they report the email server is not accepting connections when the network drops.

The server also runs an older version of BES 4 with one or two BBS devices. Sometimes in the event log, the SRP connection to Blackberry will fail, indicating that the server is also having network connection problems during the specified times above. However, this event log warning only appears about 10% of the time, since the network doesn’t stay long.

Since this server is virtualized, I also checked our history for other servers running the VMWare Host, no other guest VMs are dropping during that time. The vSwitch that the Exchange server is running on is the same as the other other guest VMs on that host.

Troubleshooting Steps:
Checked Application and System Logs, nothing indicative of causing network connection loss. System Logs will have 0 entries before and after network drops. Application log will have logs, but nothing that indicates a network connection drop
Checked Network Adapter uptime, does not change.
Changed network adapter from VMWare Flexible to VMWare VMXNET3, no change
Increased Exchange logging, still not event logs that indicate
Checked task scheduler, no tasks on the server and no hidden tasks on the server
Exchange maintenance is set to run DAILY from 1am to 5am, I doubt this is the cause if the server only drops on WED and SAT.
Our backup agent runs at 11pm nightly and never lasts more than 2 hours, so it starts/stops prior to network dropping

I’m really frustrated by Windows Server 2003 lack of logging. If it was a Server 2008+, logging appears to be way better with system components.

Any suggestions as to how to troubleshoot the network dropping b/w 3am and 7am EST on WED and SAT?
Who is Participating?
piattndConnect With a Mentor Commented:
I would say start your exchange server up with only MS services running then (that's one of the first troubleshooting steps Microsoft Professional Support goes through).  This will help you identify whether the problem is with something in the Microsoft services or perhaps introduced through another third party service.

Does the VM Host Server have any logs referencing virtual NIC issues?
Have you checked to make sure that updates are not running during this times which is causing the server to restart after the completion of updates.
Do any other virtuals do this same thing?  What do the system logs show before and after the connectivity goes down?  Do you see any evidence of services going into a stopping state (before) and starting state (after)?
Has Powershell sent you back into the Stone Age?

If managing Active Directory using Windows Powershell® is making you feel like you stepped back in time, you are not alone.  For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why.

If you still have issues with no luck in the event viewer, try booting the server with all non Microsoft services disabled.  To do this:

start > run > msconfig

Click on the Services tab.  Click the box for "hide all Microsoft services".  Disable all of these services EXCEPT potentially critical services (critical meaning exchange or the operating system won't function without it).  Run the server like this for a while and see if the issue happens again.  If it does, it's definitely a Microsoft based program/feature of the operating system causing the problem, but I think vmagan may be on to something with the reboot.  It may not be Automatic Updates, but it seems odd connectivity would drop the same time on those days repeatedly and not have a scheduled task or something like that causing it.
brainsurf1Author Commented:
@piattnd - No other virtual machine on the same host is dropping connections, there are a few other guest VMs on that host that we monitor and never receive notifications. Monitoring thresholds are not different between the guest VMs.

The System Event Log for the server shows the oldest shadow copy being deleted at 1:10am EST and then the next event log is at 12:46pm about system uptime.

There are no event logs for services stopping at that time in the SYSTEM event log.

In the APPLICATION logs, I am seeing that there is an ActiveSync error right before the event happens. However, this same error message appears frequently throughout the day and the server does not drop in the middle of they.
Unexpected Exchange mailbox Server error: Server: [] User: [] HTTP status code: [409]. Verify that the Exchange mailbox Server is working correctly. 

Open in new window

brainsurf1Author Commented:
@vmagan - we have windows updates turned off on this server and use a RMM tool to push out updates to the server. This tool runs daily to install updates that haven't been installed. We checked our scheduled tasks for this server in our RMM tool and do not see any tasks for WED and SAT.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.