Solved

Windows Server 2003 w/ Exchange 2003 SP2 drops network connection every WED and SAT b/w 3am and 7am EST

Posted on 2013-12-12
6
435 Views
Last Modified: 2014-01-08
We have a virtualized (ESXi 4.1) Windows Server 2003 server running Exchange 2003 R2 that intermittently drops the network connection for a few minutes every Wednesday and Saturday between 3am EST and 7am EST. The network drops out between 5-15 minutes and happens 2 to 5 times during the 3am to 7am period.

We monitor this server remotely and our monitoring software detects that the server is down. In addition, we use an external spam filtering service and they report the email server is not accepting connections when the network drops.

The server also runs an older version of BES 4 with one or two BBS devices. Sometimes in the event log, the SRP connection to Blackberry will fail, indicating that the server is also having network connection problems during the specified times above. However, this event log warning only appears about 10% of the time, since the network doesn’t stay long.

Since this server is virtualized, I also checked our history for other servers running the VMWare Host, no other guest VMs are dropping during that time. The vSwitch that the Exchange server is running on is the same as the other other guest VMs on that host.

Troubleshooting Steps:
Checked Application and System Logs, nothing indicative of causing network connection loss. System Logs will have 0 entries before and after network drops. Application log will have logs, but nothing that indicates a network connection drop
Checked Network Adapter uptime, does not change.
Changed network adapter from VMWare Flexible to VMWare VMXNET3, no change
Increased Exchange logging, still not event logs that indicate
Checked task scheduler, no tasks on the server and no hidden tasks on the server
Exchange maintenance is set to run DAILY from 1am to 5am, I doubt this is the cause if the server only drops on WED and SAT.
Our backup agent runs at 11pm nightly and never lasts more than 2 hours, so it starts/stops prior to network dropping

I’m really frustrated by Windows Server 2003 lack of logging. If it was a Server 2008+, logging appears to be way better with system components.

Any suggestions as to how to troubleshoot the network dropping b/w 3am and 7am EST on WED and SAT?
0
Comment
Question by:brainsurf1
  • 3
  • 2
6 Comments
 
LVL 6

Expert Comment

by:vmagan
ID: 39714320
Have you checked to make sure that updates are not running during this times which is causing the server to restart after the completion of updates.
0
 
LVL 12

Expert Comment

by:piattnd
ID: 39714330
Do any other virtuals do this same thing?  What do the system logs show before and after the connectivity goes down?  Do you see any evidence of services going into a stopping state (before) and starting state (after)?
0
 
LVL 12

Expert Comment

by:piattnd
ID: 39714339
If you still have issues with no luck in the event viewer, try booting the server with all non Microsoft services disabled.  To do this:

start > run > msconfig

Click on the Services tab.  Click the box for "hide all Microsoft services".  Disable all of these services EXCEPT potentially critical services (critical meaning exchange or the operating system won't function without it).  Run the server like this for a while and see if the issue happens again.  If it does, it's definitely a Microsoft based program/feature of the operating system causing the problem, but I think vmagan may be on to something with the reboot.  It may not be Automatic Updates, but it seems odd connectivity would drop the same time on those days repeatedly and not have a scheduled task or something like that causing it.
0
Problems using Powershell and Active Directory?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

 

Author Comment

by:brainsurf1
ID: 39714757
@piattnd - No other virtual machine on the same host is dropping connections, there are a few other guest VMs on that host that we monitor and never receive notifications. Monitoring thresholds are not different between the guest VMs.

The System Event Log for the server shows the oldest shadow copy being deleted at 1:10am EST and then the next event log is at 12:46pm about system uptime.

There are no event logs for services stopping at that time in the SYSTEM event log.


In the APPLICATION logs, I am seeing that there is an ActiveSync error right before the event happens. However, this same error message appears frequently throughout the day and the server does not drop in the middle of they.
Unexpected Exchange mailbox Server error: Server: [exchange.domain.com] User: [user@domain.com] HTTP status code: [409]. Verify that the Exchange mailbox Server is working correctly. 

Open in new window

0
 

Author Comment

by:brainsurf1
ID: 39714786
@vmagan - we have windows updates turned off on this server and use a RMM tool to push out updates to the server. This tool runs daily to install updates that haven't been installed. We checked our scheduled tasks for this server in our RMM tool and do not see any tasks for WED and SAT.
0
 
LVL 12

Accepted Solution

by:
piattnd earned 500 total points
ID: 39715060
I would say start your exchange server up with only MS services running then (that's one of the first troubleshooting steps Microsoft Professional Support goes through).  This will help you identify whether the problem is with something in the Microsoft services or perhaps introduced through another third party service.

Does the VM Host Server have any logs referencing virtual NIC issues?
0

Featured Post

NAS Cloud Backup Strategies

This article explains backup scenarios when using network storage. We review the so-called “3-2-1 strategy” and summarize the methods you can use to send NAS data to the cloud

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Distribution List Owner Permissions Lost 2 18
MS Endpoint Protection 2 19
Password recovery software 4 25
ACTIVE DIRECTORY 18 43
Restoring deleted objects in Active Directory has been a standard feature in Active Directory for many years, yet some admins may not know what is available.
Scam emails are a huge burden for many businesses. Spotting one is not always easy. Follow our tips to identify if an email you receive is a scam.
In this video we show how to create a User Mailbox in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Recipients >> Mailb…
This video discusses moving either the default database or any database to a new volume.

813 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now