Link to home
Start Free TrialLog in
Avatar of TheSonicGod
TheSonicGod

asked on

Exchange 2010 Server keeps locking up

Hi Everyone,

I am helping out a frend who is a network admin as he seems to be having some major issues with his exchange server.

It is a Exchange 2010 running on a windows 2008 x64 OS.

He has to reboot the unit at least once a day as mail flow just seems to stop.

I reviewed the windows logs and ran BPA utility and found a number of issues. Because he advised to me that he has been promoting 2008 servers as DC's on the network and demoting 2003 DC's from the network the following 2 errors from the BPA report peeked my interest:

1)     Unrecognized Exchange Signature - Active Directory domain 'ERBERB' has an unrecognized Exchange signature. Current DomainPrep version: 12639

2)      Microsoft Exchange System Attendent ‘homeMDB’ is missing - The 'homeMDB' value for the Microsoft Exchange System Attendant service on server ERBEXCHANGE is missing. This will cause reliability problems

In the windows System logs I also found a lot of errors including (in order from after reboot to required next reboot:

3) Event ID: 1500 - Source: SNMP - The SNMP Service encountered an error while accessing the registry key SYSTEM\CurrentControlSet\Services\SNMP\Parameters\TrapConfiguration. - logged: 7/30/2012 11:16:53 PM

4) Event ID: 1006 - Source: GroupPolicy - The processing of Group Policy failed. Windows could not authenticate to the Active Directory service on a domain controller. (LDAP Bind function call failed). Look in the details tab for error code and description. - logged: 7/31/2012 9:43:05 AM

Reboot completed - 11:00 am 7/31/2012

5) Event ID: 1 - Source: VDS Basic Provider - Unexpected failure. Error code: 490@01010004 - logged: 7/31/2012 8:01:12 PM

6) Event ID: 21 - Source: ARCFlashVolDrv - Failed to Write File \Device\HarddiskVolume3\CAVolTrc.dat Status : C00000A2. - logged: 7/31/2012 8:02:19 PM

7) Event ID: 137 - Source: Ntfs - The default transaction resource manager on volume \\?\Volume{00d75fb7-db21-11e1-b91e-0050569b0007} encountered a non-retryable error and could not start.  The data contains the error code. - logged: 7/31/2012 8:02:21 PM

8) Event ID: 21 - Source: ARCFlashVolDrv - Failed to Write File \Device\HarddiskVolume4\CAVolTrc.dat Status : C0000010. - logged: 7/31/2012 8:02:22 PM


In the Application logs I also found errors:

9) Event ID: 1906 - Source: MSExchangeApplicationLog - Service MSExchangeMailSubmission. Exchange topology discovery encountered an exception Microsoft.Exchange.Data.Directory.ADTransientException: Could not find any available Domain Controller.
   at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, ADObjectId domain, String serverName, Int32 port, NetworkCredential credential)
   at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType)
   at Microsoft.Exchange.Data.Directory.ADSession.GetConnection(String preferredServer, Boolean isWriteOperation, Boolean isNotifyOperation, String optionalBaseDN, ADObjectId& rootId, ADScope scope)
   at Microsoft.Exchange.Data.Directory.ADSession.GetReadConnection(String preferredServer, String optionalBaseDN, ADObjectId& rootId, ADRawEntry scopeDeteriminingObject)
   at Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId, String optionalBaseDN, ADObjectId readId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCreator, CreateObjectsDelegate arrayCreator, Boolean includeDeletedObjects)
   at Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCtor, CreateObjectsDelegate arrayCtor)
   at Microsoft.Exchange.Data.Directory.ADSession.Find[TResult](ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties)
   at Microsoft.Exchange.Data.Directory.SystemConfiguration.ADSystemConfigurationSession.Find[TResult](ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults)
   at Microsoft.Exchange.Data.Directory.SystemConfiguration.ADSystemConfigurationSession.FindServerByFqdn(String serverFqdn)
   at Microsoft.Exchange.Data.Directory.SystemConfiguration.ADSystemConfigurationSession.FindLocalServer()
   at Microsoft.Exchange.Data.ApplicationLogic.PickerServerList.LoadFromAD()
   at Microsoft.Exchange.Data.ApplicationLogic.ADConfigurationLoader`2.<>c__DisplayClass1.<ReadConfiguration>b__0()
   at Microsoft.Exchange.Data.Directory.ADNotificationAdapter.RunADOperation(ADOperation adOperation, Int32 retryCount)
   at Microsoft.Exchange.Data.Directory.ADNotificationAdapter.TryRunADOperation(ADOperation adOperation, Int32 retryCount) when trying to load topology information. - logged: 7/30/2012 11:13:31 PM

10) Event ID: 105 - Source: MSExchange Mailbox Replication - The Mailbox Replication service was unable to determine the list of mailbox databases hosted in the local Active Directory site. - logged: 7/30/2012 11:15:02 PM

11) Event ID: 1000 - Source: Application Error - Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: KERNELBASE.dll, version: 6.1.7601.17651, time stamp: 0x4e21213c
Exception code: 0xe0434f4d
Fault offset: 0x000000000000cacd
Faulting process id: 0x3b90
Faulting application start time: 0x01cd6e65aba21671
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: eb12c7df-dabd-11e1-9977-0050569b0007
Error: Could not find any available Domain Controller.  - logged: 7/30/2012 11:15:03 PM

12) Event ID: 2114 - Source: MSExchange ADAccess - Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=2644). Topology discovery failed, error 0x80040956 (LDAP_AUTH_UNKNOWN (An unknown authentication error occurred)). Look up the Lightweight Directory Access Protocol (LDAP) error code specified in the event description. To do this, use Microsoft Knowledge Base article 218185, "Microsoft LDAP Error Codes." Use the information in that article to learn more about the cause and resolution to this error. Use the Ping or PathPing command-line tools to test network connectivity to local domain controllers. - logged: 7/30/2012 11:15:14 PM

13) Event ID: 2917 - Source: MSExchange ADAccess - Process Microsoft.Exchange.Imap4.exe (PopImap) (PID=5656). A budget charge was encountered that exceeded the limit of '5.04973333333333' minutes.  Budget Owner: 'Sid~DOMAIN\itsupport~IMAP~False', Component: 'IMAP', CostType: 'CAS'. - logged: 7/30/2012 11:15:14 PM

14) Event ID: 2102 - Source: MSExchange ADAccess - Process MAD.EXE (PID=7380). All Domain Controller Servers in use are not responding:
SRV005.DOMAIN.local
SRVSQL1.DOMAIN.local
logged: 7/30/2012 11:15:16 PM

And about another 20 others - I can post if required.

To me it looks there is an issue with the AD configurtation or possible the ArcServ backup software but I am not sure where to go from here

Any suggestions would be helpful, thanks in advise for your responses,

TheSonicGod
Avatar of K-H
K-H
Flag of Germany image

It seems that there are more than one issues.
For getting rid of the NTFS errors try to run checkdisk with options "Automatically fix file system errors" and "Scan for and attemp recovery of bad sectors"
(This could take a while)

The Exchange theme looks like it is an DNS issue.
Has the Exchange the correct DNS Servers?
Is the DCs DNS server itself?
run dcdiag /fix
run ipconfig /flushdns on the exchange
run ipconfig /registerdns on both servers. (AD& Ex)
Take a client (or the Exchange) and ping the FQDN of the domain.

Regards
SOLUTION
Avatar of Exchange_Geek
Exchange_Geek
Flag of India image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of TheSonicGod
TheSonicGod

ASKER

Hi K-H,

here are the items I completed as requested:

1) Checked DNS servers for the network and both DC's are also DNS and set statically on the exchange server
2) I ran dcdiag /fix on SERVER005 and got the following error:
Starting test: NCSecDesc
   Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
      Replicating Directory Changes In Filtered Set
   access rights for the naming context:
   DC=DomainDnsZones,DC=DOMAIN,DC=local
   ......................... ERBERB failed test NCSecDesc  

No other error were reported except this - everything else passed

3) Ran ipconfig /flushdns on the exchange
4) Ran ipconfig /registerdns on both servers. (AD& Ex) - checked error logs for both - no errors related to DNS were found
5) Ping erberb.local and got reply back from one of the DC's IP address

Let me know if anything else is required.

thanks,

TheSonicGod
@TheSonicGod: Were you able to point to your DC using the cmdlet I provided earlier?

Regards,
Exchange_Geek
Just typing you my post Exchange_Geek:

I tried you command line in EMS console and got the following error:

[PS] C:\Windows\system32>Set-ExchangeServer -Identity EXCHSERVER -StaticDomainControllers SERVER005.ERBERB.LOCAL -StaticGl
obalCatalogs SERVER005.ERBERB.LOCAL -StaticExcludedDomainControllers SERVER005.ERBERB.LOCAL
The specified domain controller "SERVER005.ERBERB.LOCAL" can't be in both the excluded domain controller and the other three.
    + CategoryInfo          : InvalidOperation: (EXCHSERVER:ServerIdParameter) [Set-ExchangeServer], InvalidOperation
   Exception
    + FullyQualifiedErrorId : 7834D20B,Microsoft.Exchange.Management.SystemConfigurationTasks.SetExchangeServer

Did I not do the command correctly?
Also Ran the dcdiag /fix on the 2nd DC server and got these errors:

 Starting test: NCSecDesc
    Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
       Replicating Directory Changes In Filtered Set
    access rights for the naming context:
    DC=DomainDnsZones,DC=erberb,DC=local
    ......................... SERVERSQL1 failed test NCSecDesc

      Starting test: SystemLog
         An error event occurred.  EventID: 0xC0002720
            Time Generated: 08/02/2012   01:01:17
            Event String:
            The application-specific permission settings do not grant Local Laun
ch permission for the COM Server application with CLSID
         An error event occurred.  EventID: 0xC0002720
            Time Generated: 08/02/2012   01:01:17
     ......................... SERVERSQL1 failed test SystemLog
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Hi H-K,

I ran repadmin /syncall and here is the output:

CALLBACK MESSAGE: The following replication is in progress:
    From: 485bb582-95a3-4e8f-bf14-2d943c44392a._msdcs.erberb.local
    To  : ea937a96-1205-4a69-a43e-42d863d68c2d._msdcs.erberb.local
CALLBACK MESSAGE: The following replication completed successfully:
    From: 485bb582-95a3-4e8f-bf14-2d943c44392a._msdcs.erberb.local
    To  : ea937a96-1205-4a69-a43e-42d863d68c2d._msdcs.erberb.local
CALLBACK MESSAGE: SyncAll Finished.
SyncAll terminated with no errors.

There are no 2003 DC's anymore as my friend apparently demoted all of them a couple of weeks ago and this is when these issues started - there server are still on the network but have been completely demoted

Thanks for your help - let me know your thoughts on the above.

TheSonicGod
Hi Exchange_Geek,

I ran the command in EMS: Set-ExchangeServer -Identity “Your Mail Server Name” -StaticDomainControllers "FQDN of local DC" -StaticGlobalCatalogs "FQDN of local GC" -StaticConfigDomainController "FQDN of local GC"

both with then without the quotes (updated the parameters within the quote areas) and it seem to accept it both ways (was not sure if I should leave the quotes or not)

I assume I do not have to reboot or restart exchange services after this command for these settings to take affect.

And I checked and there is no Read-Only DC's on the netowork (RODC) so I will ignore the NCSecDesc error from dcdiag results as per your MS article link.

Please advise if there is anything else I need to run.

thanks,

TheSonicGod
ASKER CERTIFIED SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Hi Exchange_Geek, H-K,

The exchange shell command seems to have stabilized the email - except that BES services (also installed on this server) crashed over the weekend).

I have not got a chance to do any of the the Domain diagnosis yet but will be complete this this week.

I will advise further once I have more.

thanks,

TheSonicGod
Great, some good news always lightens up the day. We'll wait for your response.

Regards,
Exchange_Geek
Hi Exchange_Geek & H-K,

Sorry for the delay in my reply - Ran into some major issues with this setup. It seems that someone in the past has changed and removed the ability for the administrator account in security that now prevents the administrator from running any of the adprep settings.

I am having to manually change/fix the items found but it is a slow process.

I will advise further once I have more.

thanks,

TheSonicGod
Thanks Everyone