TheSonicGod
asked on
Exchange 2010 Server keeps locking up
Hi Everyone,
I am helping out a frend who is a network admin as he seems to be having some major issues with his exchange server.
It is a Exchange 2010 running on a windows 2008 x64 OS.
He has to reboot the unit at least once a day as mail flow just seems to stop.
I reviewed the windows logs and ran BPA utility and found a number of issues. Because he advised to me that he has been promoting 2008 servers as DC's on the network and demoting 2003 DC's from the network the following 2 errors from the BPA report peeked my interest:
1) Unrecognized Exchange Signature - Active Directory domain 'ERBERB' has an unrecognized Exchange signature. Current DomainPrep version: 12639
2) Microsoft Exchange System Attendent ‘homeMDB’ is missing - The 'homeMDB' value for the Microsoft Exchange System Attendant service on server ERBEXCHANGE is missing. This will cause reliability problems
In the windows System logs I also found a lot of errors including (in order from after reboot to required next reboot:
3) Event ID: 1500 - Source: SNMP - The SNMP Service encountered an error while accessing the registry key SYSTEM\CurrentControlSet\S ervices\SN MP\Paramet ers\TrapCo nfiguratio n. - logged: 7/30/2012 11:16:53 PM
4) Event ID: 1006 - Source: GroupPolicy - The processing of Group Policy failed. Windows could not authenticate to the Active Directory service on a domain controller. (LDAP Bind function call failed). Look in the details tab for error code and description. - logged: 7/31/2012 9:43:05 AM
Reboot completed - 11:00 am 7/31/2012
5) Event ID: 1 - Source: VDS Basic Provider - Unexpected failure. Error code: 490@01010004 - logged: 7/31/2012 8:01:12 PM
6) Event ID: 21 - Source: ARCFlashVolDrv - Failed to Write File \Device\HarddiskVolume3\CA VolTrc.dat Status : C00000A2. - logged: 7/31/2012 8:02:19 PM
7) Event ID: 137 - Source: Ntfs - The default transaction resource manager on volume \\?\Volume{00d75fb7-db21-1 1e1-b91e-0 050569b000 7} encountered a non-retryable error and could not start. The data contains the error code. - logged: 7/31/2012 8:02:21 PM
8) Event ID: 21 - Source: ARCFlashVolDrv - Failed to Write File \Device\HarddiskVolume4\CA VolTrc.dat Status : C0000010. - logged: 7/31/2012 8:02:22 PM
In the Application logs I also found errors:
9) Event ID: 1906 - Source: MSExchangeApplicationLog - Service MSExchangeMailSubmission. Exchange topology discovery encountered an exception Microsoft.Exchange.Data.Di rectory.AD TransientE xception: Could not find any available Domain Controller.
at Microsoft.Exchange.Data.Di rectory.Co nnectionPo olManager. GetConnect ion(Connec tionType connectionType, ADObjectId domain, String serverName, Int32 port, NetworkCredential credential)
at Microsoft.Exchange.Data.Di rectory.Co nnectionPo olManager. GetConnect ion(Connec tionType connectionType)
at Microsoft.Exchange.Data.Di rectory.AD Session.Ge tConnectio n(String preferredServer, Boolean isWriteOperation, Boolean isNotifyOperation, String optionalBaseDN, ADObjectId& rootId, ADScope scope)
at Microsoft.Exchange.Data.Di rectory.AD Session.Ge tReadConne ction(Stri ng preferredServer, String optionalBaseDN, ADObjectId& rootId, ADRawEntry scopeDeteriminingObject)
at Microsoft.Exchange.Data.Di rectory.AD Session.Fi nd(ADObjec tId rootId, String optionalBaseDN, ADObjectId readId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCreator, CreateObjectsDelegate arrayCreator, Boolean includeDeletedObjects)
at Microsoft.Exchange.Data.Di rectory.AD Session.Fi nd(ADObjec tId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCtor, CreateObjectsDelegate arrayCtor)
at Microsoft.Exchange.Data.Di rectory.AD Session.Fi nd[TResult ](ADObject Id rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties)
at Microsoft.Exchange.Data.Di rectory.Sy stemConfig uration.AD SystemConf igurationS ession.Fin d[TResult] (ADObjectI d rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults)
at Microsoft.Exchange.Data.Di rectory.Sy stemConfig uration.AD SystemConf igurationS ession.Fin dServerByF qdn(String serverFqdn)
at Microsoft.Exchange.Data.Di rectory.Sy stemConfig uration.AD SystemConf igurationS ession.Fin dLocalServ er()
at Microsoft.Exchange.Data.Ap plicationL ogic.Picke rServerLis t.LoadFrom AD()
at Microsoft.Exchange.Data.Ap plicationL ogic.ADCon figuration Loader`2.< >c__Displa yClass1.<R eadConfigu ration>b__ 0()
at Microsoft.Exchange.Data.Di rectory.AD Notificati onAdapter. RunADOpera tion(ADOpe ration adOperation, Int32 retryCount)
at Microsoft.Exchange.Data.Di rectory.AD Notificati onAdapter. TryRunADOp eration(AD Operation adOperation, Int32 retryCount) when trying to load topology information. - logged: 7/30/2012 11:13:31 PM
10) Event ID: 105 - Source: MSExchange Mailbox Replication - The Mailbox Replication service was unable to determine the list of mailbox databases hosted in the local Active Directory site. - logged: 7/30/2012 11:15:02 PM
11) Event ID: 1000 - Source: Application Error - Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: KERNELBASE.dll, version: 6.1.7601.17651, time stamp: 0x4e21213c
Exception code: 0xe0434f4d
Fault offset: 0x000000000000cacd
Faulting process id: 0x3b90
Faulting application start time: 0x01cd6e65aba21671
Faulting application path: c:\windows\system32\inetsr v\w3wp.exe
Faulting module path: C:\Windows\system32\KERNEL BASE.dll
Report Id: eb12c7df-dabd-11e1-9977-00 50569b0007
Error: Could not find any available Domain Controller. - logged: 7/30/2012 11:15:03 PM
12) Event ID: 2114 - Source: MSExchange ADAccess - Process MSEXCHANGEADTOPOLOGYSERVIC E.EXE (PID=2644). Topology discovery failed, error 0x80040956 (LDAP_AUTH_UNKNOWN (An unknown authentication error occurred)). Look up the Lightweight Directory Access Protocol (LDAP) error code specified in the event description. To do this, use Microsoft Knowledge Base article 218185, "Microsoft LDAP Error Codes." Use the information in that article to learn more about the cause and resolution to this error. Use the Ping or PathPing command-line tools to test network connectivity to local domain controllers. - logged: 7/30/2012 11:15:14 PM
13) Event ID: 2917 - Source: MSExchange ADAccess - Process Microsoft.Exchange.Imap4.e xe (PopImap) (PID=5656). A budget charge was encountered that exceeded the limit of '5.04973333333333' minutes. Budget Owner: 'Sid~DOMAIN\itsupport~IMAP ~False', Component: 'IMAP', CostType: 'CAS'. - logged: 7/30/2012 11:15:14 PM
14) Event ID: 2102 - Source: MSExchange ADAccess - Process MAD.EXE (PID=7380). All Domain Controller Servers in use are not responding:
SRV005.DOMAIN.local
SRVSQL1.DOMAIN.local
logged: 7/30/2012 11:15:16 PM
And about another 20 others - I can post if required.
To me it looks there is an issue with the AD configurtation or possible the ArcServ backup software but I am not sure where to go from here
Any suggestions would be helpful, thanks in advise for your responses,
TheSonicGod
I am helping out a frend who is a network admin as he seems to be having some major issues with his exchange server.
It is a Exchange 2010 running on a windows 2008 x64 OS.
He has to reboot the unit at least once a day as mail flow just seems to stop.
I reviewed the windows logs and ran BPA utility and found a number of issues. Because he advised to me that he has been promoting 2008 servers as DC's on the network and demoting 2003 DC's from the network the following 2 errors from the BPA report peeked my interest:
1) Unrecognized Exchange Signature - Active Directory domain 'ERBERB' has an unrecognized Exchange signature. Current DomainPrep version: 12639
2) Microsoft Exchange System Attendent ‘homeMDB’ is missing - The 'homeMDB' value for the Microsoft Exchange System Attendant service on server ERBEXCHANGE is missing. This will cause reliability problems
In the windows System logs I also found a lot of errors including (in order from after reboot to required next reboot:
3) Event ID: 1500 - Source: SNMP - The SNMP Service encountered an error while accessing the registry key SYSTEM\CurrentControlSet\S
4) Event ID: 1006 - Source: GroupPolicy - The processing of Group Policy failed. Windows could not authenticate to the Active Directory service on a domain controller. (LDAP Bind function call failed). Look in the details tab for error code and description. - logged: 7/31/2012 9:43:05 AM
Reboot completed - 11:00 am 7/31/2012
5) Event ID: 1 - Source: VDS Basic Provider - Unexpected failure. Error code: 490@01010004 - logged: 7/31/2012 8:01:12 PM
6) Event ID: 21 - Source: ARCFlashVolDrv - Failed to Write File \Device\HarddiskVolume3\CA
7) Event ID: 137 - Source: Ntfs - The default transaction resource manager on volume \\?\Volume{00d75fb7-db21-1
8) Event ID: 21 - Source: ARCFlashVolDrv - Failed to Write File \Device\HarddiskVolume4\CA
In the Application logs I also found errors:
9) Event ID: 1906 - Source: MSExchangeApplicationLog - Service MSExchangeMailSubmission. Exchange topology discovery encountered an exception Microsoft.Exchange.Data.Di
at Microsoft.Exchange.Data.Di
at Microsoft.Exchange.Data.Di
at Microsoft.Exchange.Data.Di
at Microsoft.Exchange.Data.Di
at Microsoft.Exchange.Data.Di
at Microsoft.Exchange.Data.Di
at Microsoft.Exchange.Data.Di
at Microsoft.Exchange.Data.Di
at Microsoft.Exchange.Data.Di
at Microsoft.Exchange.Data.Di
at Microsoft.Exchange.Data.Ap
at Microsoft.Exchange.Data.Ap
at Microsoft.Exchange.Data.Di
at Microsoft.Exchange.Data.Di
10) Event ID: 105 - Source: MSExchange Mailbox Replication - The Mailbox Replication service was unable to determine the list of mailbox databases hosted in the local Active Directory site. - logged: 7/30/2012 11:15:02 PM
11) Event ID: 1000 - Source: Application Error - Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: KERNELBASE.dll, version: 6.1.7601.17651, time stamp: 0x4e21213c
Exception code: 0xe0434f4d
Fault offset: 0x000000000000cacd
Faulting process id: 0x3b90
Faulting application start time: 0x01cd6e65aba21671
Faulting application path: c:\windows\system32\inetsr
Faulting module path: C:\Windows\system32\KERNEL
Report Id: eb12c7df-dabd-11e1-9977-00
Error: Could not find any available Domain Controller. - logged: 7/30/2012 11:15:03 PM
12) Event ID: 2114 - Source: MSExchange ADAccess - Process MSEXCHANGEADTOPOLOGYSERVIC
13) Event ID: 2917 - Source: MSExchange ADAccess - Process Microsoft.Exchange.Imap4.e
14) Event ID: 2102 - Source: MSExchange ADAccess - Process MAD.EXE (PID=7380). All Domain Controller Servers in use are not responding:
SRV005.DOMAIN.local
SRVSQL1.DOMAIN.local
logged: 7/30/2012 11:15:16 PM
And about another 20 others - I can post if required.
To me it looks there is an issue with the AD configurtation or possible the ArcServ backup software but I am not sure where to go from here
Any suggestions would be helpful, thanks in advise for your responses,
TheSonicGod
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hi K-H,
here are the items I completed as requested:
1) Checked DNS servers for the network and both DC's are also DNS and set statically on the exchange server
2) I ran dcdiag /fix on SERVER005 and got the following error:
Starting test: NCSecDesc
Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
Replicating Directory Changes In Filtered Set
access rights for the naming context:
DC=DomainDnsZones,DC=DOMAI N,DC=local
......................... ERBERB failed test NCSecDesc
No other error were reported except this - everything else passed
3) Ran ipconfig /flushdns on the exchange
4) Ran ipconfig /registerdns on both servers. (AD& Ex) - checked error logs for both - no errors related to DNS were found
5) Ping erberb.local and got reply back from one of the DC's IP address
Let me know if anything else is required.
thanks,
TheSonicGod
here are the items I completed as requested:
1) Checked DNS servers for the network and both DC's are also DNS and set statically on the exchange server
2) I ran dcdiag /fix on SERVER005 and got the following error:
Starting test: NCSecDesc
Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
Replicating Directory Changes In Filtered Set
access rights for the naming context:
DC=DomainDnsZones,DC=DOMAI
......................... ERBERB failed test NCSecDesc
No other error were reported except this - everything else passed
3) Ran ipconfig /flushdns on the exchange
4) Ran ipconfig /registerdns on both servers. (AD& Ex) - checked error logs for both - no errors related to DNS were found
5) Ping erberb.local and got reply back from one of the DC's IP address
Let me know if anything else is required.
thanks,
TheSonicGod
@TheSonicGod: Were you able to point to your DC using the cmdlet I provided earlier?
Regards,
Exchange_Geek
Regards,
Exchange_Geek
ASKER
Just typing you my post Exchange_Geek:
I tried you command line in EMS console and got the following error:
[PS] C:\Windows\system32>Set-Ex changeServ er -Identity EXCHSERVER -StaticDomainControllers SERVER005.ERBERB.LOCAL -StaticGl
obalCatalogs SERVER005.ERBERB.LOCAL -StaticExcludedDomainContr ollers SERVER005.ERBERB.LOCAL
The specified domain controller "SERVER005.ERBERB.LOCAL" can't be in both the excluded domain controller and the other three.
+ CategoryInfo : InvalidOperation: (EXCHSERVER:ServerIdParame ter) [Set-ExchangeServer], InvalidOperation
Exception
+ FullyQualifiedErrorId : 7834D20B,Microsoft.Exchang e.Manageme nt.SystemC onfigurati onTasks.Se tExchangeS erver
Did I not do the command correctly?
I tried you command line in EMS console and got the following error:
[PS] C:\Windows\system32>Set-Ex
obalCatalogs SERVER005.ERBERB.LOCAL -StaticExcludedDomainContr
The specified domain controller "SERVER005.ERBERB.LOCAL" can't be in both the excluded domain controller and the other three.
+ CategoryInfo : InvalidOperation: (EXCHSERVER:ServerIdParame
Exception
+ FullyQualifiedErrorId : 7834D20B,Microsoft.Exchang
Did I not do the command correctly?
ASKER
Also Ran the dcdiag /fix on the 2nd DC server and got these errors:
Starting test: NCSecDesc
Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
Replicating Directory Changes In Filtered Set
access rights for the naming context:
DC=DomainDnsZones,DC=erber b,DC=local
......................... SERVERSQL1 failed test NCSecDesc
Starting test: SystemLog
An error event occurred. EventID: 0xC0002720
Time Generated: 08/02/2012 01:01:17
Event String:
The application-specific permission settings do not grant Local Laun
ch permission for the COM Server application with CLSID
An error event occurred. EventID: 0xC0002720
Time Generated: 08/02/2012 01:01:17
......................... SERVERSQL1 failed test SystemLog
Starting test: NCSecDesc
Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
Replicating Directory Changes In Filtered Set
access rights for the naming context:
DC=DomainDnsZones,DC=erber
......................... SERVERSQL1 failed test NCSecDesc
Starting test: SystemLog
An error event occurred. EventID: 0xC0002720
Time Generated: 08/02/2012 01:01:17
Event String:
The application-specific permission settings do not grant Local Laun
ch permission for the COM Server application with CLSID
An error event occurred. EventID: 0xC0002720
Time Generated: 08/02/2012 01:01:17
......................... SERVERSQL1 failed test SystemLog
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hi H-K,
I ran repadmin /syncall and here is the output:
CALLBACK MESSAGE: The following replication is in progress:
From: 485bb582-95a3-4e8f-bf14-2d 943c44392a ._msdcs.er berb.local
To : ea937a96-1205-4a69-a43e-42 d863d68c2d ._msdcs.er berb.local
CALLBACK MESSAGE: The following replication completed successfully:
From: 485bb582-95a3-4e8f-bf14-2d 943c44392a ._msdcs.er berb.local
To : ea937a96-1205-4a69-a43e-42 d863d68c2d ._msdcs.er berb.local
CALLBACK MESSAGE: SyncAll Finished.
SyncAll terminated with no errors.
There are no 2003 DC's anymore as my friend apparently demoted all of them a couple of weeks ago and this is when these issues started - there server are still on the network but have been completely demoted
Thanks for your help - let me know your thoughts on the above.
TheSonicGod
I ran repadmin /syncall and here is the output:
CALLBACK MESSAGE: The following replication is in progress:
From: 485bb582-95a3-4e8f-bf14-2d
To : ea937a96-1205-4a69-a43e-42
CALLBACK MESSAGE: The following replication completed successfully:
From: 485bb582-95a3-4e8f-bf14-2d
To : ea937a96-1205-4a69-a43e-42
CALLBACK MESSAGE: SyncAll Finished.
SyncAll terminated with no errors.
There are no 2003 DC's anymore as my friend apparently demoted all of them a couple of weeks ago and this is when these issues started - there server are still on the network but have been completely demoted
Thanks for your help - let me know your thoughts on the above.
TheSonicGod
ASKER
Hi Exchange_Geek,
I ran the command in EMS: Set-ExchangeServer -Identity “Your Mail Server Name” -StaticDomainControllers "FQDN of local DC" -StaticGlobalCatalogs "FQDN of local GC" -StaticConfigDomainControl ler "FQDN of local GC"
both with then without the quotes (updated the parameters within the quote areas) and it seem to accept it both ways (was not sure if I should leave the quotes or not)
I assume I do not have to reboot or restart exchange services after this command for these settings to take affect.
And I checked and there is no Read-Only DC's on the netowork (RODC) so I will ignore the NCSecDesc error from dcdiag results as per your MS article link.
Please advise if there is anything else I need to run.
thanks,
TheSonicGod
I ran the command in EMS: Set-ExchangeServer -Identity “Your Mail Server Name” -StaticDomainControllers "FQDN of local DC" -StaticGlobalCatalogs "FQDN of local GC" -StaticConfigDomainControl
both with then without the quotes (updated the parameters within the quote areas) and it seem to accept it both ways (was not sure if I should leave the quotes or not)
I assume I do not have to reboot or restart exchange services after this command for these settings to take affect.
And I checked and there is no Read-Only DC's on the netowork (RODC) so I will ignore the NCSecDesc error from dcdiag results as per your MS article link.
Please advise if there is anything else I need to run.
thanks,
TheSonicGod
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hi Exchange_Geek, H-K,
The exchange shell command seems to have stabilized the email - except that BES services (also installed on this server) crashed over the weekend).
I have not got a chance to do any of the the Domain diagnosis yet but will be complete this this week.
I will advise further once I have more.
thanks,
TheSonicGod
The exchange shell command seems to have stabilized the email - except that BES services (also installed on this server) crashed over the weekend).
I have not got a chance to do any of the the Domain diagnosis yet but will be complete this this week.
I will advise further once I have more.
thanks,
TheSonicGod
Great, some good news always lightens up the day. We'll wait for your response.
Regards,
Exchange_Geek
Regards,
Exchange_Geek
ASKER
Hi Exchange_Geek & H-K,
Sorry for the delay in my reply - Ran into some major issues with this setup. It seems that someone in the past has changed and removed the ability for the administrator account in security that now prevents the administrator from running any of the adprep settings.
I am having to manually change/fix the items found but it is a slow process.
I will advise further once I have more.
thanks,
TheSonicGod
Sorry for the delay in my reply - Ran into some major issues with this setup. It seems that someone in the past has changed and removed the ability for the administrator account in security that now prevents the administrator from running any of the adprep settings.
I am having to manually change/fix the items found but it is a slow process.
I will advise further once I have more.
thanks,
TheSonicGod
ASKER
Thanks Everyone
For getting rid of the NTFS errors try to run checkdisk with options "Automatically fix file system errors" and "Scan for and attemp recovery of bad sectors"
(This could take a while)
The Exchange theme looks like it is an DNS issue.
Has the Exchange the correct DNS Servers?
Is the DCs DNS server itself?
run dcdiag /fix
run ipconfig /flushdns on the exchange
run ipconfig /registerdns on both servers. (AD& Ex)
Take a client (or the Exchange) and ping the FQDN of the domain.
Regards