Solved

Excahnge 2010 server logs MSExchange ADAccess errors when connected to child doamin server

Posted on 2013-10-27
16
1,526 Views
Last Modified: 2013-11-05
Hi all, I hope someone on here can help me. Some back story.

Last week we demoted the last 2003 domain controller from our network. We are now running 3 2008r2 standard domain controllers for the root domain and 2 2008r2 in the child domain.

Everything went well with the demotion. Thought to myself “job well done” that went pretty smooth :) Oh did that come back to haunt me.

A few hours later, calls started coming in. First off OWA users could not access email. Then phones and devices stopped sending receiving email. Then finely outlook clients went off line.

Logged onto the exchange server (Exchange 2010) and found a lot of “MSExchange ADAccess” Errors Event ID 2130 in application event viewer. Also around the same time, I see “GroupPolicy” 1006 errors in System logs.

The only way to get exchange working again was to restart all the services.
That gave me some breathing room to do some digging. What I found was this only happened when the exchange server connect to one of the child DCs.

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=10188). The Configuration Domain Controller has been changed from ROOT.DC.com to CHILD.DC.com.  

The reason it changes is

Process Microsoft.Exchange.Search.ExSearch.exe (PID=7684).  Exchange Active Directory Provider lost contact with domain controller ROOT.DC.com.  Error was 0x51 (ServerDown) (Active directory response: The LDAP server is unavailable.).  Exchange Active Directory Provider will attempt to reconnect with this domain controller when it is reachable.  

I thought maybe the exchange server does not have the correct permissions in the child domain. So I checked the default domain controllers policy for “Manage auditing and security log” and made sure “MyDomain\Exchange Servers, BUILTIN\Administrators, MyDomin\Exchange Enterprise Servers” was included.

Ok so this is where I am. Anyone run into this before? Or have any suggestions on what to do from here?

Thanks in advance.
0
Comment
Question by:ruaidhrigh
  • 9
  • 7
16 Comments
 
LVL 11

Expert Comment

by:NetoMeter Screencasts
Comment Utility
Hi,

You might find helpful the following steps:

1. Make sure the demoted DC object is removed from sites and services -  you will see it as a pending object, without connections under it. It is not removed automatically after demotion, and you will have to remove it manually.
2. Double check that all DC are GC as well - check their NTDS properties in AD Sites and Services.
3. Make sure that all DC in each domain see the same FSMO role distribution with "NETDOM QUERY FSMO"
4. Force a Push/Pull replication in each domain and check the result - "repadmin /syncall /AeP" and "repadmin /syncall /Ae" (the command switches are case sensitive). Then check the result with "repadmin /replsum". if there are errors, you might find it helpful to export the result in CSV file - "repadmin /showrepl * /csv > showrepl.csv"
5. Run DCDIAG in verbose (if there are issues, add "/d" for debug mode) - "dcdiag /v /f:dcdiag.log"
6. Probably, that's redundant, but check whether you get the GC listed with "nslookup gc._msdcs" in the parent and child domains
7. Inspect the AD integrated DNS zones - the demoted DC should be automatically removed from the "Name Servers" list. I am sure, you've already removed the demoted DC from the DNS settings of the static and DHCP clients.  
8. If you suspect DNS problems, download DNSLINT and run (replace the IP with the actual IP of a DC) "dnslint /ad /s 192.168.1.10 /v".
9. Check which is the preferred AD server in Exchange management Shell (EMS) with "Get-ADServerSettings | fl"
Preferred AD Server in Exchange 201010. Then start LDP.exe, connect to that server, and make sure that it responds with "isGlobalCatalogReady: TRUE;"
isGlobalCatalogReady
0
 

Author Comment

by:ruaidhrigh
Comment Utility
First of all thank you for all the info and help. This is my first time asking for help on this site, but not my first getting answers off it…It is helpful people like you that make my job look easy…

1. Make sure the demoted DC object is removed from sites and services -  you will see it as a pending object, without connections under it. It is not removed automatically after demotion, and you will have to remove it manually.

      Demoted PC has been removed.

2. Double check that all DC are GC as well - check their NTDS properties in AD Sites and Services.

      All DCs are GCs

3. Make sure that all DC in each domain see the same FSMO role distribution with "NETDOM QUERY FSMO"
Child Domain
Schema Master                   - RootDC2
Domain Naming Master       - RootDC2
PDC                                        - ChildDC2
RID                                        - ChildDC2
Infra Master                          - ChildDC2

Root Domain
Schema Master                   - RootDC2
Domain Naming Master       - RootDC2
PDC                                        - RootDC1
RID                                        - RootDC3
Infra Master                          - RootDC3


4. Force a Push/Pull replication in each domain and check the result - "repadmin /syncall /AeP" and "repadmin /syncall /Ae" (the command switches are case sensitive). Then check the result with "repadmin /replsum". if there are errors, you might find it helpful to export the result in CSV file - "repadmin /showrepl * /csv > showrepl.csv"

                   No errors shown

5. Run DCDIAG in verbose (if there are issues, add "/d" for debug mode) - "dcdiag /v /f:dcdiag.log"

                    Failed with , NCSecDesc but that ok as we do not plan on deploying a RODC.

6. Probably, that's redundant, but check whether you get the GC listed with "nslookup gc._msdcs" in the parent and child domains

                    Yes on both.

7. Inspect the AD integrated DNS zones - the demoted DC should be automatically removed from the "Name Servers" list. I am sure, you've already removed the demoted DC from the DNS settings of the static and DHCP clients.  
                   
                     Yes

8. If you suspect DNS problems, download DNSLINT and run (replace the IP with the actual IP of a DC) "dnslint /ad /s 192.168.1.10 /v".
                     
                      No Errors

9. Check which is the preferred AD server in Exchange management Shell (EMS) with "Get-ADServerSettings | fl"

RunspaceId                                                   : 5d2f8536-6786-46cb-9ce0-062e2efba02f
DefaultGlobalCatalog                                   : RootDC2.MyDomain.com                    PreferredDomainControllerForDomain               : {}                                           DefaultConfigurationDomainController             : ChildDC2.MyChild.MyDomain.com      DefaultPreferredDomainControllers               : {RootDC2. MyDomain.com}                  UserPreferredGlobalCatalog                             :                                              UserPreferredConfigurationDomainController       :                                              UserPreferredDomainControllers                   : {}
RecipientViewRoot                                      : MyDomain.com
ViewEntireForest                                       : False
WriteOriginatingChangeTimestamp                  : False  
WriteShadowProperties                                  : False  
Identity                                                    :                                            
IsValid                                                      : True                                    



10. Then start LDP.exe, connect to that server, and make sure that it responds with "isGlobalCatalogReady: TRUE;"

            isGlobalCatalogReady: TRUE; On all DCs
0
 
LVL 11

Expert Comment

by:NetoMeter Screencasts
Comment Utility
It seems you are using "ChildDC2.MyChild.MyDomain.com" as the configuration DC.

Could you open EMC, choose Organization Configuration, and click on "Modify Configuration Domain Controller" in the Action Pane. Then choose the root domain the the "Domain field" and try one of the RooDC (it must be in the same site where Exchange is located):
Specify Exchange 2010 Configuration Domain Controller
0
 

Author Comment

by:ruaidhrigh
Comment Utility
I have tried that before, It will work for a few hours, but then it will fail too.

Forgot to mention, we have 2 2010 exchange servers. Both have the same issue.

I did notice that GroupPolicy fails for computer configuration only just before we get MSExchange ADAccess 2130 error

The processing of Group Policy failed. Windows could not authenticate to the Active Directory service on a domain controller. (LDAP Bind function call failed). Look in the details tab for error code and description.
0
 

Author Comment

by:ruaidhrigh
Comment Utility
Also I cant run EMC because of this error, so I imported Exchange modules into Powershell with  "add-pssnapin Microsoft.Exchange.Management.PowerShell.E2010 and run "Get-ADServerSettings | fl" on the effected server and get this.


DefaultGlobalCatalog                       :
PreferredDomainControllerForDomain         : {}
DefaultConfigurationDomainController       :
DefaultPreferredDomainControllers          : {}
UserPreferredGlobalCatalog                 :
UserPreferredConfigurationDomainController :
UserPreferredDomainControllers             : {}
RecipientViewRoot                          :
ViewEntireForest                           : True
WriteOriginatingChangeTimestamp            : False
WriteShadowProperties                      : False
Identity                                   :
IsValid                                    : True

This is a very strange.
0
 
LVL 11

Expert Comment

by:NetoMeter Screencasts
Comment Utility
Could you run DCDIAG on all Domain controllers and check the logs for errors.

Also, check whether you have DCs with more than one NIC registered in DNS. If the second NIC has an IP registered in the DNS zone, that might present connectivity problem to clients (trying to access the second IP instead of the main one).
0
 

Author Comment

by:ruaidhrigh
Comment Utility
ChildDC01
      No Errors

ChildDC02
Starting test: Replications

         * Replications Check
         [Replications Check,ChildDC02] DsReplicaGetInfo(PENDING_OPS, NULL) failed, error 0x2105

         "Replication access was denied."

         ......................... ChildDC02 failed test Replications

A warning event occurred.  EventID: 0x0000008E

            Time Generated: 10/27/2013   23:35:58

            Event String:

            The time service has stopped advertising as a time source because the local clock is not synchronized.

Do you think this could be the cause?


RootDC01
      No Errors


RootDC02
      No Errors


RootDC03
      No Errors
0
 
LVL 11

Expert Comment

by:NetoMeter Screencasts
Comment Utility
Check this for configuring Time Source - Keeping your domains in time.

Are you running the DCs on Virtual Machines and if yes, what kind of Hypervisor are you using?
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:ruaidhrigh
Comment Utility
Time is configured properly. We have 2 in root domain and 1 in child. We are running VMWare 4.1 I have unchecked time synchronization between host and server on all DC VMs.

Here is the progression of the errors in the event logs.

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1344). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
 (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
In-site:
ChildDC02.Domain.MyDomain.com      CDG 1 7 7 1 0 1 1 7 1
ChildDC01.Domain.MyDomain.com      CDG 1 7 7 1 0 1 1 7 1
RootDC01. MyDomain.com            CDG 1 7 7 1 0 1 1 7 1
RootDC02. MyDomain.com            CDG 1 7 7 1 0 1 1 7 1
RootDC03. MyDomain.com            CDG 1 7 7 1 0 1 1 7 1
 Out-of-site:

10 Minutes later

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1344). Topology discovery failed, error 0x80040952 (LDAP_LOCAL_ERROR (Client-side internal error or bad LDAP message)). Look up the Lightweight Directory Access Protocol (LDAP) error code specified in the event description. To do this, use Microsoft Knowledge Base article 218185, "Microsoft LDAP Error Codes." Use the information in that article to learn more about the cause and resolution to this error. Use the Ping or PathPing command-line tools to test network connectivity to local domain controllers.

Then

Process Microsoft.Exchange.Search.ExSearch.exe (PID=3704).  Exchange Active Directory Provider lost contact with domain controller RootDC2.MyDomain.com.  Error was 0x51 (ServerDown) (Active directory response: The LDAP server is unavailable.).  Exchange Active Directory Provider will attempt to reconnect with this domain controller when it is reachable.  

Then

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1344). The Configuration Domain Controller has been changed from RootDC2.MyDomain.com ChildDC1.Domain.MyDomain.com.

Then 10 seconds later

 Error Logs
0
 

Author Comment

by:ruaidhrigh
Comment Utility
This is happening right now, Outlook , ActiveSync is working but OWA is not. I will have to restart the exchange services to get it working again.
0
 
LVL 11

Expert Comment

by:NetoMeter Screencasts
Comment Utility
You are getting a time sync error, and that can be the reason for the sync issues.

Could you check and compare time on all DC by pasting these commands in CMD on the Root Domain PDC - RootDC1 (replace "yourootdomain.local" and "child.yourootdomain.local" with your internal AD Domains):

net time \\RootDC1.yourootdomain.local
net time \\RootDC2.yourootdomain.local
net time \\RootDC3.yourootdomain.local

net time \\RootDC2.child.yourootdomain.local
net time \\ChildDC2.child.yourootdomain.local

Open in new window


Make sure that you compare also the year, month, and day.

Then set the PDC at the root (just the PDC at the root - RootDC1) to sync with external source:

W32tm /config /manualpeerlist:time.nist.gov /syncfromflags:manual /reliable:yes /update
W32tm /resync /rediscover
net stop w32time &&  net start w32time 

Open in new window


and sync all remaining DC - in the root and in the child by running these commands on each of the remaining DC (including the PDC in the child domain):

w32tm /config /syncfromflags:domhier /update
W32tm /resync /rediscover
net stop w32time &&  net start w32time 

Open in new window


Finally, compare again time:

net time \\RootDC1.yourootdomain.local
net time \\RootDC2.yourootdomain.local
net time \\RootDC3.yourootdomain.local

net time \\RootDC2.child.yourootdomain.local
net time \\ChildDC2.child.yourootdomain.local

Open in new window

0
 

Author Comment

by:ruaidhrigh
Comment Utility
Time is correct on all DCs. I fixed the time sync issue when I seen it coming up by stopping it from syncing with the ESX host.

I think I might have to open a call with Microsoft. Will let you know how I get on.
0
 
LVL 11

Expert Comment

by:NetoMeter Screencasts
Comment Utility
That will be nice.
0
 
LVL 11

Assisted Solution

by:NetoMeter Screencasts
NetoMeter Screencasts earned 500 total points
Comment Utility
I've noticed that the Topology service discovers fine the DCs:
--------------------------------------------------------------------------------------
Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1344). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
 (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
In-site:
ChildDC02.Domain.MyDomain.com      CDG 1 7 7 1 0 1 1 7 1
ChildDC01.Domain.MyDomain.com      CDG 1 7 7 1 0 1 1 7 1
RootDC01. MyDomain.com            CDG 1 7 7 1 0 1 1 7 1
RootDC02. MyDomain.com            CDG 1 7 7 1 0 1 1 7 1
RootDC03. MyDomain.com            CDG 1 7 7 1 0 1 1 7 1
-------------------------------------------------------------------------------
and the SACL (Manage auditing and security log) rights are fine on the DC’s.

The problem is that Exchange is simply losing connectivity with "RootDC2.MyDomain.com":
-------------------------
Exchange Active Directory Provider lost contact with domain controller RootDC2.MyDomain.com.  Error was 0x51 (ServerDown) (Active directory response: The LDAP server is unavailable.).  

-------------------------

and is choosing one of the child domain controllers.

I would suggest:
1. Check the connection on the "RootDC2.MyDomain.com". Change the port on the switch (could be a flapping port), make sure it is not connected with multiple NICS, check the NIC bindings and make sure the correct NIC is on top of the bindings, set the connection speed to a fixed value (100 MB) instead of auto.
2. The root and the child domains are in the same site. If the child domain is using a different subnet (which it most probably is), just create a second site in Sites and Services and place the Child Domain in it. Exchange will choose a DC from the Root domain in this case, as only root DCs will be in the local/Exchange site.
0
 

Accepted Solution

by:
ruaidhrigh earned 0 total points
Comment Utility
Ok, finally got this resolved.

I spent the last few days on the phone with Microsoft they.  It turns out it was a combination of things, simple things. Isn’t it always? ¿

1: IPv6 – They recommend to disable IPv6 on all exchange servers running server 2008 r2
2: Windows update – Fully update windows.
3: Restart the DCs to clear out any cashed entry’s they might have.

I hope this helps someone else if they have the same issue.

Thank you NetoMeter for all you help…
0
 

Author Closing Comment

by:ruaidhrigh
Comment Utility
This is what worked for me.
0

Featured Post

What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

Join & Write a Comment

Exchange server is not supported in any cloud-hosted platform (other than Azure with Azure Premium Storage).
Restoring deleted objects in Active Directory has been a standard feature in Active Directory for many years, yet some admins may not know what is available.
This tutorial will give a an overview on how to deploy remote agents in Backup Exec 2012 to new servers. Click on the Backup Exec button in the upper left corner. From here, are global settings for the application such as connecting to a remote Back…
This tutorial will show how to configure a new Backup Exec 2012 server and move an existing database to that server with the use of the BEUtility. Install Backup Exec 2012 on the new server and apply all of the latest hotfixes and service packs. The…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now