Link to home
Start Free TrialLog in
Avatar of dqnet
dqnet

asked on

DC Replication Issues

Hello Experts,

I seem to be having an issue with replication between our domain controllers, the setup is as follows;

One domain
two domain controllers
one is virtulised
one is physical
same site
ping between domain controllers is fine.


Ok, so basically I had to do a bios upgrade to the server that was hosting the virtual machines (domain controller was one of the virtual machines). After the update we had a problem with our cisco switch as smart ports was enabled and stopping traffic between all virtual machines and the physical network containing all other physical machines.

Now we fixed this by disabling smart ports on the 2960 and all virtual machines can communicate successfully with the physical machines and everything is fine.
However; when we started the VM for the domain controller it took a VERY long time to bootup (common with AD / DNS problems I know). When it finally boot up I logged in and immediately tried to ping the second DC. The ping responded fine and all was ok network wise. But all of a sudden the domain controllers are not synchronising. I tried repadmin /syncall and errors are coming up, I tried dcdiag /q and I am also getting errors.
The RPC service cannot communicate with the FSMO holder (in short).

I checked and the dfsr service is running fine. I switched of any firewalls and antivirus and still they cannot communicate except with ping. Nothing has changed at all??

Can someone point me in the right direction as to where to start?
For testing purposes I created an object on the second DC and it did not replicate to the first DC (FSMO holder).

Thanks experts! :)
Avatar of Mike Kline
Mike Kline
Flag of United States of America image

What events are you seeing in your event log?   What  is the result of repadmin /showreps

Have you tested the ports being open using portqry or telnet?

A network trace (wireshar/netmon) might also help but you probably don't need that right now.

Thanks

Mike
Avatar of dqnet
dqnet

ASKER

Hi Mike,

Thanks for your response. See below;




Microsoft Windows [Version 6.0.6002]
Copyright (c) 2006 Microsoft Corporation.  All rights reserved.

C:\Users\Administrator>dcdiag /q
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... IME-DC1 failed test DFSREvent
         [Replications Check,IME-DC1] A recent replication attempt failed:
            From IME-DC2 to IME-DC1
            Naming Context: DC=ForestDnsZones,DC=XXX,DC=com
            The replication generated an error (1726):
            The remote procedure call failed.
            The failure occurred at 2013-10-02 21:11:34.
            The last success occurred at 2013-10-02 20:05:07.
            2 failures have occurred since the last success.
         [Replications Check,IME-DC1] A recent replication attempt failed:
            From IME-DC2 to IME-DC1
            Naming Context: DC=DomainDnsZones,DC=XXX,DC=com
            The replication generated an error (1726):
            The remote procedure call failed.
            The failure occurred at 2013-10-02 21:09:56.
            The last success occurred at 2013-10-02 20:04:39.
            2 failures have occurred since the last success.
         [Replications Check,IME-DC1] A recent replication attempt failed:
            From IME-DC2 to IME-DC1
            Naming Context: CN=Schema,CN=Configuration,DC=XXX,DC=com
            The replication generated an error (1726):
            The remote procedure call failed.
            The failure occurred at 2013-10-02 21:02:40.
            The last success occurred at 2013-10-02 17:55:42.
            6 failures have occurred since the last success.
         [Replications Check,IME-DC1] A recent replication attempt failed:
            From IME-DC2 to IME-DC1
            Naming Context: CN=Configuration,DC=XXX,DC=com
            The replication generated an error (1726):
            The remote procedure call failed.
            The failure occurred at 2013-10-02 20:57:56.
            The last success occurred at 2013-10-02 20:04:36.
            3 failures have occurred since the last success.
         [Replications Check,IME-DC1] A recent replication attempt failed:
            From IME-DC2 to IME-DC1
            Naming Context: DC=XXX,DC=com
            The replication generated an error (1726):
            The remote procedure call failed.
            The failure occurred at 2013-10-02 21:05:29.
            The last success occurred at 2013-10-02 20:05:10.
            2 failures have occurred since the last success.
         ......................... IME-DC1 failed test Replications
         An Error Event occurred.  EventID: 0x00000457
            Time Generated: 10/02/2013   21:47:42
            Event String:
            Driver Microsoft XPS Document Writer v4 required for printer Microso
ft XPS Document Writer is unknown. Contact the administrator to install the driv
er before you log in again.
         ......................... IME-DC1 failed test SystemLog

C:\Users\Administrator>






Log Name:      Directory Service
Source:        Microsoft-Windows-ActiveDirectory_DomainService
Date:          02/10/2013 22:13:33
Event ID:      1308
Task Category: Knowledge Consistency Checker
Level:         Warning
Keywords:      Classic
User:          ANONYMOUS LOGON
Computer:      IME-DC1.XXX.com
Description:
The Knowledge Consistency Checker (KCC) has detected that successive attempts to replicate with the following directory service has consistently failed.
 
Attempts:
7
Directory service:
CN=NTDS Settings,CN=IME-DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=XXX,DC=com
Period of time (minutes):
128
 
The Connection object for this directory service will be ignored, and a new temporary connection will be established to ensure that replication continues. Once replication with this directory service resumes, the temporary connection will be removed.
 
Additional Data
Error value:
1818 The remote procedure call was cancelled.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-ActiveDirectory_DomainService" Guid="{0e8478c5-3605-4e8c-8497-1e730c959516}" EventSourceName="NTDS KCC" />
    <EventID Qualifiers="32768">1308</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>1</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2013-10-02T18:13:33.071Z" />
    <EventRecordID>12274</EventRecordID>
    <Correlation />
    <Execution ProcessID="652" ThreadID="1332" />
    <Channel>Directory Service</Channel>
    <Computer>IME-DC1.XXX.com</Computer>
    <Security UserID="S-1-5-7" />
  </System>
  <EventData>
    <Data>7</Data>
    <Data>CN=NTDS Settings,CN=IME-DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=XXX,DC=com</Data>
    <Data>128</Data>
    <Data>The remote procedure call was cancelled.</Data>
    <Data>1818</Data>
  </EventData>
</Event>


Thanks mate!
On the virtual DC how do you have DNS setup?  Is it pointing to the other DC for primary and itself for secondary?

What are the event log entries?
Avatar of dqnet

ASKER

Yes, each DC points to the other server.

Everything was working fine, no changes were made whatsoever to the DC's.
The only thing was there was a connection interruption between DC's until we got the CISCO switch up and saved so around 10 minutes.


I've included one Error in the Active Directory Log

Thanks!
If you run repadmin /kcc do you get any errors?   Is the time in sync on the two DCs.   You have already checked firewall/ports but double check that too.
I will recommend to reboot the server once and force the replication from AD sites and services or use repadmin /syncall /AdEP.

It seems that there is still some traffic block which is causing the issue you can download port query tool to check that required port are open.
Avatar of dqnet

ASKER

I don't believe it, I went to control panel on DC2 and checked the firewall, it said it is disabled however I clicked switch on/off and it was switched on. I switched it off, rebooted the server and everything is syncing again? Could it have been the firewall on DC2? I thought you could use firewalls between Domain controllers providing the correct exceptions are in place!

Then........ I rebooted DC1 to make sure everything would be ok and the issues have come back again. But basically speaking it shows there are actually no physical connectivity issues, it is something holding up RPC replication from being carried out.

What would be your initial thoughts?
Avatar of dqnet

ASKER

Ok, I ran portqueryUI on both Domain controllers and from DC2 to DC1 everything is OK and it runs extremely quick;


=============================================

 Starting portqry.exe -n 127.0.0.1 -e 135 -p TCP ...


Querying target system called:

 127.0.0.1

Attempting to resolve IP address to a name...


IP address resolved to IME-DC2.XXX.com

querying...

TCP port 135 (epmap service): LISTENING

Using ephemeral source port
Querying Endpoint Mapper Database...
Server's response:

UUID: 12345678-1234-abcd-ef00-01234567cffb
ncacn_ip_tcp:127.0.0.1[49157]

UUID: 2f5f6521-cb55-1059-b446-00df0bce31db Unimodem LRPC Endpoint
ncacn_np:127.0.0.1[\\pipe\\tapsrv]

UUID: 897e2e5f-93f3-4376-9c9c-fd2277495c27 Frs2 Service
ncacn_ip_tcp:127.0.0.1[5722]

UUID: 367abb81-9844-35f1-ad32-98f038001003
ncacn_ip_tcp:127.0.0.1[53392]

UUID: 50abc2a4-574d-40b3-9d66-ee4fd5fba076
ncacn_ip_tcp:127.0.0.1[53391]

UUID: 12345678-1234-abcd-ef00-0123456789ab IPSec Policy agent endpoint
ncacn_ip_tcp:127.0.0.1[53075]

UUID: 6b5bdd1e-528c-422c-af8c-a4079be4fe48 Remote Fw APIs
ncacn_ip_tcp:127.0.0.1[53075]

UUID: 3473dd4d-2e88-4006-9cba-22570909dd10 WinHttp Auto-Proxy Service
ncacn_np:127.0.0.1[\\PIPE\\wkssvc]

UUID: 3473dd4d-2e88-4006-9cba-22570909dd10 WinHttp Auto-Proxy Service
ncacn_np:127.0.0.1[\\PIPE\\W32TIME_ALT]

UUID: 1ff70682-0a51-30e8-076d-740be8cee98b
ncacn_np:127.0.0.1[\\PIPE\\atsvc]

UUID: 378e52b0-c0a9-11cf-822d-00aa0051e40f
ncacn_np:127.0.0.1[\\PIPE\\atsvc]

UUID: 86d35949-83c9-4044-b424-db363231fd0c
ncacn_np:127.0.0.1[\\PIPE\\atsvc]

UUID: 86d35949-83c9-4044-b424-db363231fd0c
ncacn_ip_tcp:127.0.0.1[49158]

UUID: a398e520-d59a-4bdd-aa7a-3c1e0303a511 IKE/Authip API
ncacn_np:127.0.0.1[\\PIPE\\atsvc]

UUID: a398e520-d59a-4bdd-aa7a-3c1e0303a511 IKE/Authip API
ncacn_ip_tcp:127.0.0.1[49158]

UUID: a398e520-d59a-4bdd-aa7a-3c1e0303a511 IKE/Authip API
ncacn_np:127.0.0.1[\\PIPE\\srvsvc]

UUID: 30b044a5-a225-43f0-b3a4-e060df91f9c1
ncacn_np:127.0.0.1[\\PIPE\\atsvc]

UUID: 30b044a5-a225-43f0-b3a4-e060df91f9c1
ncacn_ip_tcp:127.0.0.1[49158]

UUID: 30b044a5-a225-43f0-b3a4-e060df91f9c1
ncacn_np:127.0.0.1[\\PIPE\\srvsvc]

UUID: c9ac6db5-82b7-4e55-ae8a-e464ed7b4277 Impl friendly name
ncacn_np:127.0.0.1[\\PIPE\\atsvc]

UUID: c9ac6db5-82b7-4e55-ae8a-e464ed7b4277 Impl friendly name
ncacn_ip_tcp:127.0.0.1[49158]

UUID: c9ac6db5-82b7-4e55-ae8a-e464ed7b4277 Impl friendly name
ncacn_np:127.0.0.1[\\PIPE\\srvsvc]

UUID: f6beaff7-1e19-4fbb-9f8f-b89e2018337c Event log TCPIP
ncacn_np:127.0.0.1[\\pipe\\eventlog]

UUID: f6beaff7-1e19-4fbb-9f8f-b89e2018337c Event log TCPIP
ncacn_ip_tcp:127.0.0.1[49153]

UUID: 3c4728c5-f0ab-448b-bda1-6ce01eb0a6d5 DHCP Client LRPC Endpoint
ncacn_np:127.0.0.1[\\pipe\\eventlog]

UUID: 3c4728c5-f0ab-448b-bda1-6ce01eb0a6d5 DHCP Client LRPC Endpoint
ncacn_ip_tcp:127.0.0.1[49153]

UUID: 3c4728c5-f0ab-448b-bda1-6ce01eb0a6d6 DHCPv6 Client LRPC Endpoint
ncacn_np:127.0.0.1[\\pipe\\eventlog]

UUID: 3c4728c5-f0ab-448b-bda1-6ce01eb0a6d6 DHCPv6 Client LRPC Endpoint
ncacn_ip_tcp:127.0.0.1[49153]

UUID: 76f226c3-ec14-4325-8a99-6a46348418af
ncacn_np:127.0.0.1[\\PIPE\\InitShutdown]

UUID: d95afe70-a6d5-4259-822e-2c84da1ddb0d
ncacn_np:127.0.0.1[\\PIPE\\InitShutdown]

UUID: d95afe70-a6d5-4259-822e-2c84da1ddb0d
ncacn_ip_tcp:127.0.0.1[49152]

UUID: c9ac6db5-82b7-4e55-ae8a-e464ed7b4277 Impl friendly name
ncacn_np:127.0.0.1[\\pipe\\lsass]

UUID: c9ac6db5-82b7-4e55-ae8a-e464ed7b4277 Impl friendly name
ncacn_np:127.0.0.1[\\PIPE\\protected_storage]

UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_np:127.0.0.1[\\pipe\\lsass]

UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_np:127.0.0.1[\\PIPE\\protected_storage]

UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_ip_tcp:127.0.0.1[49155]

UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_http:127.0.0.1[49156]

UUID: 12345778-1234-abcd-ef00-0123456789ab
ncacn_np:127.0.0.1[\\pipe\\lsass]

UUID: 12345778-1234-abcd-ef00-0123456789ab
ncacn_np:127.0.0.1[\\PIPE\\protected_storage]

UUID: 12345778-1234-abcd-ef00-0123456789ab
ncacn_ip_tcp:127.0.0.1[49155]

UUID: 12345778-1234-abcd-ef00-0123456789ab
ncacn_http:127.0.0.1[49156]

UUID: 12345778-1234-abcd-ef00-0123456789ac
ncacn_np:127.0.0.1[\\pipe\\lsass]

UUID: 12345778-1234-abcd-ef00-0123456789ac
ncacn_np:127.0.0.1[\\PIPE\\protected_storage]

UUID: 12345778-1234-abcd-ef00-0123456789ac
ncacn_ip_tcp:127.0.0.1[49155]

UUID: 12345778-1234-abcd-ef00-0123456789ac
ncacn_http:127.0.0.1[49156]

UUID: 12345778-1234-abcd-ef00-0123456789ac
ncacn_ip_tcp:127.0.0.1[49157]

UUID: 12345678-1234-abcd-ef00-01234567cffb
ncacn_np:127.0.0.1[\\pipe\\lsass]

UUID: 12345678-1234-abcd-ef00-01234567cffb
ncacn_np:127.0.0.1[\\PIPE\\protected_storage]

UUID: 12345678-1234-abcd-ef00-01234567cffb
ncacn_ip_tcp:127.0.0.1[49155]

UUID: 12345678-1234-abcd-ef00-01234567cffb
ncacn_http:127.0.0.1[49156]

Total endpoints found: 50



==== End of RPC Endpoint Mapper query response ====
portqry.exe -n 127.0.0.1 -e 135 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 127.0.0.1 -e 389 -p BOTH ...


Querying target system called:

 127.0.0.1

Attempting to resolve IP address to a name...


IP address resolved to IME-DC2.XXX.com

querying...

TCP port 389 (ldap service): LISTENING

Using ephemeral source port
Sending LDAP query to TCP port 389...

LDAP query response:


currentdate: 10/03/2013 08:30:30 (unadjusted GMT)
subschemaSubentry: CN=Aggregate,CN=Schema,CN=Configuration,DC=XXX,DC=com
dsServiceName: CN=NTDS Settings,CN=IME-DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=XXX,DC=com
namingContexts: DC=XXX,DC=com
defaultNamingContext: DC=XXX,DC=com
schemaNamingContext: CN=Schema,CN=Configuration,DC=XXX,DC=com
configurationNamingContext: CN=Configuration,DC=XXX,DC=com
rootDomainNamingContext: DC=XXX,DC=com
supportedControl: 1.2.840.113556.1.4.319
supportedLDAPVersion: 3
supportedLDAPPolicies: MaxPoolThreads
highestCommittedUSN: 18037146
supportedSASLMechanisms: GSSAPI
dnsHostName: IME-DC2.XXX.com
ldapServiceName: XXX.com:ime-dc2$@XXX.COM
serverName: CN=IME-DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=XXX,DC=com
supportedCapabilities: 1.2.840.113556.1.4.800
isSynchronized: TRUE
isGlobalCatalogReady: TRUE
domainFunctionality: 3
forestFunctionality: 3
domainControllerFunctionality: 3


======== End of LDAP query response ========

UDP port 389 (unknown service): NOT LISTENING
portqry.exe -n 127.0.0.1 -e 389 -p BOTH exits with return code 0x00000001.
=============================================

 Starting portqry.exe -n 127.0.0.1 -e 636 -p TCP ...


Querying target system called:

 127.0.0.1

Attempting to resolve IP address to a name...


IP address resolved to IME-DC2.XXX.com

querying...

TCP port 636 (ldaps service): LISTENING
portqry.exe -n 127.0.0.1 -e 636 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 127.0.0.1 -e 3268 -p TCP ...


Querying target system called:

 127.0.0.1

Attempting to resolve IP address to a name...


IP address resolved to IME-DC2.XXX.com

querying...

TCP port 3268 (msft-gc service): LISTENING

Using ephemeral source port
Sending LDAP query to TCP port 3268...

LDAP query response:


currentdate: 10/03/2013 08:30:30 (unadjusted GMT)
subschemaSubentry: CN=Aggregate,CN=Schema,CN=Configuration,DC=XXX,DC=com
dsServiceName: CN=NTDS Settings,CN=IME-DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=XXX,DC=com
namingContexts: DC=XXX,DC=com
defaultNamingContext: DC=XXX,DC=com
schemaNamingContext: CN=Schema,CN=Configuration,DC=XXX,DC=com
configurationNamingContext: CN=Configuration,DC=XXX,DC=com
rootDomainNamingContext: DC=XXX,DC=com
supportedControl: 1.2.840.113556.1.4.319
supportedLDAPVersion: 3
supportedLDAPPolicies: MaxPoolThreads
highestCommittedUSN: 18037146
supportedSASLMechanisms: GSSAPI
dnsHostName: IME-DC2.XXX.com
ldapServiceName: XXX.com:ime-dc2$@XXX.COM
serverName: CN=IME-DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=XXX,DC=com
supportedCapabilities: 1.2.840.113556.1.4.800
isSynchronized: TRUE
isGlobalCatalogReady: TRUE
domainFunctionality: 3
forestFunctionality: 3
domainControllerFunctionality: 3


======== End of LDAP query response ========
portqry.exe -n 127.0.0.1 -e 3268 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 127.0.0.1 -e 3269 -p TCP ...


Querying target system called:

 127.0.0.1

Attempting to resolve IP address to a name...


IP address resolved to IME-DC2.XXX.com

querying...

TCP port 3269 (msft-gc-ssl service): LISTENING
portqry.exe -n 127.0.0.1 -e 3269 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 127.0.0.1 -e 53 -p BOTH ...


Querying target system called:

 127.0.0.1

Attempting to resolve IP address to a name...


IP address resolved to IME-DC2.XXX.com

querying...

TCP port 53 (domain service): LISTENING

UDP port 53 (domain service): LISTENING
portqry.exe -n 127.0.0.1 -e 53 -p BOTH exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 127.0.0.1 -e 88 -p BOTH ...


Querying target system called:

 127.0.0.1

Attempting to resolve IP address to a name...


IP address resolved to IME-DC2.XXX.com

querying...

TCP port 88 (kerberos service): LISTENING

UDP port 88 (kerberos service): NOT LISTENING
portqry.exe -n 127.0.0.1 -e 88 -p BOTH exits with return code 0x00000001.
=============================================

 Starting portqry.exe -n 127.0.0.1 -e 445 -p TCP ...


Querying target system called:

 127.0.0.1

Attempting to resolve IP address to a name...


IP address resolved to IME-DC2.XXX.com

querying...

TCP port 445 (microsoft-ds service): LISTENING
portqry.exe -n 127.0.0.1 -e 445 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 127.0.0.1 -e 137 -p UDP ...


Querying target system called:

 127.0.0.1

Attempting to resolve IP address to a name...


IP address resolved to IME-DC2.XXX.com

querying...

UDP port 137 (netbios-ns service): NOT LISTENING
portqry.exe -n 127.0.0.1 -e 137 -p UDP exits with return code 0x00000001.
=============================================

 Starting portqry.exe -n 127.0.0.1 -e 138 -p UDP ...


Querying target system called:

 127.0.0.1

Attempting to resolve IP address to a name...


IP address resolved to IME-DC2.XXX.com

querying...

UDP port 138 (netbios-dgm service): NOT LISTENING
portqry.exe -n 127.0.0.1 -e 138 -p UDP exits with return code 0x00000001.
=============================================

 Starting portqry.exe -n 127.0.0.1 -e 139 -p TCP ...


Querying target system called:

 127.0.0.1

Attempting to resolve IP address to a name...


IP address resolved to IME-DC2.XXX.com

querying...

TCP port 139 (netbios-ssn service): NOT LISTENING
portqry.exe -n 127.0.0.1 -e 139 -p TCP exits with return code 0x00000001.
=============================================

 Starting portqry.exe -n 127.0.0.1 -e 42 -p TCP ...


Querying target system called:

 127.0.0.1

Attempting to resolve IP address to a name...


IP address resolved to IME-DC2.XXX.com

querying...

TCP port 42 (nameserver service): NOT LISTENING
portqry.exe -n 127.0.0.1 -e 42 -p TCP exits with return code 0x00000001.











I've totally disconnected it from the CISCO switch and I am running them using a dumb 8 port switch to prevent any sort of errors originating from the switch (not that there is any)

The whole test takes a long time and below are the results:





=============================================

 Starting portqry.exe -n 10.10.10.9 -e 135 -p TCP ...


Querying target system called:

 10.10.10.9

Attempting to resolve IP address to a name...


IP address resolved to ime-dc2.XXX.com

querying...

TCP port 135 (epmap service): LISTENING

Using ephemeral source port
Querying Endpoint Mapper Database...
Server's response:

UUID: 12345678-1234-abcd-ef00-01234567cffb
ncacn_ip_tcp:10.10.10.9[49157]

UUID: 2f5f6521-cb55-1059-b446-00df0bce31db Unimodem LRPC Endpoint
ncacn_np:10.10.10.9[\\pipe\\tapsrv]

UUID: 897e2e5f-93f3-4376-9c9c-fd2277495c27 Frs2 Service
ncacn_ip_tcp:10.10.10.9[5722]

UUID: 367abb81-9844-35f1-ad32-98f038001003
ncacn_ip_tcp:10.10.10.9[53392]

UUID: 50abc2a4-574d-40b3-9d66-ee4fd5fba076
ncacn_ip_tcp:10.10.10.9[53391]

UUID: 12345678-1234-abcd-ef00-0123456789ab IPSec Policy agent endpoint
ncacn_ip_tcp:10.10.10.9[53075]

UUID: 6b5bdd1e-528c-422c-af8c-a4079be4fe48 Remote Fw APIs
ncacn_ip_tcp:10.10.10.9[53075]

UUID: 3473dd4d-2e88-4006-9cba-22570909dd10 WinHttp Auto-Proxy Service
ncacn_np:10.10.10.9[\\PIPE\\wkssvc]

UUID: 3473dd4d-2e88-4006-9cba-22570909dd10 WinHttp Auto-Proxy Service
ncacn_np:10.10.10.9[\\PIPE\\W32TIME_ALT]

UUID: 1ff70682-0a51-30e8-076d-740be8cee98b
ncacn_np:10.10.10.9[\\PIPE\\atsvc]

UUID: 378e52b0-c0a9-11cf-822d-00aa0051e40f
ncacn_np:10.10.10.9[\\PIPE\\atsvc]

UUID: 86d35949-83c9-4044-b424-db363231fd0c
ncacn_np:10.10.10.9[\\PIPE\\atsvc]

UUID: 86d35949-83c9-4044-b424-db363231fd0c
ncacn_ip_tcp:10.10.10.9[49158]

UUID: a398e520-d59a-4bdd-aa7a-3c1e0303a511 IKE/Authip API
ncacn_np:10.10.10.9[\\PIPE\\atsvc]

UUID: a398e520-d59a-4bdd-aa7a-3c1e0303a511 IKE/Authip API
ncacn_ip_tcp:10.10.10.9[49158]

UUID: a398e520-d59a-4bdd-aa7a-3c1e0303a511 IKE/Authip API
ncacn_np:10.10.10.9[\\PIPE\\srvsvc]

UUID: 30b044a5-a225-43f0-b3a4-e060df91f9c1
ncacn_np:10.10.10.9[\\PIPE\\atsvc]

UUID: 30b044a5-a225-43f0-b3a4-e060df91f9c1
ncacn_ip_tcp:10.10.10.9[49158]

UUID: 30b044a5-a225-43f0-b3a4-e060df91f9c1
ncacn_np:10.10.10.9[\\PIPE\\srvsvc]

UUID: c9ac6db5-82b7-4e55-ae8a-e464ed7b4277 Impl friendly name
ncacn_np:10.10.10.9[\\PIPE\\atsvc]

UUID: c9ac6db5-82b7-4e55-ae8a-e464ed7b4277 Impl friendly name
ncacn_ip_tcp:10.10.10.9[49158]

UUID: c9ac6db5-82b7-4e55-ae8a-e464ed7b4277 Impl friendly name
ncacn_np:10.10.10.9[\\PIPE\\srvsvc]

UUID: f6beaff7-1e19-4fbb-9f8f-b89e2018337c Event log TCPIP
ncacn_np:10.10.10.9[\\pipe\\eventlog]

UUID: f6beaff7-1e19-4fbb-9f8f-b89e2018337c Event log TCPIP
ncacn_ip_tcp:10.10.10.9[49153]

UUID: 3c4728c5-f0ab-448b-bda1-6ce01eb0a6d5 DHCP Client LRPC Endpoint
ncacn_np:10.10.10.9[\\pipe\\eventlog]

UUID: 3c4728c5-f0ab-448b-bda1-6ce01eb0a6d5 DHCP Client LRPC Endpoint
ncacn_ip_tcp:10.10.10.9[49153]

UUID: 3c4728c5-f0ab-448b-bda1-6ce01eb0a6d6 DHCPv6 Client LRPC Endpoint
ncacn_np:10.10.10.9[\\pipe\\eventlog]

UUID: 3c4728c5-f0ab-448b-bda1-6ce01eb0a6d6 DHCPv6 Client LRPC Endpoint
ncacn_ip_tcp:10.10.10.9[49153]

UUID: 76f226c3-ec14-4325-8a99-6a46348418af
ncacn_np:10.10.10.9[\\PIPE\\InitShutdown]

UUID: d95afe70-a6d5-4259-822e-2c84da1ddb0d
ncacn_np:10.10.10.9[\\PIPE\\InitShutdown]

UUID: d95afe70-a6d5-4259-822e-2c84da1ddb0d
ncacn_ip_tcp:10.10.10.9[49152]

UUID: c9ac6db5-82b7-4e55-ae8a-e464ed7b4277 Impl friendly name
ncacn_np:10.10.10.9[\\pipe\\lsass]

UUID: c9ac6db5-82b7-4e55-ae8a-e464ed7b4277 Impl friendly name
ncacn_np:10.10.10.9[\\PIPE\\protected_storage]

UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_np:10.10.10.9[\\pipe\\lsass]

UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_np:10.10.10.9[\\PIPE\\protected_storage]

UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_ip_tcp:10.10.10.9[49155]

UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_http:10.10.10.9[49156]

UUID: 12345778-1234-abcd-ef00-0123456789ab
ncacn_np:10.10.10.9[\\pipe\\lsass]

UUID: 12345778-1234-abcd-ef00-0123456789ab
ncacn_np:10.10.10.9[\\PIPE\\protected_storage]

UUID: 12345778-1234-abcd-ef00-0123456789ab
ncacn_ip_tcp:10.10.10.9[49155]

UUID: 12345778-1234-abcd-ef00-0123456789ab
ncacn_http:10.10.10.9[49156]

UUID: 12345778-1234-abcd-ef00-0123456789ac
ncacn_np:10.10.10.9[\\pipe\\lsass]

UUID: 12345778-1234-abcd-ef00-0123456789ac
ncacn_np:10.10.10.9[\\PIPE\\protected_storage]

UUID: 12345778-1234-abcd-ef00-0123456789ac
ncacn_ip_tcp:10.10.10.9[49155]

UUID: 12345778-1234-abcd-ef00-0123456789ac
ncacn_http:10.10.10.9[49156]

UUID: 12345778-1234-abcd-ef00-0123456789ac
ncacn_ip_tcp:10.10.10.9[49157]

UUID: 12345678-1234-abcd-ef00-01234567cffb
ncacn_np:10.10.10.9[\\pipe\\lsass]

UUID: 12345678-1234-abcd-ef00-01234567cffb
ncacn_np:10.10.10.9[\\PIPE\\protected_storage]

UUID: 12345678-1234-abcd-ef00-01234567cffb
ncacn_ip_tcp:10.10.10.9[49155]

UUID: 12345678-1234-abcd-ef00-01234567cffb
ncacn_http:10.10.10.9[49156]

Total endpoints found: 50



==== End of RPC Endpoint Mapper query response ====
portqry.exe -n 10.10.10.9 -e 135 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 10.10.10.9 -e 389 -p BOTH ...


Querying target system called:

 10.10.10.9

Attempting to resolve IP address to a name...


IP address resolved to ime-dc2.XXX.com

querying...

TCP port 389 (ldap service): LISTENING

Using ephemeral source port
Sending LDAP query to TCP port 389...

LDAP query to port 389 failed
Server did not respond to LDAP query


Server did not respond to LDAP query


UDP port 389 (unknown service): LISTENING or FILTERED

Using ephemeral source port
Sending LDAP query to UDP port 389...

LDAP query to port 389 failed
Server did not respond to LDAP query

portqry.exe -n 10.10.10.9 -e 389 -p BOTH exits with return code 0x00000001.
=============================================

 Starting portqry.exe -n 10.10.10.9 -e 636 -p TCP ...


Querying target system called:

 10.10.10.9

Attempting to resolve IP address to a name...


IP address resolved to ime-dc2.XXX.com

querying...

TCP port 636 (ldaps service): LISTENING
portqry.exe -n 10.10.10.9 -e 636 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 10.10.10.9 -e 3268 -p TCP ...


Querying target system called:

 10.10.10.9

Attempting to resolve IP address to a name...


IP address resolved to ime-dc2.XXX.com

querying...

TCP port 3268 (msft-gc service): LISTENING

Using ephemeral source port
Sending LDAP query to TCP port 3268...

LDAP query to port 3268 failed
Server did not respond to LDAP query


Server did not respond to LDAP query

portqry.exe -n 10.10.10.9 -e 3268 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 10.10.10.9 -e 3269 -p TCP ...


Querying target system called:

 10.10.10.9

Attempting to resolve IP address to a name...


IP address resolved to ime-dc2.XXX.com

querying...

TCP port 3269 (msft-gc-ssl service): LISTENING
portqry.exe -n 10.10.10.9 -e 3269 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 10.10.10.9 -e 53 -p BOTH ...


Querying target system called:

 10.10.10.9

Attempting to resolve IP address to a name...


IP address resolved to ime-dc2.XXX.com

querying...

TCP port 53 (domain service): LISTENING

UDP port 53 (domain service): LISTENING
portqry.exe -n 10.10.10.9 -e 53 -p BOTH exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 10.10.10.9 -e 88 -p BOTH ...


Querying target system called:

 10.10.10.9

Attempting to resolve IP address to a name...


IP address resolved to ime-dc2.XXX.com

querying...

TCP port 88 (kerberos service): LISTENING

UDP port 88 (kerberos service): LISTENING or FILTERED
portqry.exe -n 10.10.10.9 -e 88 -p BOTH exits with return code 0x00000002.
=============================================

 Starting portqry.exe -n 10.10.10.9 -e 445 -p TCP ...


Querying target system called:

 10.10.10.9

Attempting to resolve IP address to a name...


IP address resolved to ime-dc2.XXX.com

querying...

TCP port 445 (microsoft-ds service): LISTENING
portqry.exe -n 10.10.10.9 -e 445 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 10.10.10.9 -e 137 -p UDP ...


Querying target system called:

 10.10.10.9

Attempting to resolve IP address to a name...


IP address resolved to ime-dc2.XXX.com

querying...

UDP port 137 (netbios-ns service): LISTENING or FILTERED

Using ephemeral source port
Attempting NETBIOS adapter status query to UDP port 137...

Server's response: MAC address 0023ae9ebb0e
UDP port: LISTENING
portqry.exe -n 10.10.10.9 -e 137 -p UDP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 10.10.10.9 -e 138 -p UDP ...


Querying target system called:

 10.10.10.9

Attempting to resolve IP address to a name...


IP address resolved to ime-dc2.XXX.com

querying...

UDP port 138 (netbios-dgm service): LISTENING or FILTERED
portqry.exe -n 10.10.10.9 -e 138 -p UDP exits with return code 0x00000002.
=============================================

 Starting portqry.exe -n 10.10.10.9 -e 139 -p TCP ...


Querying target system called:

 10.10.10.9

Attempting to resolve IP address to a name...


IP address resolved to ime-dc2.XXX.com

querying...

TCP port 139 (netbios-ssn service): LISTENING
portqry.exe -n 10.10.10.9 -e 139 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 10.10.10.9 -e 42 -p TCP ...


Querying target system called:

 10.10.10.9

Attempting to resolve IP address to a name...


IP address resolved to ime-dc2.XXX.com

querying...

TCP port 42 (nameserver service): NOT LISTENING
portqry.exe -n 10.10.10.9 -e 42 -p TCP exits with return code 0x00000001.




Very sorry about the long post. Thanks.
ASKER CERTIFIED SOLUTION
Avatar of Jaihunt
Jaihunt
Flag of India image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of dqnet

ASKER

Ok I read those articles and a very interesting point was in the second one way down.
"Network Drives"

Now we ran a PSP update on our Host Server that runs the VM's last night.
HP Proliant DL380 G5 and part of the update package was to update the network drivers.

I have compared the drivers with another server of the same spec in which we haven't run the PSP on yet and those drivers are obviously older.

Could this be the issue?
Should I downgrade the drivers to the old ones?
Nothing has changed AT ALL except for the PSP upgrade.
They were working perfectly until then...
Avatar of dqnet

ASKER

The drivers were the issue! Removing those new 'updated' drivers and installing the older ones cleared all errors and replication started again.

The updated host drivers must have been causing network connectivity issues for its own virtual machines.

Thanks to all

@Jailhunt, thanks for those links!