Link to home
Start Free TrialLog in
Avatar of David Williamson
David WilliamsonFlag for United States of America

asked on

DFS and FRS-here comes the punishment

Getting Event log error that looks like this:

_____________________________________________________
The File Replication Service is having trouble enabling replication from SERVER2 to IRVINE for d:\datastore using the DNS name server2.wse.com. FRS will keep retrying.
 Following are some of the reasons you would see this warning.
 
 [1] FRS can not correctly resolve the DNS name server2.wse.com from this computer.
 [2] FRS is not running on server2.wse.com.
 [3] The topology information in the Active Directory for this replica has not yet replicated to all the Domain Controllers.
 
 This event log message will appear once per connection, After the problem is fixed you will see another event log message indicating that the connection has been established.
___________________________________________________

Any ideas?
Avatar of Netman66
Netman66
Flag of Canada image

Hi theamzngq,

Ok........

Let's start with the basics.  

1)  Is DNS representing both servers correctly?  (all Service records intact)
2)  Is the FRS service running on both DCs?
3)  Do you have sites configured?

Regards,

Netman66, MCSE, MCT
Microsoft MVP Team
Avatar of David Williamson

ASKER

Just a quick note:

We've got 3 DCs,
-server2, PDC
-server3
-irvine

At the moment, all are in the same site, same LAN

I created a text file in the sysvol share of server2 to test whether sysvol was replicating.  I checked for it in sysvol on irvine-not there.  I checked for it in sysvol on server3-not there.  So, I created a new text file, but this time on irvine.  I checked on server3-it was there.  I created one on server3, and it showed up on irvine.  However, neither of those two files showed up in sysvol on server2, and nothing I put on server2 showed up on either irvine or server3...whew

Make sense?  From this, it seems that server2 is not replicating, while irvine and server3 are.  And remember, I just ran adprep only minutes ago, after which I created the DFS root on server2, then created a root replica on irvine.
lemme check, was writing above post while you posted yours..;
theamzngq,
> server2.wse.com

Is this what I think it is?  You are using .com for internal AD?

This could be a resolution problem - it could simply be looking to the Internet for your server, not local.

You might try adding each server's FQDN to the HOSTS file on each DC - as a test.

1) I can ping both machines from both machines using FQDN.  Should I check something else?
2) FRS is running on both machines according to Services MMC
3) I THINK I have sites configured...I may need some more specifics to verify that.
theamzngq,

We're stepping on each other - I'll wait for you, let me know when we're synch'd!

ok we're good.  Yes, you are correct about wse.com.  I didn't know that could be a problem.  This was all setup before I arrived 1 year ago, and I've learned about domains and AD during that time.  I created a domain-valid lmhosts file just last week and it appears to have worked properly, but I only added server2 and a regular machine.  Here it is:

192.168.33.201   SERVER2   #PRE #DOM:WSE
192.168.33.201   "WSE            \0x1b"   #PRE
192.168.111.3    IRVINE01  #PRE #DOM:WSE

Is there anything special I need to do regarding adding a DC line, like the one for server2?
theamzngq,

What's the special character is WSE?  Not understanding that entry...


WSE would refer to a name... rename this file temporarily.  If you have a functioning DNS server then LMHOSTS isn't of value.

Add this to HOSTS on all servers. (replace x for your IPs)

192.168.33.201  server2.wse.com
192.168.33.x    server3.wse.com
192.168.33.x    irvine.wse.com

theamzngq,

Regarding my checks...

1) In DNS, make certain the all the SRV records exist for each server.  Make sure that Server2 is showing up.  Also, make sure that the NIC on Server2 is set to register in DNS.

2) FRS should be running on all DCs (sorry I thought you had 2).

3) In AD Sites and Services - did you create manual sites?  If so, does each site have an associated subnet?  

Expand the site, then expand the server2.  Right-click on NTDS and select properties.  You should see <Auto generated> site links to each of the other DCs.  Are they there?
I made that lmhosts file by following steps on http://support.microsoft.com/default.aspx?scid=kb;EN-US;180094

I have entered the info as you suggest in a HOSTS file on all DCs (in the SYSTEM32\DRIVERS\ETC folder, I assume).

Your checks:
1) You mean like in this screenshot from server2? www.wrightcustomhome.com/dns01.jpg

2) Yes, FRS is running on all three

3) I simply used the Default-First-site.  They are all in that site now, but once the Irvine server actually goes to Irvine, I plan on creating a new Irvine site and a new subnet 192.168.111.x for it.

In Sites and Services on server2: www.wrightcustomhome.com/dns02.jpg
theamzngq,

Those entries look good.  Clear all you logs on server2. Right click on the site links from you screenshot and select Replicate Now.  Do this for each link.

Let's see what the logs have to say.
I cleared all logs in event viewer (hope that was the right place).

I clicked Replicate Now on all site links and for each server and each time got a pop up that said: 'Active Directory has replicated the connections.'

Event viewer still shows no events in any log.
theamzngq,

Are your test files replicating now?
not to or from server2.  Only between server3 and irvine still.
Ok, do you have the Support tools installed on Server2?  Install them, if not.

Run Replmon or repadmin (depending on whether you're a GUI guy or command line jukkie!).  Let's see what the replication status is showing for errors.

what switch should I use in repadmin?
I ran /showreps.  It came up with an entry that mentioned a server that crashed and that I removed using ASDIedit - IrvineServer.  Here is what that command showed:

C:\Documents and Settings\Administrator>repadmin /showreps
Default-First-Site-Name\SERVER2
DSA Options : IS_GC
objectGuid  : 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
invocationID: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1

==== INBOUND NEIGHBORS ======================================

CN=Schema,CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\IRVINESERVER
DEL:5e005e93-4de4-40bc-b3d2-de78ea0a5172 (deleted DSA) via RPC
        objectGuid: 839d3c9e-072e-4155-bcdf-230e716ee5e6
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261
        Last attempt @ 2004-03-24 20:01.49 was successful.
    Default-First-Site-Name\SERVER3 via RPC
        objectGuid: 000282ea-4bba-4049-b3e8-fc70bb38c6f3
        Last attempt @ 2004-03-24 20:01.53 was successful.

CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\IRVINESERVER
DEL:5e005e93-4de4-40bc-b3d2-de78ea0a5172 (deleted DSA) via RPC
        objectGuid: 839d3c9e-072e-4155-bcdf-230e716ee5e6
    Default-First-Site-Name\SERVER3 via RPC
        objectGuid: 000282ea-4bba-4049-b3e8-fc70bb38c6f3
        Last attempt @ 2004-03-24 20:07.17 was successful.
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261
        Last attempt @ 2004-03-24 20:19.29 was successful.

DC=wse,DC=com
    Default-First-Site-Name\IRVINESERVER
DEL:5e005e93-4de4-40bc-b3d2-de78ea0a5172 (deleted DSA) via RPC
        objectGuid: 839d3c9e-072e-4155-bcdf-230e716ee5e6
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261
        Last attempt @ 2004-03-24 20:01.49 was successful.
    Default-First-Site-Name\SERVER3 via RPC
        objectGuid: 000282ea-4bba-4049-b3e8-fc70bb38c6f3
        Last attempt @ 2004-03-24 20:13.37 was successful.

==== OUTBOUND NEIGHBORS FOR CHANGE NOTIFICATIONS ============

CN=Schema,CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\SERVER3 via RPC
        objectGuid: 000282ea-4bba-4049-b3e8-fc70bb38c6f3
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261

CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\SERVER3 via RPC
        objectGuid: 000282ea-4bba-4049-b3e8-fc70bb38c6f3
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261

DC=wse,DC=com
    Default-First-Site-Name\SERVER3 via RPC
        objectGuid: 000282ea-4bba-4049-b3e8-fc70bb38c6f3
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261

Does that mean anything, I mean the entries referring to IrvineServer?
theamzngq,

Ok, before we get off on a tangent here, let's look at a few easy things.

Type net share at a command prompt on server2 - is SYSVOL listed?

Next, type \\server2 in the run box.  Check the permissions on the share and folders within the share - make sure they match the other servers.

Advise on this.
net share from server2 shows, among other things, SYSVOL.

typing \\server2 in the run box produces a windows explorer window.  I right-click on SYSVOL, go to Security tab.  Permissions on SYSVOL on server2 are Administrators-Full Control, Everyone-everything BUT Full Control (that seems weird).  

Sysvol share on server3 has very different permissions:
Administrators-Full Control;
Authenticated Users: read&exec,list folder contents, and read;
creator owner: nothing;
server operators: same as auth users;
system: Full Control

Irvine, same as server3

Hmmm....seems like a pattern emerging...
theamzngq,

Yes.

You'll need to clean this up.

http://support.microsoft.com/default.aspx?scid=kb;EN-US;216498

Fix your permissions on the SYSVOL also!

....and one more for good measure:

http://support.microsoft.com/default.aspx?scid=kb;en-us;312862&Product=win2000

I have to get some sleep now - it's 1am here.  I will check on you tomorrow.  Post any further info you have and I'll get back to you then.

Cheers.
thx, you're the best...
I'm back!!!

Any progress?
hehe, wow, 4:50 am PST... you must be back east?

Well, yes, I think.  Let me check some stuff out and post in a minute or two...
Yes...on the East coast (New Brunswick)
I did a lot of things last night, it seems, and I can't remember the exact sequence, but one thing I did do was fix the permissions on sysvol on server2 to match the other two.  Now my test file in sysvol appears in all of them.  Renaming it causes the change to replicate immediately also to all three DCs.  As far as the links you posted, I haven't gone through those yet, with the exception of http://support.microsoft.com/default.aspx?scid=kb;EN-US;216498.  I had already gone through that procedure yesterday before we started the thread.  All the operations seemed to work without issue.

However, I'm still getting 13508 in the event viewer regarding DFS.  Just this morning Irvine had this in the event viewer:
_____________________________________________________________________
The File Replication Service is having trouble enabling replication from SERVER2 to IRVINE for d:\datastore using the DNS name server2.wse.com. FRS will keep retrying.
 Following are some of the reasons you would see this warning.
 
 [1] FRS can not correctly resolve the DNS name server2.wse.com from this computer.
 [2] FRS is not running on server2.wse.com.
 [3] The topology information in the Active Directory for this replica has not yet replicated to all the Domain Controllers.
 
 This event log message will appear once per connection, After the problem is fixed you will see another event log message indicating that the connection has been established.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
__________________________________________________________________________________

So, sysvol works, but not FRS
OK.  Cleaning up the AD and removing the bad replica might solve this.

Give it some time.  Most of the time the 13508 is followed by a 13509 letting you know that normal replication is occurring.

Try this from Server2:

NTFRSUTL VERSION <FQDN of remote DC name>


Post results.
The part from server2's run of repadmin /showreps where it says
"Default-First-Site-Name\IRVINESERVER DEL:5e005e93-4de4-40bc-b3d2-de78ea0a5172 (deleted DSA) via RPC  objectGuid: 839d3c9e-072e-4155-bcdf-230e716ee5e6"
It seems to indicate that it is deleted (deleted DSA), because like I mentioned above, I went through the steps on the link you posted already.  I'll go through them again right now; is there something/somewhere else I need to check in order to be sure its totally cleaned out?

Here's a repadmin /showreps I just ran on server3:

C:\Documents and Settings\Administrator.WSE>repadmin /showreps
Default-First-Site-Name\SERVER3
DSA Options : (none)
objectGuid  : 000282ea-4bba-4049-b3e8-fc70bb38c6f3
invocationID: 53091175-ed92-4f73-9255-0a2d00ac5922

==== INBOUND NEIGHBORS ======================================

CN=Schema,CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261
        Last attempt @ 2004-03-25 06:53.09 failed, result 1722:
            The RPC server is unavailable.
        Last success @ 2004-03-24 21:52.02.
        9 consecutive failure(s).
    Default-First-Site-Name\SERVER2 via RPC
        objectGuid: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
        Last attempt @ 2004-03-25 06:53.09 was successful.

CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261
        Last attempt @ 2004-03-25 07:31.47 was successful.
    Default-First-Site-Name\SERVER2 via RPC
        objectGuid: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
        Last attempt @ 2004-03-25 07:36.44 was successful.

DC=wse,DC=com
    Default-First-Site-Name\SERVER2 via RPC
        objectGuid: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
        Last attempt @ 2004-03-25 07:48.54 was successful.
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261
        Last attempt @ 2004-03-25 07:49.39 was successful.

==== OUTBOUND NEIGHBORS FOR CHANGE NOTIFICATIONS ============

CN=Schema,CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\SERVER2 via RPC
        objectGuid: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261

CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\SERVER2 via RPC
        objectGuid: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261

DC=wse,DC=com
    Default-First-Site-Name\SERVER2 via RPC
        objectGuid: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261
will do ntfrsutil and post...
Also, check Server2 to make sure that Authenticated Users is in the policy, "Access this computer from the Network"

Another ALSO!

Check to make sure File and Print sharing is enabled on all your servers.

Do a Netdiag /v from server3

Also, can you ping Irvine's FQDN from Server3?

Oops... do Netdiag from Irvine.
Output Netdiag /v to text and put it up on your site.

First task, ntfrsutl results from server2:

C:\Documents and Settings\Administrator>ntfrsutl version server2.wse.com
NtFrsApi Version Information
   NtFrsApi Major      : 0
   NtFrsApi Minor      : 0
   NtFrsApi Compiled on: May  6 2003 14:14:57
NtFrs Version Information
   NtFrs Major        : 0
   NtFrs Minor        : 0
   NtFrs Compiled on  : May  6 2003 14:15:26
   Latest changes:
   WIN2K-SP4
    + QFE #2 - force replication
OS Version 5.0 (2195) -
SP (4.0) SM: 0x0000  PT: 0x02
Processor:  INTEL Level: 0x0006  Revision: 0x0b01  Processor num/mask: 2/0000000
3
1) GPO had the Everyone group assigned to 'access this computer from the network'.  I added Auth Users.

2) File and Printer sharing is found on the NIC properties of each server.

3) I can ping Irvine's FQDN from server3

4) Netdiag /v output from Irvine: www.wrightcustomhome.com/netdiag_Irvine.log

5) Netdiag /v output from Server3: www.wrightcustomhome.com/netdiag_server3.log

6) Netdiag /v output from Server2: www.wrightcustomhome.com/netdiag_server2.log

Server2, by the way, is the main file server.  I have never used netdiag and have no idea what it does!  Looks like some cool stuff, though.
possible a related question: server2, being the PDC, has its own IP as Primary DNS, and then forwarders are configured in the DNS server.  What should the other DCs have as their Primary and Secondary DNS?  And should some of the forwards on server2 be the IPs of server3 and Irvine?
Primary should be the main DNS server.  No secondary.

Make sure DNS is AD integrated (both Forward and Reverse zones) on each DNS server you have.


Looking at the logs now.
thx.  Ok, that's how I believe I have it set on all DCs, I think.   But then what happens if the primary DNS server goes down?  Backup DNS for internet name resolution or machine name resolution?
maybe not...looking at the logs too.
You can add secondaries as the opposite peer if you want.  Probably a good idea.

I know you're looking at the logs at the moment (thanks).  I have been as well, and on server2's log, it shows several entries where it is trying to query the Secondary DNS (which I know now I need to remove) which is our ISPs DNS server.  Here's a chunk:
__________________________________________________________________________


PASS - All the DNS entries for DC are registered on DNS server '192.168.33.201' and other DCs also have some of the names registered.
Check the DNS registration for DCs entries on DNS server '207.69.188.185'
Query for DC DNS entry _ldap._tcp.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry _ldap._tcp.pdc._msdcs.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry _ldap._tcp.gc._msdcs.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry _ldap._tcp.dc._msdcs.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry _kerberos._udp.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry _ldap._tcp.Default-First-Site-Name._sites.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry _gc._tcp.Default-First-Site-Name._sites.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1._msdcs.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry _gc._tcp.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry _kpasswd._tcp.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry _kpasswd._udp.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry _kerberos._tcp.dc._msdcs.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry _kerberos._tcp.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry _ldap._tcp.Default-First-Site-Name._sites.gc._msdcs.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry _kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry _kerberos._tcp.Default-First-Site-Name._sites.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
Query for DC DNS entry _ldap._tcp.c673d087-cb98-4097-9fee-5f4289bc99db.domains._msdcs.wse.com. on DNS server 207.69.188.185 failed.
DNS Error code: DNS_ERROR_RCODE_NAME_ERROR (Name does not exist on DNS server)
The Record is different on DNS server '207.69.188.185'.
_______________________________________________________________________________________________
Whoa....

Ok...

Try this:  Ping <fqdn of dsa>._msdcs.forestroot  <= where FQDN is the GUID of the domain {C673D087-CB98-4097-9FEE-5F4289BC99DB}

Now, make sure that your main DNS server is not trying to register it's external adapter in DNS (which would be the ISP).  Make sure there are no references to the ISP's DNS anywhere on the internal LAN except in the Forwarders tab on the main DNS server.


As I suspected yesterday, it's looking outside your own DNS for records - a NO, NO.

This looks like it boils down to DNS now.

Which is your main DNS server?  The one that Forwards to the ISP?



Well, when you say anywhere on the LAN, do you mean workstations as well?  Very early on, I had all the workstations set with Primary DNS as 192.168.33.201 only.  However, workstations were having trouble getting internet pages to load (ie, name resolution).  So, I added 207.69.188.185 as a secondary on all workstations.  Then the internet started working.  Recently, I have set our firewall to allow outgoing DNS from server2 and server3's IPs only, denying the rest.  The workstations seem to be able to get internet pages just fine even though I am denying their external DNS requests.

So what should I do?

Ping results from server2:

C:\Documents and Settings\Administrator>ping {C673D087-CB98-4097-9FEE-5F4289BC99
DB}._msdcs.forestroot
Unknown host {C673D087-CB98-4097-9FEE-5F4289BC99DB}._msdcs.forestroot

Did I do it right?
server2 is the main DNS server, as far as I understand.  It is set to forward to our ISP's DNS, yes.
OK here is what to do to fix the basics.

Server2
Internal NIC - DNS itself.
External NIC - ISP.
Forwarding - ISP.

Irvine - DNS - primary Server2, secondary Server3
Forwarding - Server2

Server3 - DNS - primary Server2, secondary Irvine.
Forwarding - Server2

Primary site All clients - DNS - primary Server2, seconday Server3.


Once you move Irvine, you will create sites and subnets and move the server into the right subnet.  You must the reconfigure DNS so that Irvine forwards to the ISP there.  All clients at Irvine use Irvine only for DNS.  Remove Irvine from Server3's secondary DNS.  Clear out ALL Irvine's records with the old IP since this will change.



Now for that ping - wrong syntax before.

ping C673D087-CB98-4097-9FEE-5F4289BC99DB._msdcs.wse.com

this too:

ping 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1._msdcs.wse.com

ping results for 1st one on server2:
___________
C:\Documents and Settings\Administrator>ping C673D087-CB98-4097-9FEE-5F4289BC99D
B._msdcs.wse.com
Unknown host C673D087-CB98-4097-9FEE-5F4289BC99DB._msdcs.wse.com.
________________

ping results for second request:
______________________________

C:\Documents and Settings\Administrator>ping 6233f4eb-40c9-47a7-9096-2f1e88d0c8b
1._msdcs.wse.com

Pinging server2.wse.com [192.168.33.201] with 32 bytes of data:

Reply from 192.168.33.201: bytes=32 time<10ms TTL=128
Reply from 192.168.33.201: bytes=32 time<10ms TTL=128
Reply from 192.168.33.201: bytes=32 time<10ms TTL=128
Reply from 192.168.33.201: bytes=32 time<10ms TTL=128

Ping statistics for 192.168.33.201:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum =  0ms, Average =  0ms
________________________________________________________

Ok, regarding fixing the basics:

1)  There is only one NIC in server2; I have set it as follows: www.wrightcustomhome.com/server2_nic.jpg

2)  Irvine: www.wrightcustomhome.com/irvine_NIC_and_DNS_forwarding.jpg

3) Server3: www.wrightcustomhome.com/server3_NIC_and_DNS_forwarding.jpg

To quote one of my favorite songs by Jamiroqui, "where do we go from here?"
Good!

Now......

Go through DNS on Server2 and remove any entries (A) that do not match what the IP and host really are.

Remove any stale (no longer used) entries.

Restart the Netlogon service on all DCs.  Check for and 13509 Events on each DC then clear the events on all DCs.

Let's see what happens.

should I remove absolutely everything that isn't right, not just server2 A entries?  If so, you'll have to give me a couple minutes...

and what about the ping command of the two you had me try that didn't work?  Is that a big deal?
Yes, all objects that are not correct.

Ping was good - the server responded correctly when pinging the object directly in AD.

I have done as you requested and restarted netlogon on all 3 DCs.  I haven't seen any 13509s yet; i'll keep an eye on it.
other things come to mind:

should I enable netbios over tcp/ip on all DCs?
It's not necessary, no.

Make sure that you are not trying to do zone transfers to the ISP.  Check this in DNS.

I'm still looking at the logs.

Do you see any 13508's yet?
where do I check for zone tranfers to ISP?  No 13508s yet...I assume I should be looking in the event viewer in FRS log?
It's in the same property dialogues as Forwarders.

Where did you see the last FRS errors?  That's what I'm looking for - if there are no further FRS errors then let's see where the data is at.

the last FRS errors happen at 7:30 am or so.  However, I haven't seen any 'connection restored' errors.

I found it right-clicking on the wse.com entry under forward lookup zones.  I set it to 'only to servers listed on the Name Servers tab', which are irvine, server2, server3.  It WAS set to 'to any server'.
Good stuff.

Let's see what happens now with replication.

It may take a little while.
There doesn't seem to be any data activity in the DFS root folders of irvine or server3 (which, if I didn't metion it, I added as an additional Root Replica to the server2 DFS root).  They should be replicating the stuff from server2's DFS root folder, right?

Should I restart FRS on all servers?
You'll also see this zone transfer setting on Reverse Lookup zones AND on all other DNS servers - so correct them too.

some of the reverse lookup zones don't have the 'allow zone tranfers' box checked.  Should I check it and set it to 'only to servers listedonthe Name Servers tab' or leave it alone?
You can set it for your servers if you like.

Since your DNS is AD integrated there likely is not benefit to zone transfers at all.



What do you think about restarting FRS on all servers?
You can, yes.

I would like some fresh Netdiag logs from server2.  Can you repost?

coming right up....
results of restarting FRS on all servers:

Server2: event ids 13552 & 13555 in FRS log
Server3: 13516, 13508
Irvine: 13516, 13508

working on the log...
Sorry, on server2, events 13552 & 13555 were followed by 13516
btw, the test files still replicate in the wse.com folder inside sysvol on all three servers....
Its working now.

13516 indicates it's in the correct state.
that's weird.  the times for all three of those errors were exactly the same.  Why would it come up with seemingly critical errors only in the same instant to be 'fixed'?
You should see some 13509's on Server3 and Irvine soon.

I've got my eye out for 'em, that's for sure...I just looked and there is a 13508 on both server3 and irvine from about 15 minutes ago or so.
No, not too weird.  Sometime the dependency services take a little longer to start causing some anomalies.  Sometimes it network latency.

Log for Server2 is clean.

Can you post the other 2 servers again?


you got it...
Just so that I learn something out of all this, what things are missing in server2's netdiag log that make it 'clean'?
All the errors!

Compare the two - no more error flags now and it's no longer going to your ISP for service record lookups.

Run the following command, changing servername for each test - post them for me, please!

dcdiag /s:server2 /v > dcd-server2.txt

If I cannot see anything further in those new logs I'm going to ask you to run a chkdsk /f on each controller - starting with Server2.  You'll need to do this when everyone leaves and nobody is logged in - since you will be required to restart the server to kick in chkdsk.  Please clear the logs before you start so we can see any errors easily.

I'm not sure if I'm doing that command right....I did it for server2, like you have above, and the resulting file is empty...
Shoot...

run this instead - sorry!

dcdiag /a /v /c > dcd-site.txt
do I still need to change something, or is that an all-inclusive commnd?
Perhaps output to c:\dcd-site.txt

/a means all DCs, /v means verbose, /c means all tests.

Run it from a command prompt on Server2.
just noticed the RPC Locator error for Irvine.  I checked, and the service on that machine was not running and was set to manual startup...I started it and changed the type to auto
there are two other service tests that irvine failed...I don't recognize the service names, though
Can you do a repadmin /showreps again?

C:\Documents and Settings\Administrator.WSE>repadmin /showreps
Default-First-Site-Name\SERVER3
DSA Options : (none)
objectGuid  : 000282ea-4bba-4049-b3e8-fc70bb38c6f3
invocationID: 53091175-ed92-4f73-9255-0a2d00ac5922

==== INBOUND NEIGHBORS ======================================

CN=Schema,CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\SERVER2 via RPC
        objectGuid: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
        Last attempt @ 2004-03-25 12:52.00 was successful.
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261
        Last attempt @ 2004-03-25 12:52.00 was successful.

CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261
        Last attempt @ 2004-03-25 12:52.00 was successful.
    Default-First-Site-Name\SERVER2 via RPC
        objectGuid: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
        Last attempt @ 2004-03-25 12:52.08 was successful.

DC=wse,DC=com
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261
        Last attempt @ 2004-03-25 12:52.00 was successful.
    Default-First-Site-Name\SERVER2 via RPC
        objectGuid: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
        Last attempt @ 2004-03-25 12:52.00 was successful.

==== OUTBOUND NEIGHBORS FOR CHANGE NOTIFICATIONS ============

CN=Schema,CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\SERVER2 via RPC
        objectGuid: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261

CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\SERVER2 via RPC
        objectGuid: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261

DC=wse,DC=com
    Default-First-Site-Name\SERVER2 via RPC
        objectGuid: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261
this one is from server2:

Default-First-Site-Name\SERVER2
DSA Options : IS_GC
objectGuid  : 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
invocationID: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1

==== INBOUND NEIGHBORS ======================================

CN=Schema,CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\SERVER3 via RPC
        objectGuid: 000282ea-4bba-4049-b3e8-fc70bb38c6f3
        Last attempt @ 2004-03-25 12:57.08 was successful.
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261
        Last attempt @ 2004-03-25 12:57.08 was successful.
    Default-First-Site-Name\IRVINESERVER
DEL:5e005e93-4de4-40bc-b3d2-de78ea0a5172 (deleted DSA) via RPC
        objectGuid: 839d3c9e-072e-4155-bcdf-230e716ee5e6

CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261
        Last attempt @ 2004-03-25 12:57.08 was successful.
    Default-First-Site-Name\SERVER3 via RPC
        objectGuid: 000282ea-4bba-4049-b3e8-fc70bb38c6f3
        Last attempt @ 2004-03-25 12:57.08 was successful.
    Default-First-Site-Name\IRVINESERVER
DEL:5e005e93-4de4-40bc-b3d2-de78ea0a5172 (deleted DSA) via RPC
        objectGuid: 839d3c9e-072e-4155-bcdf-230e716ee5e6

DC=wse,DC=com
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261
        Last attempt @ 2004-03-25 12:57.08 was successful.
    Default-First-Site-Name\IRVINESERVER
DEL:5e005e93-4de4-40bc-b3d2-de78ea0a5172 (deleted DSA) via RPC
        objectGuid: 839d3c9e-072e-4155-bcdf-230e716ee5e6
    Default-First-Site-Name\SERVER3 via RPC
        objectGuid: 000282ea-4bba-4049-b3e8-fc70bb38c6f3
        Last attempt @ 2004-03-25 12:57.33 was successful.

==== OUTBOUND NEIGHBORS FOR CHANGE NOTIFICATIONS ============

CN=Schema,CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\SERVER3 via RPC
        objectGuid: 000282ea-4bba-4049-b3e8-fc70bb38c6f3
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261

CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\SERVER3 via RPC
        objectGuid: 000282ea-4bba-4049-b3e8-fc70bb38c6f3
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261

DC=wse,DC=com
    Default-First-Site-Name\SERVER3 via RPC
        objectGuid: 000282ea-4bba-4049-b3e8-fc70bb38c6f3
    Default-First-Site-Name\IRVINE via RPC
        objectGuid: 51f814c3-f364-482a-8553-72a476a41261
those all seem good despite the (deleted DSA) entry for IrvineServer
This is what I want you to do tonight:

1)  Delete the CONTENTS only of the Forward and Reverse Lookup Zones.  Do Server3, then Irvine then Server2.  Do not restart.  Make sure any static entries you might have made are recorded for later.

2) Stop the Netlogon and Replication services on each DC.  Delete the file "Ntfrs.jdb" from the <windir>\Ntfrs\Jet directory.  Delete the file "Edb.txt" from the file <windir>\Ntfrs\Jet\Sys.  Delete the file "Edb.txt",  from the file "Res1.txt" and the file "Res2.txt" from the <windir>\Ntfrs\Jet\Log directory.

3)  Clear out the Event logs.

4)  From the Run box starting with Server2, type CHKDSK /F and restart the server.  Move on to the other servers.

4)  When Server 2 comes up check DNS to make sure that the service records are registered (see you screenshot from earlier to know what to look for).  Check for each of the other serversin DNS in turn as they come up.

5)  You should now have a clean slate to start with again.  All services should be functional.


Should have mentioned - everyone MUST be off the servers before you start the above.
meaning no files opened, no email connections, no nothing, right?
# 2 didn't come out the way I was thinking...

Use this # 2:

Stop the Netlogon and Replication Services.  Delete the following files:

%systemroot%\ntfrs\jet\Ntfrs.jdb
%systemroot%\ntfrs\jet\Sys\Edb.chk
%systemroot%\ntfrs\jet\log\edb.log
%systemroot%\ntfrs\jet\log\res1.log
%systemroot%\ntfrs\jet\log\res2.log
Yes, no open connections - it'll only hurt them! ;o)

I was just going to ask about that...
any particular drive with the chkdsk?
and the service records in step 4 you are referring to are these: www.wrightcustomhome.com/dns01.jpg, correct?

Just making sure....

Hey, I've got to take a sec and thank you for all you help with this!  So much of this is beyond me; I would have had no idea where to look!  Here's to hoping this things is working by tomorrow afternoon, cause that is when its heading for Irvine.
Open AD Sites and Services on Server2.
Expand Sites.
Expand Default-First-Site (unless you renamed it)
Expand Servers.
Expand Server2
Select NTDS Settings on the left.
On the right if there is still a connection object for IRVINESERVER delete it.


Run repadmin /sync /force

Run repadmin /showreps again and post.
We'll figure it out.

And yes, those entries - pay attention more to the folder structure on the left - it must come back to life and look like that.

no connection for IrvineServer in sites and services

repadmin didn't like that syntax

From the command prompt:

C:\Documents and Settings\Administrator>repadmin /sync /force
Usage: repadmin <cmd> <args> [/u:{domain\\user}] [/pw:{password|*}]

Supported <cmd>s & args:
     /sync <Naming Context> <Dest DSA> <Source DSA UUID> [/force] [/async]
            [/full] [/addref] [/allsources]
     /syncall <Dest DSA> [<Naming Context>] [<flags>]
     /kcc [DSA] [/async]
     /bind [DSA]
     /propcheck <Naming Context> <Originating DSA Invocation ID>
         <Originating USN> [DSA from which to enumerate host DSAs]
     /getchanges NamingContext [SourceDSA] [/cookie:<file>]
     /getchanges NamingContext [DestDSA] SourceDSAObjectGuid
          [/verbose] [/statistics]

     /showreps [Naming Context] [DSA [Source DSA objectGuid]] [/verbose]
         [/unreplicated] [/nocache]
     /showvector <Naming Context> [DSA] [/nocache]
     /showmeta <Object DN> [DSA] [/nocache]
     /showtime <DS time value>
     /showmsg <Win32 error>
     /showism [<Transport DN>] [/verbose] (must be executed locally)
     /showsig [DSA]
     /showconn [DSA] [Container DN | <DSA guid>] (default is local site)
     /showcert [DSA]

     /queue [DSA]
     /failcache [DSA]
     /showctx [DSA] [/nocache]

Note:- <Dest DSA>, <Source DSA>, <DSA> : Names of the appropriate servers
       <Naming Context> is the Distinguished Name of the root of the NC
              Example: DC=My-Domain,DC=Microsoft,DC=Com

C:\Documents and Settings\Administrator>repadmin /syncall
Invalid commandline; use repadmin /SyncAll /h for help.

C:\Documents and Settings\Administrator>repadmin /syncAll
Invalid commandline; use repadmin /SyncAll /h for help.

C:\Documents and Settings\Administrator>
Is there a Connection object under any of the servers for Irvinserver?

Yes, I boned up the command.

I'll see what the arguments should be.

no connection for IrvineServer under any of the servers
Heading home now.  Will catch you from there.

thanks!
Ok, I'm home now.

Any more news for me?
There have been no more entries in the event viewer on any of the servers.  There doesn't appear to be any data moving around either.

Did you figure out the proper syntax for that repadmin command you wanted me to run?
Not yet.  Just sent off an email to a contact at MS - waiting for some advice.

I know that I can get some events to come up if I restart FRS...hehe
Could any of this have anything to do with adprep and the introduction of Server 2003?  Or the manner in which I created the DFS root and its replicas?  I do recall getting a pop up error ('invalid pointer', I think) when trying to add Irvine as a root replica of the DFS root on server2 (I did the adding FROM server2).

Perhaps I should delete the DFS root and the replicas from each server and do some starting over as well.
Don't change anything just yet.

I need an email from you.  I have to send you a utility to run to create some logs for me.  Make sure the email is spamproof when you post it.

In the meantime, run these tests for me and post here.

1)  Run repadmin /showreps from Irvine.
2)  ping 51f814c3-f364-482a-8553-72a476a41261._msdcs.wse.com from Server3.
3)  ping 000282ea-4bba-4049-b3e8-fc70bb38c6f3._msdcs.wse.com from Irvine.
4)  If you Telnet to port 135 on Irvine from Server3 does that work?
5)  If you Telnet to port 135 on Server3 from Irvine does that work?
6)  Start Trace on Irvine and Server3 at the same time.  Run ipconfig /flushdns on each.  Go to AD Sites and Services then into the NTDS settings for Server3 and initiate a Replication from Irvine's Connection Object.  Stop Trace and post the logs on your site.

Once I get your email I will instruct on what to do with the tools I send.
my addy: www.wrightcustomhome.com/myaddy.jpg

1) C:\Documents and Settings\Administrator.WSE>repadmin /showreps
Default-First-Site-Name\IRVINE
DC Options: (none)
Site Options: (none)
DC object GUID: 51f814c3-f364-482a-8553-72a476a41261
DC invocationID: ba8b3fc4-dd78-4614-8bf1-0e933e7450e5

==== INBOUND NEIGHBORS ======================================

DC=wse,DC=com
    Default-First-Site-Name\SERVER2 via RPC
        DC object GUID: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
        Last attempt @ 2004-03-25 15:47:22 was successful.
    Default-First-Site-Name\SERVER3 via RPC
        DC object GUID: 000282ea-4bba-4049-b3e8-fc70bb38c6f3
        Last attempt @ 2004-03-25 15:51:28 was successful.

CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\SERVER2 via RPC
        DC object GUID: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
        Last attempt @ 2004-03-25 15:52:14 was successful.
    Default-First-Site-Name\SERVER3 via RPC
        DC object GUID: 000282ea-4bba-4049-b3e8-fc70bb38c6f3
        Last attempt @ 2004-03-25 15:52:28 was successful.

CN=Schema,CN=Configuration,DC=wse,DC=com
    Default-First-Site-Name\SERVER2 via RPC
        DC object GUID: 6233f4eb-40c9-47a7-9096-2f1e88d0c8b1
        Last attempt @ 2004-03-25 14:56:29 was successful.
    Default-First-Site-Name\SERVER3 via RPC
        DC object GUID: 000282ea-4bba-4049-b3e8-fc70bb38c6f3
        Last attempt @ 2004-03-25 14:56:29 was successful.

2) Ping gets a response, no problem

3) Ping gets a response, no problem

4) when I type telnet irvine 135 in a command prompt, I get a blank screen

5) same

6) I don't know how to start a trace....(help!) sorry :(
I'm clarifying what kind of trace he's looking for right now.

Work on posting those cabs.

Once I get them, I will ask you to remove the links to them.
btw, what does this util do?
It creates some logs about the structure and health of your AD and it will tell us why that phantom server is hanging around.  We figure this is what's interfering with normal replication.

wow.  utils are running as I type.  No need to run it on server2?
No, not yet.

Hmmm...seems to be good so far.  Just one log shows that old DC and I can't figure out where it's coming from.

Too bizarre.

As per your instructions, I followed the 6 'homework' steps:


1) Deleted everything out of Forward and Reverse lookup zones and their subfolders on all three servers

2) stopped the netlogon and NTFRS services on all three, then deleted the files Ntfrs.jdb, Edb.chk, edb.log, res1.log, res2.log on all three

3) emptied all the event logs on all three

4) set chkdsk /f on all three, then restarted server2, then server3, then irvine, one right after the other

5) waiting on server2 to restart

6) we'll see....
server3 and irvine (being much newer machines) restarted way faster than server2, even though I started it first.  I hope that's not a problem...
Server2 has restarted.  Checking DNS on server2 in the same folder tree as shown in the screen shot, there are records for server2 and irvine, but not server3.

In the FRS log, all three had the same sequence of events: 13501, 13553, 13554, 13520, 13553, 13554, 13508, 135080, 13508.

I'll check in the morning if there are any others.  That whole process, btw, caused all the folders and files that the users connect to to be moved in the 'ntfrs pre-existing' folder.  That would have put me in a world of hurt if I hadn't discovered that before work started tomorrow!!!  they are all in DataStore, which is the main and original DFS root.

I am simply floored that this is taking so much effort...
the server3 entry has appeared in DNS (same place in the screenshot).
I'm wondering if the fact that the data was moved to "ntfrs pre-existing" is not significant.  I wonder if you had have left it there if it would have replicated correctly.

I expected all the FRS errors - after all we deleted all the log files.

All the FRS errors are normal - the 13508 could just have been because it took some time for all servers to come up.  It would make me happy to hear you have some 13509 events this morning!

To be honest with you - the entire structure there is questionable.  There are lots of little issues, likely because of the way it was originally setup, that are causing the 'cascading' effect.  Much of the effort we are putting in is to simply clean up the little things so that the bigger issues surface.  Most of the time the compounded, small issues create the big issue - so it's critical we tidy things up to see if the big stuff goes away.  Another thing that makes it harder is trying to nail issues like you're having 'blind' - while not having the luxury of using my own eyes.

Its' all part of the challenge though.

HELP!  None of my users can connect to any of the network drive this morning!  When I type \\server2, it ask for username/password.  I've tried mine as well as the administrator's, doesn't work!!!  People are freaking out.
ASKER CERTIFIED SOLUTION
Avatar of Netman66
Netman66
Flag of Canada image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
I must admint a certain level of stress has left me...

Things seem to be good still, a day or so later.  I had server3 participating in the DFS as well, but its frs-staging folder was on the C drive (which has much less space), whereas the DFS replica is on the D drive.  It filled up the C drive (is there a way to specify the location of the frs-staging folder?), causing not only the inital replication to stop, but the server to begin acting 'weird'.  So, I simply removed it as a DFS replica for now.  I can always start it up again.  It isn't going anywhere.

There were two items in the Irvine event logs, 13523 & 13522, which caused replication to pause; so I used regedit to increase the staging area size, and shortly there after, replication resumed and is still humming along.

I chose (wisely, I'm sure you'll agree, Netman) to leave Irvine on the LAN until its initial replication had entirely completed, which should be within an hour or so.  We'll ship it to CA on Monday, where it will be put in its new home in Irvine, CA on Tuesday morning.  then, the excitment of moving it into it's Site begins...

I wanted to ask about fixing DNS once Irvine's IP changes.  There are a lot of entries to change; is there a way to do it faster, or all at once?
Netman66, please post something here...

https://www.experts-exchange.com/questions/20931172/this-thread-might-need-to-be-here.html

I believe these points belong to you as well. :)

Scavenging is normally how these stale entries get removed.

However, since this is a DC I am more concerned with getting things cleaned up in a way that you are able to see.

We'll talk once you have it up in CA.
When it rains, it pours!  I was just checking on the replication progress from home and got no response from server2!  I ran down there (luckily only 10-15 away) and what did I see but BSOD.  One of the RAID 5 drives tanked...thank goodness for RAID 5, though, because it started right back up!  I'll be talking to Dell right away, I think its still under warranty.

We only lack about 5 GB for replication to be complete, so it should be done in less than two hours (now that server2 is back online.

I wonder if the cause of the crash was one of my scheduled tasks: I have a little third party utility that restarts server2 every 2 weeks.  Could that have caused the blue screen you think?

Poor server2, working so hard...
Is there a way to specify the location of the frs-staging folder?  Now that one of server2's drives is down, I'm more concerned about getting a DFS replica on Server3, but it stopped because the staging area was on the C drive.  It should work if its on the D drive.  I'l like to start it up again...
Not sure about that - I'll check.

About Server2 - I noticed lots of events related to the controller - ou might want to have them bring a new controller with them too.

The events were just regarding the battery, I thought...
You know, one other thing: While server2 was down, I was curious to see how the Datastore was working, so I typed \\wse\datastore in explorer, and it asked me for a username/password!  I entered my own and no dice.  I entered the domain administrator credentials, and it let me in.  Shouldn't the DFS be available to all authenticated users, where regular permissions allow?  Is that something we need to check?  It would be pretty useless otherwise if server2 actually did go down.  I'd have to give everyone the admin user/pass for them to reconnect!
I'll check on that one too.
Here's the answer for moving the Pre-Staging area - too ugly.  It requires a D2 again on the server you want to move areas then an ADSIEdit hack to relocate the volume.

Here's the relocation article:
http://support.microsoft.com/default.aspx?scid=kb;en-us;265085&Product=win2000

To answer your question of the other day - you can pre-stage your DFS like this:
http://support.microsoft.com/default.aspx?scid=kb;en-us;266679&Product=win2000


This could be your DFS issue - just a guess:
http://support.microsoft.com/default.aspx?scid=kb;en-us;282080&Product=win2000

If replication has't finished - it might just be a case of the permissions not being finalized.

interesting about the pre-staging article.  I wonder if Veritas counts as a viable third-party app?  I would imagine so, since the built in Backup program in windows is actually the Veritas engine, isn't it?

Do I understand the article correctly that you add the third replica without enabling replication until the backup has been restored?  Would it work to restore the backup to the shared folder, and then add it as a root replica?  

I went ahead and added server3 as a root replica from server2  When I right-click on the DFS root and choose Replication Policy, I get a pop up error that says 'invalid pointer'.  Strange.  When I 'show replication info' on Irvine, server3 is listed as 'not eligible'.  why would it not be eligible, I wonder?

I would love to be able to restore from a Veritas backup, and the enabe replication from there.  That would probably speed things up, I imagine.
I started a new thread in which to continue this discussion, as it is moving away from the original question focus.

https://www.experts-exchange.com/questions/20935398/DFS-and-remote-sites-part-II.html