mcse63
asked on
File Replication Error on DC
Event Type: Error
Event Source: NtFrs
Event Category: None
Event ID: 13555
Date: 8/24/2004
Time: 1:55:00 PM
User: N/A
Computer: Server
Description:
The File Replication Service is in an error state. Files will not replicate to or from one or all of the replica sets on this computer until the following recovery steps are performed:
I have stop and restarted the NTFRS service with no success. Is there any other way besides restoring from backup?
Event Source: NtFrs
Event Category: None
Event ID: 13555
Date: 8/24/2004
Time: 1:55:00 PM
User: N/A
Computer: Server
Description:
The File Replication Service is in an error state. Files will not replicate to or from one or all of the replica sets on this computer until the following recovery steps are performed:
I have stop and restarted the NTFRS service with no success. Is there any other way besides restoring from backup?
Have you ran a dcdiag, and a netdiag. Also have a look at repadmin - let's see what state the server is in... Any KCC events in the event log?
ASKER
Here is what dcdiag shows.
Testing server: Default-First-Site-Name\Se rver
Starting test: Replications
......................... Server passed test Replications
Starting test: NCSecDesc
......................... server passed test NCSecDesc
Starting test: NetLogons
......................... server passed test NetLogons
Starting test: Advertising
......................... server passed test Advertising
Starting test: KnowsOfRoleHolders
......................... Server passed test KnowsOfRoleHolders
Starting test: RidManager
......................... Server passed test RidManager
Starting test: MachineAccount
......................... Server passed test MachineAccount
Starting test: Services
......................... Server passed test Services
Starting test: ObjectsReplicated
......................... Server passed test ObjectsReplicated
Starting test: frssysvol
Error: No record of File Replication System, SYSVOL started.
The Active Directory may be prevented from starting.
......................... Server passed test frssysvol
Starting test: kccevent
......................... Server passed test kccevent
Starting test: systemlog
......................... Server passed test systemlog
Running enterprise tests on : domain.com
Starting test: Intersite
......................... domain.com passed test Intersite
Starting test: FsmoCheck
......................... domain.com passed test FsmoCheck
Testing server: Default-First-Site-Name\Se
Starting test: Replications
......................... Server passed test Replications
Starting test: NCSecDesc
......................... server passed test NCSecDesc
Starting test: NetLogons
......................... server passed test NetLogons
Starting test: Advertising
......................... server passed test Advertising
Starting test: KnowsOfRoleHolders
......................... Server passed test KnowsOfRoleHolders
Starting test: RidManager
......................... Server passed test RidManager
Starting test: MachineAccount
......................... Server passed test MachineAccount
Starting test: Services
......................... Server passed test Services
Starting test: ObjectsReplicated
......................... Server passed test ObjectsReplicated
Starting test: frssysvol
Error: No record of File Replication System, SYSVOL started.
The Active Directory may be prevented from starting.
......................... Server passed test frssysvol
Starting test: kccevent
......................... Server passed test kccevent
Starting test: systemlog
......................... Server passed test systemlog
Running enterprise tests on : domain.com
Starting test: Intersite
......................... domain.com passed test Intersite
Starting test: FsmoCheck
......................... domain.com passed test FsmoCheck
Type net share and see if the sysvol is shared. Have you recently added or removed any domain controllers?
Also ensure that these 4 groups have the right to "access this computer from the network"
Administrators
Authenticated Users
ENTERPRISE DOMAIN CONTROLLERS
Everyone
Administrators
Authenticated Users
ENTERPRISE DOMAIN CONTROLLERS
Everyone
ASKER
Here are the results of net share
Share name Resource Remark
-------------------------- ---------- ---------- ---------- ---------- ---------- --
G$ G:\
IPC$ Remote IPC
D$ D:\ Default share
Resources$ D:\Program Files\Exchsrvr\res "Event logging files"
E$ E:\ Default share
ADMIN$ C:\WINNT Remote Admin
C$ C:\ Default share
Address D:\Program Files\Exchsrvr\address
"Access to address objects"
NETLOGON C:\WINNT\SYSVOL\sysvol\pin nacle.shop 4zero.com\ SCRIPTS
Logon server share
S4ZJAXNS0301.log
D:\Program Files\Exchsrvr\S4ZJAXNS030 1.log
"Exchange message tracking logs"
SYSVOL C:\WINNT\SYSVOL\sysvol Logon server share
The command completed successfully.
Share name Resource Remark
--------------------------
G$ G:\
IPC$ Remote IPC
D$ D:\ Default share
Resources$ D:\Program Files\Exchsrvr\res "Event logging files"
E$ E:\ Default share
ADMIN$ C:\WINNT Remote Admin
C$ C:\ Default share
Address D:\Program Files\Exchsrvr\address
"Access to address objects"
NETLOGON C:\WINNT\SYSVOL\sysvol\pin
Logon server share
S4ZJAXNS0301.log
D:\Program Files\Exchsrvr\S4ZJAXNS030
"Exchange message tracking logs"
SYSVOL C:\WINNT\SYSVOL\sysvol Logon server share
The command completed successfully.
ASKER
I have updated to sp4 and exchange sp3 recently. No new DC's have been added or removed.
Thanks,
Thanks,
The most likely cause of these errors is that the FRS Jet database, or a Jet database log file, is corrupt. To fix the problem:
1. Open a CMD prompt on the domain controller and stop the NetLogon and Ntfrs services:
net stop NetLogon
net stop Ntfrs
2. Type:
del %systemroot%\ntfrs\jet\Ntf rs.jdb
del %systemroot%\ntfrs\jet\Sys \Edb.chk
del %systemroot%\ntfrs\jet\log \edb.log
del %systemroot%\ntfrs\jet\log \res1.log
del %systemroot%\ntfrs\jet\log \res2.log
3. Type:
net start NetLogon
net start Ntfrs
4. Check the FRS event log for expected warning 13514 and 13520.
5. Close the FRS event log and go get a cup of coffee.
6. In five minutes, recheck the FRS event log for informational messages 13553, 13554, and 13516.
1. Open a CMD prompt on the domain controller and stop the NetLogon and Ntfrs services:
net stop NetLogon
net stop Ntfrs
2. Type:
del %systemroot%\ntfrs\jet\Ntf
del %systemroot%\ntfrs\jet\Sys
del %systemroot%\ntfrs\jet\log
del %systemroot%\ntfrs\jet\log
del %systemroot%\ntfrs\jet\log
3. Type:
net start NetLogon
net start Ntfrs
4. Check the FRS event log for expected warning 13514 and 13520.
5. Close the FRS event log and go get a cup of coffee.
6. In five minutes, recheck the FRS event log for informational messages 13553, 13554, and 13516.
ASKER
I had my cup of coffee and this is the new event message. Do I need to create this file?
Event Type: Error
Event Source: NtFrs
Event Category: None
Event ID: 13559
Date: 8/30/2004
Time: 9:46:48 AM
User: N/A
Computer: Server
Description:
The File Replication Service has detected that the replica root path has changed from "c:\winnt\sysvol\domain" to "c:\winnt\sysvol\domain". If this is an intentional move then a file with the name NTFRS_CMD_FILE_MOVE_ROOT needs to be created under the new root path.
This was detected for the following replica set:
"DOMAIN SYSTEM VOLUME (SYSVOL SHARE)"
Changing the replica root path is a two step process which is triggered by the creation of the NTFRS_CMD_FILE_MOVE_ROOT file.
[1] At the first poll which will occur in 5 minutes this computer will be deleted from the replica set.
[2] At the poll following the deletion this computer will be re-added to the replica set with the new root path. This re-addition will trigger a full tree sync for the replica set. At the end of the sync all the files will be at the new location. The files may or may not be deleted from the old location depending on whether they are needed or not.
Event Type: Error
Event Source: NtFrs
Event Category: None
Event ID: 13559
Date: 8/30/2004
Time: 9:46:48 AM
User: N/A
Computer: Server
Description:
The File Replication Service has detected that the replica root path has changed from "c:\winnt\sysvol\domain" to "c:\winnt\sysvol\domain". If this is an intentional move then a file with the name NTFRS_CMD_FILE_MOVE_ROOT needs to be created under the new root path.
This was detected for the following replica set:
"DOMAIN SYSTEM VOLUME (SYSVOL SHARE)"
Changing the replica root path is a two step process which is triggered by the creation of the NTFRS_CMD_FILE_MOVE_ROOT file.
[1] At the first poll which will occur in 5 minutes this computer will be deleted from the replica set.
[2] At the poll following the deletion this computer will be re-added to the replica set with the new root path. This re-addition will trigger a full tree sync for the replica set. At the end of the sync all the files will be at the new location. The files may or may not be deleted from the old location depending on whether they are needed or not.
Nope - just let it ride. This will probably clean itself up now. Just watch the event logs.
ASKER
Strange, now exchange on this box is no longer functioning.
Had to reboot the server and none of the service accounts are recognized. This now has gotten critical.
Had to reboot the server and none of the service accounts are recognized. This now has gotten critical.
ASKER
I just ran a dcdiag on the other DC
Doing primary tests
Testing server: Default-First-Site-Name\DC 1
Starting test: Replications
......................... DC1 passed test Replications
Starting test: NCSecDesc
......................... DC1 passed test NCSecDesc
Starting test: NetLogons
.........................D C1 passed test NetLogons
Starting test: Advertising
......................... DC1 passed test Advertising
Starting test: KnowsOfRoleHolders
......................... DC1 passed test KnowsOfRoleHolders
Starting test: RidManager
......................... DC1 passed test RidManager
Starting test: MachineAccount
......................... DC1 passed test MachineAccount
Starting test: Services
......................... DC1passed test Services
Starting test: ObjectsReplicated
......................... DC1 passed test ObjectsReplicated
Starting test: frssysvol
There are errors after the SYSVOL has been shared.
The SYSVOL can prevent the AD from starting.
......................... DC1 passed test frssysvol
Starting test: kccevent
......................... DC1 passed test kccevent
Starting test: systemlog
......................... DC1 passed test systemlog
Running enterprise tests on : pinnacle.shop4zero.com
Starting test: Intersite
......................... Domain.com passed test Intersite
Starting test: FsmoCheck
Warning: DcGetDcName(GC_SERVER_REQU IRED) call failed, error 1355
A Global Catalog Server could not be located - All GC's are down.
......................... Domain.com failed test FsmoCheck
And here it is what the original offending DC shows now.
Domain Controller Diagnosis
Performing initial setup:
Done gathering initial info.
Doing initial required tests
Testing server: Default-First-Site-Name\S4 ZJAXNS0301
Starting test: Connectivity
......................... DC2 passed test Connectivity
Doing primary tests
Testing server: Default-First-Site-Name\DC 2
Starting test: Replications
......................... DC2 passed test Replications
Starting test: NCSecDesc
......................... DC2 passed test NCSecDesc
Starting test: NetLogons
......................... DC2 passed test NetLogons
Starting test: Advertising
Warning: DsGetDcName returned information for \\Domain.com, when we were trying to reach S4ZJAXNS0301.
Server is not responding or is not considered suitable.
Warning: DC2 is not advertising as a global catalog.
Check that server finished GC promotion.
Check the event log on server that enough source replicas for the GC ar
e available.
......................... DC2 failed test Advertising
Starting test: KnowsOfRoleHolders
......................... DC2 passed test KnowsOfRoleHolders
Starting test: RidManager
......................... DC2 passed test RidManager
Starting test: MachineAccount
......................... DC2 passed test MachineAccount
Starting test: Services
......................... DC2 passed test Services
Starting test: ObjectsReplicated
......................... DC2 passed test ObjectsReplicated
Starting test: frssysvol
Error: No record of File Replication System, SYSVOL started.
The Active Directory may be prevented from starting.
There are errors after the SYSVOL has been shared.
The SYSVOL can prevent the AD from starting.
......................... DC2 passed test frssysvol
Starting test: kccevent
.........................D C2 passed test kccevent
Starting test: systemlog
An Error Event occured. EventID: 0x800009CA
Time Generated: 08/30/2004 12:21:06
Event String: The value named IRPStackSize in the server's
An Error Event occured. EventID: 0xC0001B58
Time Generated: 08/30/2004 12:21:26
Event String: The Backup Exec 8.x Device & Media Service
An Error Event occured. EventID: 0xC0001B59
Time Generated: 08/30/2004 12:21:26
Event String: The Backup Exec 8.x Job Engine service depends on
An Error Event occured. EventID: 0xC0001B58
Time Generated: 08/30/2004 12:21:26
Event String: The Backup Exec 8.x Naming Service service failed
An Error Event occured. EventID: 0xC0001B58
Time Generated: 08/30/2004 12:21:26
Event String: The Backup Exec 8.x Notification Server service
An Error Event occured. EventID: 0xC0001B59
Time Generated: 08/30/2004 12:21:26
Event String: The Backup Exec 8.x Server service depends on the
An Error Event occured. EventID: 0xC0001B59
Time Generated: 08/30/2004 12:21:46
Event String: The Backup Exec 8.x Agent Browser service depends
An Error Event occured. EventID: 0xC0001B59
Time Generated: 08/30/2004 12:21:46
Event String: The Backup Exec 8.x Alert Server service depends
An Error Event occured. EventID: 0xC0001B58
Time Generated: 08/30/2004 12:21:58
Event String: The Microsoft Exchange System Attendant service
An Error Event occured. EventID: 0xC0009007
Time Generated: 08/30/2004 12:22:08
Event String: A fatal error occurred while creating an SSL
An Error Event occured. EventID: 0xC0009007
Time Generated: 08/30/2004 12:22:08
Event String: A fatal error occurred while creating an SSL
An Error Event occured. EventID: 0xC0001B59
Time Generated: 08/30/2004 12:22:08
Event String: The Microsoft Exchange Information Store service
An Error Event occured. EventID: 0xC0001B59
Time Generated: 08/30/2004 12:22:08
Event String: The Microsoft Exchange MTA Stacks service depends
An Error Event occured. EventID: 0xC0002712
Time Generated: 08/30/2004 12:22:35
Event String: Access denied attempting to launch a DCOM Server.
An Error Event occured. EventID: 0xC0002712
Time Generated: 08/30/2004 12:31:51
Event String: Access denied attempting to launch a DCOM Server.
An Error Event occured. EventID: 0xC0002712
Time Generated: 08/30/2004 12:31:51
Event String: Access denied attempting to launch a DCOM Server.
An Error Event occured. EventID: 0xC0002712
Time Generated: 08/30/2004 12:31:51
Event String: Access denied attempting to launch a DCOM Server.
An Error Event occured. EventID: 0xC0002712
Time Generated: 08/30/2004 12:31:51
Event String: Access denied attempting to launch a DCOM Server.
An Error Event occured. EventID: 0xC0002712
Time Generated: 08/30/2004 12:31:51
Event String: Access denied attempting to launch a DCOM Server.
......................... DC2 failed test systemlog
Running enterprise tests on : pinnacle.shop4zero.com
Starting test: Intersite
......................... Domain.com passed test Intersite
Starting test: FsmoCheck
Warning: DcGetDcName(GC_SERVER_REQU IRED) call failed, error 1355
A Global Catalog Server could not be located - All GC's are down.
......................... pinnacle.shop4zero.com failed test FsmoCheck
any suggestions?
Doing primary tests
Testing server: Default-First-Site-Name\DC
Starting test: Replications
......................... DC1 passed test Replications
Starting test: NCSecDesc
......................... DC1 passed test NCSecDesc
Starting test: NetLogons
.........................D
Starting test: Advertising
......................... DC1 passed test Advertising
Starting test: KnowsOfRoleHolders
......................... DC1 passed test KnowsOfRoleHolders
Starting test: RidManager
......................... DC1 passed test RidManager
Starting test: MachineAccount
......................... DC1 passed test MachineAccount
Starting test: Services
......................... DC1passed test Services
Starting test: ObjectsReplicated
......................... DC1 passed test ObjectsReplicated
Starting test: frssysvol
There are errors after the SYSVOL has been shared.
The SYSVOL can prevent the AD from starting.
......................... DC1 passed test frssysvol
Starting test: kccevent
......................... DC1 passed test kccevent
Starting test: systemlog
......................... DC1 passed test systemlog
Running enterprise tests on : pinnacle.shop4zero.com
Starting test: Intersite
......................... Domain.com passed test Intersite
Starting test: FsmoCheck
Warning: DcGetDcName(GC_SERVER_REQU
A Global Catalog Server could not be located - All GC's are down.
......................... Domain.com failed test FsmoCheck
And here it is what the original offending DC shows now.
Domain Controller Diagnosis
Performing initial setup:
Done gathering initial info.
Doing initial required tests
Testing server: Default-First-Site-Name\S4
Starting test: Connectivity
......................... DC2 passed test Connectivity
Doing primary tests
Testing server: Default-First-Site-Name\DC
Starting test: Replications
......................... DC2 passed test Replications
Starting test: NCSecDesc
......................... DC2 passed test NCSecDesc
Starting test: NetLogons
......................... DC2 passed test NetLogons
Starting test: Advertising
Warning: DsGetDcName returned information for \\Domain.com, when we were trying to reach S4ZJAXNS0301.
Server is not responding or is not considered suitable.
Warning: DC2 is not advertising as a global catalog.
Check that server finished GC promotion.
Check the event log on server that enough source replicas for the GC ar
e available.
......................... DC2 failed test Advertising
Starting test: KnowsOfRoleHolders
......................... DC2 passed test KnowsOfRoleHolders
Starting test: RidManager
......................... DC2 passed test RidManager
Starting test: MachineAccount
......................... DC2 passed test MachineAccount
Starting test: Services
......................... DC2 passed test Services
Starting test: ObjectsReplicated
......................... DC2 passed test ObjectsReplicated
Starting test: frssysvol
Error: No record of File Replication System, SYSVOL started.
The Active Directory may be prevented from starting.
There are errors after the SYSVOL has been shared.
The SYSVOL can prevent the AD from starting.
......................... DC2 passed test frssysvol
Starting test: kccevent
.........................D
Starting test: systemlog
An Error Event occured. EventID: 0x800009CA
Time Generated: 08/30/2004 12:21:06
Event String: The value named IRPStackSize in the server's
An Error Event occured. EventID: 0xC0001B58
Time Generated: 08/30/2004 12:21:26
Event String: The Backup Exec 8.x Device & Media Service
An Error Event occured. EventID: 0xC0001B59
Time Generated: 08/30/2004 12:21:26
Event String: The Backup Exec 8.x Job Engine service depends on
An Error Event occured. EventID: 0xC0001B58
Time Generated: 08/30/2004 12:21:26
Event String: The Backup Exec 8.x Naming Service service failed
An Error Event occured. EventID: 0xC0001B58
Time Generated: 08/30/2004 12:21:26
Event String: The Backup Exec 8.x Notification Server service
An Error Event occured. EventID: 0xC0001B59
Time Generated: 08/30/2004 12:21:26
Event String: The Backup Exec 8.x Server service depends on the
An Error Event occured. EventID: 0xC0001B59
Time Generated: 08/30/2004 12:21:46
Event String: The Backup Exec 8.x Agent Browser service depends
An Error Event occured. EventID: 0xC0001B59
Time Generated: 08/30/2004 12:21:46
Event String: The Backup Exec 8.x Alert Server service depends
An Error Event occured. EventID: 0xC0001B58
Time Generated: 08/30/2004 12:21:58
Event String: The Microsoft Exchange System Attendant service
An Error Event occured. EventID: 0xC0009007
Time Generated: 08/30/2004 12:22:08
Event String: A fatal error occurred while creating an SSL
An Error Event occured. EventID: 0xC0009007
Time Generated: 08/30/2004 12:22:08
Event String: A fatal error occurred while creating an SSL
An Error Event occured. EventID: 0xC0001B59
Time Generated: 08/30/2004 12:22:08
Event String: The Microsoft Exchange Information Store service
An Error Event occured. EventID: 0xC0001B59
Time Generated: 08/30/2004 12:22:08
Event String: The Microsoft Exchange MTA Stacks service depends
An Error Event occured. EventID: 0xC0002712
Time Generated: 08/30/2004 12:22:35
Event String: Access denied attempting to launch a DCOM Server.
An Error Event occured. EventID: 0xC0002712
Time Generated: 08/30/2004 12:31:51
Event String: Access denied attempting to launch a DCOM Server.
An Error Event occured. EventID: 0xC0002712
Time Generated: 08/30/2004 12:31:51
Event String: Access denied attempting to launch a DCOM Server.
An Error Event occured. EventID: 0xC0002712
Time Generated: 08/30/2004 12:31:51
Event String: Access denied attempting to launch a DCOM Server.
An Error Event occured. EventID: 0xC0002712
Time Generated: 08/30/2004 12:31:51
Event String: Access denied attempting to launch a DCOM Server.
An Error Event occured. EventID: 0xC0002712
Time Generated: 08/30/2004 12:31:51
Event String: Access denied attempting to launch a DCOM Server.
......................... DC2 failed test systemlog
Running enterprise tests on : pinnacle.shop4zero.com
Starting test: Intersite
......................... Domain.com passed test Intersite
Starting test: FsmoCheck
Warning: DcGetDcName(GC_SERVER_REQU
A Global Catalog Server could not be located - All GC's are down.
......................... pinnacle.shop4zero.com failed test FsmoCheck
any suggestions?
Where are your DNS server(s) located - I'd bet on the Exchange server, correct?
OK - Demote the 2nd DC -(Not the DC with Exchange installed) ---
Make sure the DC with Exchange is a global catalog - if the demotion did not go 100% according to the exchange server, seize any FSMO roles that are no longer valid. Remove the demoted server from the domain all together. Reboot the exchange server is Directory services mode. Run ntdsutil, and do a metadata cleanup of the database, removing all references to the demoted domain controller. Reboot again.
You may also have to use ADSIedit to remove all replication links to/from this DC.
Once you get this far, delete any pointers to this server name in DNS and WINS.
Monitor your event logs, and make sure the AD is functioning correctly on the exchange server. Make sure that all reference to the 2nd server are completely removed from active directory or you'll be in the same situation.
You should now be able to repromote the 2nd DC. After all of this is done, exchange should start correctly - If not reinstall overtop of your what you have, and reapply sp3 with that post sp3 hotfix rollup.
OK - Demote the 2nd DC -(Not the DC with Exchange installed) ---
Make sure the DC with Exchange is a global catalog - if the demotion did not go 100% according to the exchange server, seize any FSMO roles that are no longer valid. Remove the demoted server from the domain all together. Reboot the exchange server is Directory services mode. Run ntdsutil, and do a metadata cleanup of the database, removing all references to the demoted domain controller. Reboot again.
You may also have to use ADSIedit to remove all replication links to/from this DC.
Once you get this far, delete any pointers to this server name in DNS and WINS.
Monitor your event logs, and make sure the AD is functioning correctly on the exchange server. Make sure that all reference to the 2nd server are completely removed from active directory or you'll be in the same situation.
You should now be able to repromote the 2nd DC. After all of this is done, exchange should start correctly - If not reinstall overtop of your what you have, and reapply sp3 with that post sp3 hotfix rollup.
ASKER
The exchange server is not the DNS Server. It was the GC however. I would really like to know how all of this occured and a bit more details on how to correct this problem.
ASKER
update, this is however a dc/echangeserver/and backup server.
It looks to me like the jet database got corrupted either with virus scanning software, improper shutdown, however without being there and seeing what is happening, I can not pinpoint it.
Start with demoting the DC, and see what state the directory is in. More can be determined after this is done. After the demotion is done, wait for about 15 minutes and run dcdiag test on the exchange server again. Post the results.
Start with demoting the DC, and see what state the directory is in. More can be determined after this is done. After the demotion is done, wait for about 15 minutes and run dcdiag test on the exchange server again. Post the results.
ASKER
Demoting the DNS/DC/GC or the DC with exchange server?
Put a GC on the exchange server, and demote the DNS/DC.
ASKER
I really am feeling like an imbecile at this point, but how do I go about putting a GC on the exchange server. I know I can go to ad sites and services and change it there, but if replication is not happening how can I be sure that there will be a good copy of the GC?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I was able to get Exchange going without demoting the other server. I changed the logon from the domain acct to logon locally and all is back for now. What is strange is when I switch DC1 to a GC and then DC2 to a GC those changes are recognized between servers. This would indicate that some type of replication is taking place. Let me know of any other thoughts on this as I am going to do some more digging as well.
Thanks
Thanks
I'm not surprised about that (Exchange). But - for your other errors with FRS - you are probably going to need to demote, see where the directory stands, then minimally, just repromote. Good luck.
ASKER