SCCM 2012 R2 - Site Replication Failing

We use SCCM 2012 R2 and have a "Central" ConfigManager server (site code FCC) as well as a "main" ConfigManager Server (HQ1), and a "disaster recovery" ConfigManager server (DR1) that we'd like to replicate from HQ1.
 
So the hierarchy is:
 
    FCC
     /   \
DR1  HQ1
 
Under the monitoring tab, in the Site Node Status, it shows a red "X" between FCC and HQ1. It's apparently been this way for some time, but I'd like to get it resolved.  When you hover over it, it shows for "Global Data Replication Status" that HQ1 to FCC and FCC to HQ1 is green and active. However, for "Site Data Replication Status" for HQ1 to FCC, it's red and shows a failed status.
 
When I right click and run a report for errors, nothing really stands out as problematic that's recent that would explain the communication/replication issue between HQ1 and FCC.
 
Can anyone tell me where to look to figure out why we're getting this warning/error, or how to resolve it?
 
Thank you!
LVL 1
FirstcomAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Randy DownsOWNERCommented:
Maybe this will help.

On each of your secondary databases, run

delete from SSB_DialogPool where ToService like '%PRIMARYSITECODE'

On each secondary navigate to:

C:\Program Files\Microsoft Configuration Manager\inboxes\rcm.box

And create two files, one at a time.

“Secondary_Site_Replication_Configuration.pub”

And

“Secondary Site Data.pub”  

Note the spaces above.

These two files will disappear shortly after being created.

We waited a while and everything went green but shortly after the sites failed again. L

At this point we went into Administration/Sites and selected “Recover Secondary Site” for each site.

At this point after a short wait everything started working again.

Not entirely sure if it was just the “Recover Secondary Site” task that nailed it or a combination of the two.
FirstcomAuthor Commented:
I'm curious what this actually does...
FirstcomAuthor Commented:
Okay, so I have an update on this. I haven't tried the solution suggested quite yet because I am not sure what it does.

Reading through posts though, I discovered the "Replication Link Analyzer" tool.

I keep seeing errors stating things along the lines of "IsSiteActive" failed.

5/18/2015 2:27:06 PM: ReplicationLinkAnalysis Information: 1 : All links are initialized between subscriber HQ1 and publisher FCC.
5/18/2015 2:27:06 PM: ReplicationLinkAnalysis Information: 1 : Getting results of rule: IsSiteActive execution
5/18/2015 2:27:06 PM: ReplicationLinkAnalysis Information: 1 : Loading result Xml from rule:IsSiteActive
5/18/2015 2:27:06 PM: ReplicationLinkAnalysis Information: 1 : rule:IsSiteActive has some dependent rules
5/18/2015 2:27:06 PM: ReplicationLinkAnalysis Information: 1 : Working with parent rule:IsSiteActive -> dependent rule:DoesFileReplicationRouteExist
5/18/2015 2:27:06 PM: ReplicationLinkAnalysis Information: 1 : Appending result xml
5/18/2015 2:27:06 PM: ReplicationLinkAnalysis Information: 1 : Rule IsSiteActive result: failed.
5/18/2015 2:27:06 PM: ReplicationLinkAnalysis Information: 1 : Rule: IsSiteActive failed. Stopped running replication link analysis rules.
5/18/2015 2:27:06 PM: ReplicationLinkAnalysis Information: 1 : Completed replication link analysis thread.

The Link Analyzer tool shows a red "X" for "Replication initialization is healthy on site FCC and site HQ1", so it's obviously unhealthy. But unfortunately, the tool doensn't provide more specifics..

Any tips?
The 7 Worst Nightmares of a Sysadmin

Fear not! To defend your business’ IT systems we’re going to shine a light on the seven most sinister terrors that haunt sysadmins. That way you can be sure there’s nothing in your stack waiting to go bump in the night.

Randy DownsOWNERCommented:
You could just try Recover Secondary Site.



Recover a Secondary Site

For System Center 2012 Configuration Manager SP1 and later:
Configuration Manager does not support the backup of the database at a secondary site, but does support recovery by reinstalling the secondary site. Secondary site recovery is required when a Configuration Manager secondary site fails. You can recover a secondary site by using the Recover Secondary Site action from the Sites node in the Configuration Manager console. Unlike recovery for a central administration site or primary site, recovery for a secondary site does not use a backup file and instead reinstalls the secondary site files on the failed secondary site computer. Then, the secondary site data is reinitialized with data from the parent primary site. During the recovery process, Configuration Manager verifies if the content library exists on the secondary site computer and that the appropriate content is available. The secondary site will use the existing content library, if it contains the appropriate content. Otherwise, to recover the content library of a recovered secondary site requires you to redistribute or prestage the content to that recovered site. For more information, see Operations and Maintenance for Content Management in Configuration Manager. When you have a distribution point that is not on the secondary site, you are not required to reinstall the distribution point during a recovery of the secondary site. After the secondary site recovery, the site automatically synchronizes with the distribution point.
You can verify the status of the secondary site recovery, by using the Show Install Status action from the Sites node in the Configuration Manager console.
FirstcomAuthor Commented:
The part that I am confused about is that the secondary site (HQ1) is operating just fine. It's where we essentially do everything. It's our central central (FCC) that it seems to be having a hard time talking with. That site seems to be operating alright as well. But we're still getting these warnings in the admin console about the link not working.

Even though it doesn't *seem* to be having any negative consequences, I'd like to resolve the issue. That and I suspect it has to do with why our DR1 site isn't replicating or syncing properly.
FirstcomAuthor Commented:
Running a "recovery" on the ConfigManager server we use to do everything when it's operating okay just seem rather risky - and I don't understand why a "recovery" is required.
FirstcomAuthor Commented:
Actually, correction: Both HQ1 and DR1 are considered primary sites, not secondary sites. We have no 'secondary' sites configured.
Randy DownsOWNERCommented:
Maybe this will help.

I presume that RLA was not able to fix the problem.
FirstcomAuthor Commented:
RLA has not fixed the issue. Here's the logs where it first errors:

5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Running rule: IsSiteActive.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Running rule: <IsSiteActive SourceSiteCode="FCC" TargetSiteCode="HQ1"  ></IsSiteActive>
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for High_Priority_Site from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Collection_Membership from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Medium_Priority_Site from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for EndpointProtection_Site from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for General_Site_Data from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for CI_Compliance_Rule_Details from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for CI_Compliance_Status_Details_and_History from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Software_Inventory_and_Metering from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Status_Messages from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for MDM_Site from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for MDM_SiteSpecial from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for CFD_Site from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_1 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_2 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_3 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_4 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_5 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_6 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_7 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_8 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_9 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_10 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_11 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_12 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_13 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_14 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_15 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_16 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_17 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_18 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_19 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_20 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_21 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_22 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_23 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_24 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is in progress for Hardware_Inventory_25 from publisher HQ1.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : Initialization is failed for replication group Operational_Data.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : RLA detected no re-initialization for the group Operational_Data in last 24 hours.
5/15/2015 9:37:37 AM: ReplicationLinkAnalysis Information: 1 : BCP in failed for Operational_Data for tables DrsSendHistorySummary.
5/15/2015 9:37:38 AM: ReplicationLinkAnalysis Information: 1 : All links are initialized between subscriber HQ1 and publisher FCC.
5/15/2015 9:37:38 AM: ReplicationLinkAnalysis Information: 1 : Getting results of rule: IsSiteActive execution
5/15/2015 9:37:38 AM: ReplicationLinkAnalysis Information: 1 : Loading result Xml from rule:IsSiteActive
5/15/2015 9:37:38 AM: ReplicationLinkAnalysis Information: 1 : rule:IsSiteActive has some dependent rules
5/15/2015 9:37:38 AM: ReplicationLinkAnalysis Information: 1 : Working with parent rule:IsSiteActive -> dependent rule:DoesFileReplicationRouteExist
5/15/2015 9:37:38 AM: ReplicationLinkAnalysis Information: 1 : Appending result xml
5/15/2015 9:37:38 AM: ReplicationLinkAnalysis Information: 1 : Rule IsSiteActive result: failed.
5/15/2015 9:37:38 AM: ReplicationLinkAnalysis Information: 1 : Rule: IsSiteActive failed. Stopped running replication link analysis rules.
5/15/2015 9:37:38 AM: ReplicationLinkAnalysis Information: 1 : Completed replication link analysis thread.
Randy DownsOWNERCommented:
How about the other troubleshooting methods?
2. Log file – rcmctrl.log, replmgr.log.
3. Run script spDiagDRS
4. check the SQL Service Broker log: C:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\ErrorLog

Or follow the link here to use Profiler and SSB related table to troubleshoot service broker problems: http://www.sqlteam.com/article/how-to-troubleshoot-service-broker-problems

5.  Reinitiate by sending subscription invalid message.:

Have you seen this?

First off, it is not recommend to have a CAS unless you have over 100,000 computers.

Secondly, I always recommend that you contact CSS for support anytime you need to recover a Primary site or CAS.

Since this is a lab, I would likely just re-built it and do so without a CAS. IF this is not an option then I would open a support case with CSS to get thing fixed.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Applications

From novice to tech pro — start learning today.