Solved

3 million user on Local AD to be synchronized with Office 365 FID issue

Posted on 2015-01-16
4
170 Views
Last Modified: 2016-01-02
Hello everyone,

I have a customer (University) Who has an issue with DirSync. They have 3 million users on Local AD they want to synchronize with Office 365 to enable these users for Exchange online.

Now they have users "Students" enabled for Exchange online and management and staff are enabled on the On-premises Exchange servers.

Dirsync during the day synchronize 2 times fine without any error and again 2 times doesn't synchronize and gives error with no details. the error is "Stopped Extension-dll exception"

More errors shown as below
Directory Synchronization:
An unknown error occurred with the Microsoft Online Services Sign-in Assistant. Contact Technical Support. SetCredential() failed. Contact Technical Support.  (0x8009000B)

I am attaching other errors as well

   at Microsoft.Online.Coexistence.ProvisionHelper.GetLiveCompactToken(String userName, String userPassword)
   at Microsoft.Azure.ActiveDirectory.Connector.ProvisioningServiceAdapter.Initialize()
   at Microsoft.Azure.ActiveDirectory.Connector.ProvisioningServiceAdapter.Import(Byte[] syncCookie, Boolean isFullImport)
   at Microsoft.Azure.ActiveDirectory.Connector.Connector.GetImportEntriesCore()
   at Microsoft.Azure.ActiveDirectory.Connector.Connector.GetImportEntries(GetImportEntriesRunStep getImportEntriesRunStep)
Forefront Identity Manager 4.1.3465.0"

FIMSynchronizationService:
The management agent "Windows Azure Active Directory Connector" failed on run profile "Delta Import Delta Sync" because the server encountered errors.

FIMSynchronizationService:
The management agent "Windows Azure Active Directory Connector" step execution completed on run profile "Delta Import Delta Sync" but the watermark was not saved.
 
 Additional Information
 Discovery Errors       : "0"
 Synchronization Errors : "0"
 Metaverse Retry Errors : "0"
 Export Errors          : "0"
 Warnings               : "0"
 
 User Action
 View the management agent run history for details.

Directory Synchronization:
The Management Agent Windows Azure Active Directory Connector failed on execution. Error returned is 'stopped-extension-dll-exception'.  If the problem persists, contact Technical Support.

Open in new window


Customer have tried to involve Microsoft with them through a third party technical support company but microsoft was not able to apply anything since they have tried to apply some scripts but those scripts would take 3 days without finishing.

The first time the Dirsync was applied it took 1 week without finishing until now they were not able to apply a full import and export sync.

What have really got me interested is that Microsoft did not suggest to the customer to upgrade his FIM (ForeFront Identity Manager)'s old version to the latest one.

Customer is using Full SQL deployment on a dedicated server and DirSync (FID) on a separate server too. The deployed servers are virtual and have 32 GB ram and 200 GB HDD size and 4 cores.


I have recommended to this customer that we do not touch this current deployment since Microsoft themselves couldn't do anything in regard, but what we could do is take a virtual snapshot and then apply the upgrade and see if this resolves the issue or not?

Note:

Microsoft talked to them about a limited number of synchronized items to their Azure site per week! I am not sure about this but what the customer said is that they change approximately about 25,000 user object per day.
Could this issue happens because of this limit?


Thanks
anadolu-ad.jpg
0
Comment
Question by:Mohammed Hamada
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
4 Comments
 
LVL 41

Expert Comment

by:Vasil Michev (MVP)
ID: 40553472
I'm not aware about such limits, but I will ask around, see if I can confirm this. The obvious bit is of course the predefined 300k limit for normal SKUs, but I highly doubt they have missed that. The SIA error is also puzzling, doesnt make sense to happen only sometimes if it was profile/credentials related.

Ask them to check for any mails sent to the technical contact address for the O365 tenant, they might contain additional information.
0
 
LVL 24

Author Comment

by:Mohammed Hamada
ID: 40553514
They basically have no direct connection with Microsoft, they hired a firm which applied the installation and configuration along with Microsoft's help. but in the last 2 years (Project was started right after the Office 365 hybrid integration came out) and until this moment they still have cases open with Microsoft that haven't been closed yet.

They have asked us (different technical company) to involve directly with them (the university) and see if we can get our hands on it and fix it. When I checked their Dirsync and SQL servers! they seem to be used old version of Forefront Identity Manager 2010 R2 (build 4.1.3456.0)

I am not sure if this includes SP1 in it but when I checked the (About) it didn't show anywhere if SP1 is installed or not but the version seems to be not that old but not the latest.

I've seen there's a hotfix which fixes some issues and suggested that we install it but they were terrified to do anything so I suggested that we take a snapshot of the machines and try upgrading FFIM to the latest version and install all hotfixes included and see if that works.

The worse possibility would be to Deactivate DirSync, then restore the old snapshots and reactive it which might take a day or two.

HotFix
http://support.microsoft.com/kb/2980295
0
 
LVL 24

Accepted Solution

by:
Mohammed Hamada earned 0 total points
ID: 41387957
My colleague was able to solve the problem by taking a backup of the SQL server, deleted the DirSync server and took a backup of the SQL key.

Reinstalled SQL , Dirsync and the problem was gone.
0
 
LVL 24

Author Closing Comment

by:Mohammed Hamada
ID: 41393090
Self managed
0

Featured Post

Enroll in June's Course of the Month

June's Course of the Month is now available! Every 10 seconds, a consumer gets hit with ransomware. Refresh your knowledge of ransomware best practices by enrolling in this month's complimentary course for Premium Members, Team Accounts, and Qualified Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A hard and fast method for reducing Active Directory Administrators members.
On Feb. 28, Amazon’s Simple Storage Service (S3) went down after an employee issued the wrong command during a debugging exercise. Among those affected were big names like Netflix, Spotify and Expedia.
A company’s greatest vulnerability is their email. CEO fraud, ransomware and spear phishing attacks are the no1 threat to a company’s security. Cybercrime is responsible for the largest loss of money to companies today with losses projected to r…
There are cases when e.g. an IT administrator wants to have full access and view into selected mailboxes on Exchange server, directly from his own email account in Outlook or Outlook Web Access. This proves useful when for example administrator want…

724 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question