MOSS_DELTAIMPORT stage very slow after User Profile Service synchronization reset

I realize it's entirely possible that this kind of behavior is normal, but I'd appreciate confirmation or correction...

We were no longer gathering new AD users in our synchronization and after many hours of tinkering, I followed this article to reset sync. I reconnected AD (left a BDC connection out) and ran a full sync. 36 hours later it had completed with all of the new users we were missing.

However, the next incremental sync is where I saw issues. According to the MIIS client, the process hit the MOSS_DELTAIMPORT step and slowed down incredibly. It was averaging about 6 objects per minute, which is not good with 200,000 objects to process.

I let it run for days and it never sped up. The timer job entered the 'Pausing' state and a few days later, I had enough. I stopped it, restarted services and attempted to start the incremental over. Once again, it hit the MOSS_DELTAIMPORT step and essentially stopped. Days later, a mere 2500 objects were processed and the timer job was in 'Pausing' state again. Rinse and repeat a couple more times.

One constant (aside from speed) was that it appeared that most of the objects/users were both Deletes and Adds in the statistics inside of the MIIS client. I would assume this occurs because the sync service needs to reconcile the users already in the profile store with the AD data in the metaverse. However, I would not assume that this step take about 100x longer than any other step in the sync process.

I've looked through logs and I've been tracking performance data on our SQL server and the application servers. The FIM service and the UPS service rarely exceed 5% CPU and DB IO is in the KB/s range. Aside from the timer job slipping into the 'Pausing' state, there's no evidence to suggest that the process isn't continuing to run.

I don't know what information I need to provide to get some insight here, but I will do what I can to offer what I have. I wanted to provide a little background and touch on the troubleshooting/monitoring I've been doing for over a month now. Is this normal behavior, or do I need to investigate further into logs to find a possible cause for this?

Again, any insight anyone could provide would be wonderful. I have nowhere to go next and this is a growing concern for our organization.
Josh PriceApplication EngineerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Justin SmithSr. System EngineerCommented:
What is your Farm Build Number?  Find it in Central Admin - System - Servers in farm.
Josh PriceApplication EngineerAuthor Commented:
It is happening in all three of our environments (DEV, QA, PRD). The build numbers are:

DEV/QA: 14.0.6137.5002 (April 2013 CU)
PRD: 14.0.6112.5000 (October 2011)
Josh PriceApplication EngineerAuthor Commented:
I ended up rebuilding the User Profile Service Application to fix this issue. I looked around for ways to address this and never found any hope. We were only on that farm for another 3 months so I wasn't set back too much.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Josh PriceApplication EngineerAuthor Commented:
I was not receiving input from anyone regarding this question. I failed to find any valuable answers on TechNet and on EE. I hate having to recreate something to fix a problem - sometimes it's like using a nuke on pesky ground moles.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SharePoint

From novice to tech pro — start learning today.