User Profile Synchronization Not Working

I'm running SharePoint 2010 Server SP1 June 2011 CU and this CU broke the user profile picture upload function so I decided to install the Feb 2012 CU to fix the issue.

I realize i had to restart the User Profile Synchronization service as part of the installation but didn't realize it required entering in the farm admin (spfarm) credentials which my predecessor did not document.  I was forced to change the password and this is when my problems started.

1. I changed spfarm PW through AD
2. Manually updated all app pool with new password
3. Manually updated all SharePoint services which used this account with the new password
4. Attempted to restart the UPS in central admin only to find it stuck at "Starting"
5. Rebooted serveral times, didn't work.
6. Restarted SP timer service, didn't work.
7. At this point I realized there's 2 other Windows services that uses spfarm:
     - Forefront Identity Manager Service
     - Forefront Identity Manager Synchronization Service
8. Changed the password and tried to start them.  The first service started *but* second one times out and refuses to start.  (I will post event ID and actual error message later).
9. Within Central Admin > Security > Manage Service Account and changed spfarm password again to a new one.
10. Got an error about an existing job already running by the same name so I was forced to delete it using powershell and finally the password change went through.  But as far as I can tell it doesn't fix the issue at hand.

I realize now UPS depends on this FIM service and that Microsoft does not recommend manually starting and/or modifying these 2 FIM services from within services.msc.

I'm really stuck here and need urgent assistance.
Thanks
LVL 1
randy915Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

randy915Author Commented:
I also stopped UPS via PowerShell which now causes it to be stuck at "Stopping".

Found this article, may be of help...
http://social.technet.microsoft.com/Forums/en-US/sharepoint2010setup/thread/6d1b123a-391b-4e9e-946d-c23027192e62/
randy915Author Commented:
Low hanging fruit - spfarm is already part of the local admin group.
Justin SmithSr. System EngineerCommented:
Quite a mess :)   If you ever need remote support, I'm available (email is in my profile).

What is the current status of the UPS service?  Is it still "starting" ?  Are all app pools running and can you access all sites including Central Admin?

Going forward, any time you change the password of the Farm account, update it using the stsadm -o updatefarmcredentials command.
IT Pros Agree: AI and Machine Learning Key

We’d all like to think our company’s data is well protected, but when you ask IT professionals they admit the data probably is not as safe as it could be.

randy915Author Commented:
Thanks ACH1LLES I'll keep that in mind.

Maybe some progress this morning:

1.Successfully stopped UPS by running:
stsadm -o provisionservice -action stop -servicetype "Microsoft.Office.Server.Admin
istration.ProfileSynchronizationService, Microsoft.Office.Server.UserProfiles, V
ersion=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" -servicename
FIMSynchronizationService

Open in new window


2. Did not run iisreset /noforce as instructed.
3. Noticed that both FIM services are stopped and "Disabled"
4. Tried to start UPS service and it was hanging at "Starting" for a good 20 min so I ran step 1 again to kill it.
5. Tried starting FIM Synchronization Service and after about a minute it errored out.  Event viewer logged this:

Source: FIMSynchronizationService
Event ID: 6324

The server encountered an unexpected error and stopped.
 
"BAIL: MMS(5180): storeimp.cpp(308): 0x80230443 (Service start up has failed.  Cannot open the FIM Synchronization Service database because the database schema version in existing database does not match the required version.)
ERR: MMS(5180): server.cpp(373): Failed to connect to the database Sync DB on sharepointsql
BAIL: MMS(5180): server.cpp(374): 0x80230443 (Service start up has failed.  Cannot open the FIM Synchronization Service database because the database schema version in existing database does not match the required version.)
BAIL: MMS(5180): server.cpp(3860): 0x80230443 (Service start up has failed.  Cannot open the FIM Synchronization Service database because the database schema version in existing database does not match the required version.)
BAIL: MMS(5180): service.cpp(1539): 0x80230443 (Service start up has failed.  Cannot open the FIM Synchronization Service database because the database schema version in existing database does not match the required version.)
ERR: MMS(5180): service.cpp(988): Error creating com objects. Error code: -2145188797. This is retry number 0.
BAIL: MMS(5180): clrhost.cpp(224): 0x80131022
BAIL: MMS(5180): scriptmanagerimpl.cpp(7670): 0x80131022
BAIL: MMS(5180): server.cpp(251): 0x80131022
BAIL: MMS(5180): server.cpp(3860): 0x80131022
BAIL: MMS(5180): service.cpp(1539): 0x80131022
ERR: MMS(5180): service.cpp(988): Error creating com objects. Error code: -2146234334. This is retry number 1.
BAIL: MMS(5180): clrhost.cpp(224): 0x80131022
BAIL: MMS(5180): scriptmanagerimpl.cpp(7670): 0x80131022
BAIL: MMS(5180): server.cpp(251): 0x80131022
BAIL: MMS(5180): server.cpp(3860): 0x80131022
BAIL: MMS(5180): service.cpp(1539): 0x80131022
ERR: MMS(5180): service.cpp(988): Error creating com objects. Error code: -2146234334. This is retry number 2.
BAIL: MMS(5180): clrhost.cpp(224): 0x80131022
BAIL: MMS(5180): scriptmanagerimpl.cpp(7670): 0x80131022
BAIL: MMS(5180): server.cpp(251): 0x80131022
BAIL: MMS(5180): server.cpp(3860): 0x80131022
BAIL: MMS(5180): service.cpp(1539): 0x80131022
ERR: MMS(5180): service.cpp(988): Error creating com objects. Error code: -2146234334. This is retry number 3.
BAIL: MMS(5180): service.cpp(1002): 0x80131022
Forefront Identity Manager 4.0.2450.47"

I wonder if the errors I got at the very end of the Feb2012 CU caused this whole mess because the SyncDB was not properly upgraded...

Do you think I should attempt the CU update again now that I have the correct password?
randy915Author Commented:
Justin SmithSr. System EngineerCommented:
Central Admin - Migration/Upgrade - Check Database STatus

What does it say for the Sync?
randy915Author Commented:
Sync DB         SynchronizationDatabase         No action required
Justin SmithSr. System EngineerCommented:
You might try running the config wizard one time.  You may want to reboot the box too.

Then go into services.msc and verify both ForeFront services are stopped - disabled - and set to run as Local System (if not Local System, go ahead and change to Local).

Go back into Central Admin and Start the Sync service (User Profile Service should already be running).  Jump back to services.msc and monitor the ForeFront services.  Keep hitting refresh and make a note of all changes that happen.  Then report back to here.
randy915Author Commented:
Thanks, I'll let you know the results Monday evening PST.
randy915Author Commented:
I tried to run Config Wizard and IISRESET several times and it kept failing at step 9/10:


Source: SharePoint 2010 Products Configuration Wizard
Event ID: 104

Failed to upgrade SharePoint Products.
An exception of type Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException was thrown.  Additional exception information: Failed to upgrade SharePoint Products.
Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException: Exception of type 'Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException' was thrown.
   at Microsoft.SharePoint.PostSetupConfiguration.UpgradeTask.Run()
   at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()


Found an article that helped me get through this http://www.quantumofgeek.com/2010/01/configuration-of-sharepoint-products-and-technologies-failed/

This site is currently down so the jist of it is:

1. Copy the web.config file from: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\CONFIG To: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\TEMPLATE\LAYOUTS.

2. Then run: psconfig -cmd upgrade -inplace b2b -force -wait

Upon re-running the configuration wizard via command line, my upgraded succeeded and returned. “Configuration of the SharePoint Products and Technologies has succeeded”!

3. Delete web.config from C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\TEMPLATE\LAYOUTS.

I then replaced back the original web.config.



After a reboot I again tried starting UPS in cent. admin and it's stuck at starting for at least 14 hours now so it's hosed.  The 2 forefront services are running as Local Account but they're currently disabled as per your instructions.
Justin SmithSr. System EngineerCommented:
Do you have a bunch of custom user properties?  Do you have several sync connections?  If the answer is No to both, you might just blow away your User Profile APp and recreate it.

I highly recommend you review these articles:  

http://www.harbar.net/articles/sp2010ups.aspx 
http://www.harbar.net/articles/sp2010ups2.aspx

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
randy915Author Commented:
I don't think there's anything customized since my predecessor installed pretty much a standard OOB farm.  I saw a lot of people recommending blowing away UPS and recreating but during this process will it affect People Search in terms of querying existing users?
Justin SmithSr. System EngineerCommented:
People Search should still return results, but clicking on someone will result in an error.
randy915Author Commented:
Ok thanks for the articles, will keep you posted!
randy915Author Commented:
Couple of questions:

1.  How should I go about removing the old junk before I embark on recreating UPS?  I know there's a Sync DB and Profile DB on SQL as well as the 2 FIM services, the UPS SharePoint service, etc.  Do I just delete them manually or is there a more elegant way or does the process that article highlight overwrite them?

2. I don't have the password for spservices and will need to change it, what's best practice on updating?  I see that it's being used by multiple services like Web Analytics, Server/Foundation Search, Document Conversion Load Balancer, and Claims to Windows Token Service.
randy915Author Commented:
I found this gem, which seems to answer my 1st question:
http://social.technet.microsoft.com/Forums/br/sharepoint2010setup/thread/2011e0a0-906b-40d5-86fd-490a4deb9d6f

The recommendation from Microsoft product team is to apply at least June CU! You can apply December CU if you like. I have tried it and it works perfectly! What you are experiencing is that your sync DB is not upgraded. For that you need to first of all take a screen shot of your user profile settings and stop both user profile and user profile synchronization service. Then go ahead and delete the service application WITHOUT deleting the databases! When that is done, go to your SQL box and delete ONLY sync DB! When it's done go back to your server and create the user profile service application. Fill in exactly the same names and values that you have in your screen shots! SharePoint will create only Sync DB and will keep Social DB and Profile DB. The sync DB will then get upgraded correctly. Then try to start the services and hopefully you will manage to start the service correctly! Please let me know if it doesn't help you so we can troubleshoot this together!
randy915Author Commented:
Nvm regarding #2, I've changed the password using Security > Managed Account.
Justin SmithSr. System EngineerCommented:
Stop the Sync Service, Stop the User Profile Service.  Blow away the User Profile Service APplication.  Go into SQL and verify all three db's (profile, social, sync) were deleted.  

Then create a new one.
randy915Author Commented:
What about the 2 FIM services, should I leave them disabled or on automatic?
Justin SmithSr. System EngineerCommented:
Leave them.  Never touch them.
randy915Author Commented:
Didn't work...

1. Stopped the User Profile Sync Service using the command I mentioned above (that was the only way).
2. Stopped the User Profile Service.
3. Deleted the User Profile Service Application including removing DB.
4. Verified in SQL that three DB's (profile, social, sync) were deleted.
5. IISRESET
6. Changed the 2 FIM services to Automatic since they were disabled due to step 1
6. Started the User Profile Service.
7. Started the User Profile Sync Service, waited over 30 minutes, nothing happened.
randy915Author Commented:
I tried re-running the Product Configuration Wizard which only has 9 steps now instead of 10.   I noticed it now asks about whether I want to disconnect from farm and whether I want to re-use existing server to host central admin.

Anyhow I kept all existing settings and it succeeded and opened up Cent Admin prompting me to either use wizard to proceed or cancel.  I opted for Cancel which brought me back to Cent Admin.

Tried start UPS, again stuck at "Starting" and it's been 14 minutes now.  FIM services still on Automatic but not started.

I will next try yet again to recreate the User Profile app and DB.
randy915Author Commented:
Nope, nothing.

FIM and UPS does not start... pulling my hair out now.

For creating a new UPS, it asks for the URL of this site and I put "http://my.ourdomain.com", this site has always been disabled in IIS (not sure why); but in the big picture this shouldn't matter in terms of getting UPS up and running right?

I also chose the first option "User name (do not resolve conflicts", see here:
http://www.harbar.net/images/www_harbar_net/WindowsLiveWriter/RationalGuidetoimplementingSharePointSer_F266/27-04-2010%2017-03-49_2.png

This also shouldn't matter so I chose default.  I'm on the verge of bringing in Microsoft but if you have any more ideas I'd love to hear it.  Also please email me at randy915@gmail.com if you're interested in doing remote support and if you have a rate please note that as well in the email.

Thanks for your help and patience in troubleshooting this with me.

-Randy
Justin SmithSr. System EngineerCommented:
Just sent you a message.  Hang in there.
randy915Author Commented:
ACH1LLES's tip is correct, the best method is to delete the User Profile Synchronization app completely but sometimes in order to do that you need to hack it a bit:

- the STSADM command mentioned earlier has proved very useful in killing the job when it's stuck in "Starting" or "Stopping" if reboot doesn't solve it.

- in the timer job status page, there maybe stuck jobs under the category "One Time", those should not be in there so make sure they're all deleted before you attempt to recreate the UPS app and DB's.

- always delete the DB's if you're going to delete the UPS app

- make sure the spservices account has the correct AD delegate permissions as mentioned in the articles

- make sure spfarm and spservices are part of the AD security group "Pre-Windows 2000 Compatible Access"

- keep both FIM services "Disabled" and change the log on account to "Local System", if your UPS app is properly provisioned, part of its ~10min startup process is to setup the FIM services and start them up.  As ACH1LLES mentioned, DO NOT TOUCH THEM.

Good luck.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SharePoint

From novice to tech pro — start learning today.