Solved

User Profile Synchronization Not Working

Posted on 2012-03-21
25
10,519 Views
Last Modified: 2012-08-13
I'm running SharePoint 2010 Server SP1 June 2011 CU and this CU broke the user profile picture upload function so I decided to install the Feb 2012 CU to fix the issue.

I realize i had to restart the User Profile Synchronization service as part of the installation but didn't realize it required entering in the farm admin (spfarm) credentials which my predecessor did not document.  I was forced to change the password and this is when my problems started.

1. I changed spfarm PW through AD
2. Manually updated all app pool with new password
3. Manually updated all SharePoint services which used this account with the new password
4. Attempted to restart the UPS in central admin only to find it stuck at "Starting"
5. Rebooted serveral times, didn't work.
6. Restarted SP timer service, didn't work.
7. At this point I realized there's 2 other Windows services that uses spfarm:
     - Forefront Identity Manager Service
     - Forefront Identity Manager Synchronization Service
8. Changed the password and tried to start them.  The first service started *but* second one times out and refuses to start.  (I will post event ID and actual error message later).
9. Within Central Admin > Security > Manage Service Account and changed spfarm password again to a new one.
10. Got an error about an existing job already running by the same name so I was forced to delete it using powershell and finally the password change went through.  But as far as I can tell it doesn't fix the issue at hand.

I realize now UPS depends on this FIM service and that Microsoft does not recommend manually starting and/or modifying these 2 FIM services from within services.msc.

I'm really stuck here and need urgent assistance.
Thanks
0
Comment
Question by:randy915
  • 17
  • 8
25 Comments
 
LVL 1

Author Comment

by:randy915
ID: 37750999
I also stopped UPS via PowerShell which now causes it to be stuck at "Stopping".

Found this article, may be of help...
http://social.technet.microsoft.com/Forums/en-US/sharepoint2010setup/thread/6d1b123a-391b-4e9e-946d-c23027192e62/
0
 
LVL 1

Author Comment

by:randy915
ID: 37751009
Low hanging fruit - spfarm is already part of the local admin group.
0
 
LVL 38

Expert Comment

by:Justin Smith
ID: 37752063
Quite a mess :)   If you ever need remote support, I'm available (email is in my profile).

What is the current status of the UPS service?  Is it still "starting" ?  Are all app pools running and can you access all sites including Central Admin?

Going forward, any time you change the password of the Farm account, update it using the stsadm -o updatefarmcredentials command.
0
 
LVL 1

Author Comment

by:randy915
ID: 37753990
Thanks ACH1LLES I'll keep that in mind.

Maybe some progress this morning:

1.Successfully stopped UPS by running:
stsadm -o provisionservice -action stop -servicetype "Microsoft.Office.Server.Admin
istration.ProfileSynchronizationService, Microsoft.Office.Server.UserProfiles, V
ersion=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" -servicename
FIMSynchronizationService

Open in new window


2. Did not run iisreset /noforce as instructed.
3. Noticed that both FIM services are stopped and "Disabled"
4. Tried to start UPS service and it was hanging at "Starting" for a good 20 min so I ran step 1 again to kill it.
5. Tried starting FIM Synchronization Service and after about a minute it errored out.  Event viewer logged this:

Source: FIMSynchronizationService
Event ID: 6324

The server encountered an unexpected error and stopped.
 
"BAIL: MMS(5180): storeimp.cpp(308): 0x80230443 (Service start up has failed.  Cannot open the FIM Synchronization Service database because the database schema version in existing database does not match the required version.)
ERR: MMS(5180): server.cpp(373): Failed to connect to the database Sync DB on sharepointsql
BAIL: MMS(5180): server.cpp(374): 0x80230443 (Service start up has failed.  Cannot open the FIM Synchronization Service database because the database schema version in existing database does not match the required version.)
BAIL: MMS(5180): server.cpp(3860): 0x80230443 (Service start up has failed.  Cannot open the FIM Synchronization Service database because the database schema version in existing database does not match the required version.)
BAIL: MMS(5180): service.cpp(1539): 0x80230443 (Service start up has failed.  Cannot open the FIM Synchronization Service database because the database schema version in existing database does not match the required version.)
ERR: MMS(5180): service.cpp(988): Error creating com objects. Error code: -2145188797. This is retry number 0.
BAIL: MMS(5180): clrhost.cpp(224): 0x80131022
BAIL: MMS(5180): scriptmanagerimpl.cpp(7670): 0x80131022
BAIL: MMS(5180): server.cpp(251): 0x80131022
BAIL: MMS(5180): server.cpp(3860): 0x80131022
BAIL: MMS(5180): service.cpp(1539): 0x80131022
ERR: MMS(5180): service.cpp(988): Error creating com objects. Error code: -2146234334. This is retry number 1.
BAIL: MMS(5180): clrhost.cpp(224): 0x80131022
BAIL: MMS(5180): scriptmanagerimpl.cpp(7670): 0x80131022
BAIL: MMS(5180): server.cpp(251): 0x80131022
BAIL: MMS(5180): server.cpp(3860): 0x80131022
BAIL: MMS(5180): service.cpp(1539): 0x80131022
ERR: MMS(5180): service.cpp(988): Error creating com objects. Error code: -2146234334. This is retry number 2.
BAIL: MMS(5180): clrhost.cpp(224): 0x80131022
BAIL: MMS(5180): scriptmanagerimpl.cpp(7670): 0x80131022
BAIL: MMS(5180): server.cpp(251): 0x80131022
BAIL: MMS(5180): server.cpp(3860): 0x80131022
BAIL: MMS(5180): service.cpp(1539): 0x80131022
ERR: MMS(5180): service.cpp(988): Error creating com objects. Error code: -2146234334. This is retry number 3.
BAIL: MMS(5180): service.cpp(1002): 0x80131022
Forefront Identity Manager 4.0.2450.47"

I wonder if the errors I got at the very end of the Feb2012 CU caused this whole mess because the SyncDB was not properly upgraded...

Do you think I should attempt the CU update again now that I have the correct password?
0
 
LVL 1

Author Comment

by:randy915
ID: 37754210
0
 
LVL 38

Expert Comment

by:Justin Smith
ID: 37754727
Central Admin - Migration/Upgrade - Check Database STatus

What does it say for the Sync?
0
 
LVL 1

Author Comment

by:randy915
ID: 37754965
Sync DB         SynchronizationDatabase         No action required
0
 
LVL 38

Expert Comment

by:Justin Smith
ID: 37756822
You might try running the config wizard one time.  You may want to reboot the box too.

Then go into services.msc and verify both ForeFront services are stopped - disabled - and set to run as Local System (if not Local System, go ahead and change to Local).

Go back into Central Admin and Start the Sync service (User Profile Service should already be running).  Jump back to services.msc and monitor the ForeFront services.  Keep hitting refresh and make a note of all changes that happen.  Then report back to here.
0
 
LVL 1

Author Comment

by:randy915
ID: 37758227
Thanks, I'll let you know the results Monday evening PST.
0
 
LVL 1

Author Comment

by:randy915
ID: 37767142
I tried to run Config Wizard and IISRESET several times and it kept failing at step 9/10:


Source: SharePoint 2010 Products Configuration Wizard
Event ID: 104

Failed to upgrade SharePoint Products.
An exception of type Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException was thrown.  Additional exception information: Failed to upgrade SharePoint Products.
Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException: Exception of type 'Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException' was thrown.
   at Microsoft.SharePoint.PostSetupConfiguration.UpgradeTask.Run()
   at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()


Found an article that helped me get through this http://www.quantumofgeek.com/2010/01/configuration-of-sharepoint-products-and-technologies-failed/

This site is currently down so the jist of it is:

1. Copy the web.config file from: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\CONFIG To: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\TEMPLATE\LAYOUTS.

2. Then run: psconfig -cmd upgrade -inplace b2b -force -wait

Upon re-running the configuration wizard via command line, my upgraded succeeded and returned. “Configuration of the SharePoint Products and Technologies has succeeded”!

3. Delete web.config from C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\TEMPLATE\LAYOUTS.

I then replaced back the original web.config.



After a reboot I again tried starting UPS in cent. admin and it's stuck at starting for at least 14 hours now so it's hosed.  The 2 forefront services are running as Local Account but they're currently disabled as per your instructions.
0
 
LVL 38

Accepted Solution

by:
Justin Smith earned 500 total points
ID: 37767160
Do you have a bunch of custom user properties?  Do you have several sync connections?  If the answer is No to both, you might just blow away your User Profile APp and recreate it.

I highly recommend you review these articles:  

http://www.harbar.net/articles/sp2010ups.aspx
http://www.harbar.net/articles/sp2010ups2.aspx
0
 
LVL 1

Author Comment

by:randy915
ID: 37767247
I don't think there's anything customized since my predecessor installed pretty much a standard OOB farm.  I saw a lot of people recommending blowing away UPS and recreating but during this process will it affect People Search in terms of querying existing users?
0
6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

 
LVL 38

Expert Comment

by:Justin Smith
ID: 37767260
People Search should still return results, but clicking on someone will result in an error.
0
 
LVL 1

Author Comment

by:randy915
ID: 37767276
Ok thanks for the articles, will keep you posted!
0
 
LVL 1

Author Comment

by:randy915
ID: 37768701
Couple of questions:

1.  How should I go about removing the old junk before I embark on recreating UPS?  I know there's a Sync DB and Profile DB on SQL as well as the 2 FIM services, the UPS SharePoint service, etc.  Do I just delete them manually or is there a more elegant way or does the process that article highlight overwrite them?

2. I don't have the password for spservices and will need to change it, what's best practice on updating?  I see that it's being used by multiple services like Web Analytics, Server/Foundation Search, Document Conversion Load Balancer, and Claims to Windows Token Service.
0
 
LVL 1

Author Comment

by:randy915
ID: 37768762
I found this gem, which seems to answer my 1st question:
http://social.technet.microsoft.com/Forums/br/sharepoint2010setup/thread/2011e0a0-906b-40d5-86fd-490a4deb9d6f

The recommendation from Microsoft product team is to apply at least June CU! You can apply December CU if you like. I have tried it and it works perfectly! What you are experiencing is that your sync DB is not upgraded. For that you need to first of all take a screen shot of your user profile settings and stop both user profile and user profile synchronization service. Then go ahead and delete the service application WITHOUT deleting the databases! When that is done, go to your SQL box and delete ONLY sync DB! When it's done go back to your server and create the user profile service application. Fill in exactly the same names and values that you have in your screen shots! SharePoint will create only Sync DB and will keep Social DB and Profile DB. The sync DB will then get upgraded correctly. Then try to start the services and hopefully you will manage to start the service correctly! Please let me know if it doesn't help you so we can troubleshoot this together!
0
 
LVL 1

Author Comment

by:randy915
ID: 37768835
Nvm regarding #2, I've changed the password using Security > Managed Account.
0
 
LVL 38

Expert Comment

by:Justin Smith
ID: 37769326
Stop the Sync Service, Stop the User Profile Service.  Blow away the User Profile Service APplication.  Go into SQL and verify all three db's (profile, social, sync) were deleted.  

Then create a new one.
0
 
LVL 1

Author Comment

by:randy915
ID: 37769733
What about the 2 FIM services, should I leave them disabled or on automatic?
0
 
LVL 38

Expert Comment

by:Justin Smith
ID: 37771885
Leave them.  Never touch them.
0
 
LVL 1

Author Comment

by:randy915
ID: 37774610
Didn't work...

1. Stopped the User Profile Sync Service using the command I mentioned above (that was the only way).
2. Stopped the User Profile Service.
3. Deleted the User Profile Service Application including removing DB.
4. Verified in SQL that three DB's (profile, social, sync) were deleted.
5. IISRESET
6. Changed the 2 FIM services to Automatic since they were disabled due to step 1
6. Started the User Profile Service.
7. Started the User Profile Sync Service, waited over 30 minutes, nothing happened.
0
 
LVL 1

Author Comment

by:randy915
ID: 37774837
I tried re-running the Product Configuration Wizard which only has 9 steps now instead of 10.   I noticed it now asks about whether I want to disconnect from farm and whether I want to re-use existing server to host central admin.

Anyhow I kept all existing settings and it succeeded and opened up Cent Admin prompting me to either use wizard to proceed or cancel.  I opted for Cancel which brought me back to Cent Admin.

Tried start UPS, again stuck at "Starting" and it's been 14 minutes now.  FIM services still on Automatic but not started.

I will next try yet again to recreate the User Profile app and DB.
0
 
LVL 1

Author Comment

by:randy915
ID: 37774884
Nope, nothing.

FIM and UPS does not start... pulling my hair out now.

For creating a new UPS, it asks for the URL of this site and I put "http://my.ourdomain.com", this site has always been disabled in IIS (not sure why); but in the big picture this shouldn't matter in terms of getting UPS up and running right?

I also chose the first option "User name (do not resolve conflicts", see here:
http://www.harbar.net/images/www_harbar_net/WindowsLiveWriter/RationalGuidetoimplementingSharePointSer_F266/27-04-2010%2017-03-49_2.png

This also shouldn't matter so I chose default.  I'm on the verge of bringing in Microsoft but if you have any more ideas I'd love to hear it.  Also please email me at randy915@gmail.com if you're interested in doing remote support and if you have a rate please note that as well in the email.

Thanks for your help and patience in troubleshooting this with me.

-Randy
0
 
LVL 38

Expert Comment

by:Justin Smith
ID: 37776204
Just sent you a message.  Hang in there.
0
 
LVL 1

Author Closing Comment

by:randy915
ID: 37782983
ACH1LLES's tip is correct, the best method is to delete the User Profile Synchronization app completely but sometimes in order to do that you need to hack it a bit:

- the STSADM command mentioned earlier has proved very useful in killing the job when it's stuck in "Starting" or "Stopping" if reboot doesn't solve it.

- in the timer job status page, there maybe stuck jobs under the category "One Time", those should not be in there so make sure they're all deleted before you attempt to recreate the UPS app and DB's.

- always delete the DB's if you're going to delete the UPS app

- make sure the spservices account has the correct AD delegate permissions as mentioned in the articles

- make sure spfarm and spservices are part of the AD security group "Pre-Windows 2000 Compatible Access"

- keep both FIM services "Disabled" and change the log on account to "Local System", if your UPS app is properly provisioned, part of its ~10min startup process is to setup the FIM services and start them up.  As ACH1LLES mentioned, DO NOT TOUCH THEM.

Good luck.
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Microsoft SharePoint Foundation 2010 and Microsoft SharePoint Server 2010 do not offer the option to configure the location of the SharePoint diagnostic trace log files during installation.  This can, however, be configured through Central Administr…
Summary In SharePoint 2010 it is easy to create custom color themes to jazz up a site. Theme colors can also be created in PowerPoint 2010 with a few clicks. But how do the chosen colors actually look in the SharePoint site? The attached PowerPoint…
Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now