Sharepoint 2010 User Profile Sync Stuck on Starting

Windows 2012 R2 Server
IIS 8.5
SharePoint 2010 Farm SP2

Late yesterday I noticed event Id error 6398 on my Server log then after further checking I found that the User Profile Sync Service was stuck on starting.

Stopped it started it still stuck on starting.

FIM services were started but now today they will not start either  event id 18456
logon failed for user mydom\spfarm reason failed to open the explicitly specified database client <named pipe>


All this was working two days ago  

Restarted two days ago after several windows updates. but after the restart the sites all came back up

The sites are working but functions are not 100%

Restarted the server earlier today but no change

I can not start
Forefront identity Manager Service
Forefront Identity Manger Synchronization Service  

The User Profile Synchronization Service is stopped and will not start


Please help


Thanks
LVL 23
Thomas GrassiSystems AdministratorAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Mohit NairSenior AssociateCommented:
Did you change the permission setting for farm account ? In Active directory, check if you have Replicate directory changes permission for Spfarm. Also share the event id error in detail.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Thomas GrassiSystems AdministratorAuthor Commented:
Mohit

Did you change the permission setting for farm account ?
No changes to the SPFARM account,

In Active directory, check if you have Replicate directory changes permission for Spfarm.

I found a script that checks this and it was not set so I found instructions to set Replicate directory Changes and added SPFARM to those.

PS C:\util> .\checkrdc our\spfarm
User 'our\spfarm':
        has a 'Replicating Directory Changes' permission on 'DC=our,DC=network,D
C=tgcsnet,DC=com'
        has a 'Replicating Directory Changes' permission on 'CN=Configuration,DC
=our,DC=network,DC=tgcsnet,DC=com'
PS C:\util>

Also share the event id error in detail.

Starts off with Event ID 3  then Event Id 26,  Event Id 2, then Event Id 3 again and then Event Id 0


This happens when I try to start Forefront Identity Manager Service

Log Name:      Application
Source:        Forefront Identity Manager
Date:          7/21/2015 7:00:00 AM
Event ID:      3
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SERV013.FQDN.com
Description:
.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: Could not find stored procedure 'RegisterService'.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Microsoft.ResourceManagement.Data.DataAccess.RegisterService(String hostName)
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Forefront Identity Manager" />
    <EventID Qualifiers="0">3</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-07-21T11:00:00.000000000Z" />
    <EventRecordID>36941</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SERV013.FQDN.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: Could not find stored procedure 'RegisterService'.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Microsoft.ResourceManagement.Data.DataAccess.RegisterService(String hostName)</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Microsoft.ResourceManagement.ServiceHealthSource
Date:          7/21/2015 7:00:00 AM
Event ID:      26
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SERV013.FQDN.com
Description:
The Forefront Identity Manager Service was not able to initialize a timer necessary for supporting the execution of workflows.

Upon startup, the Forefront Identity Manager Service must initialize and set a timer to support workflow execution.  If this timer fails to get created, workflows will not run successfully and there is no recovery other than to stop and start the Forefront Identity Manager Service.

Restart the Forefront Identity Manager Service.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft.ResourceManagement.ServiceHealthSource" />
    <EventID Qualifiers="0">26</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-07-21T11:00:00.000000000Z" />
    <EventRecordID>36942</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SERV013.FQDN.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>The Forefront Identity Manager Service was not able to initialize a timer necessary for supporting the execution of workflows.

Upon startup, the Forefront Identity Manager Service must initialize and set a timer to support workflow execution.  If this timer fails to get created, workflows will not run successfully and there is no recovery other than to stop and start the Forefront Identity Manager Service.

Restart the Forefront Identity Manager Service.</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Microsoft.ResourceManagement.ServiceHealthSource
Date:          7/21/2015 7:00:00 AM
Event ID:      2
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SERV013.FQDN.com
Description:
The Forefront Identity Manager Service could not bind to its endpoints.  This failure prevents clients from communicating with the Web services.

A most likely cause for the failure is another service, possibly another instance of Forefront Identity Manager Service, has already bound to the endpoint.  Another, less likely cause, is that the account under which the service runs does not have permission to bind to endpoints.

Ensure that no other processes have bound to that endpoint and that the service account has permission to bind endpoints.  Further, check the application configuration file to ensure the Forefront Identity Manager Service is binding to the correct endpoints.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft.ResourceManagement.ServiceHealthSource" />
    <EventID Qualifiers="0">2</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-07-21T11:00:00.000000000Z" />
    <EventRecordID>36943</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SERV013.FQDN.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>The Forefront Identity Manager Service could not bind to its endpoints.  This failure prevents clients from communicating with the Web services.

A most likely cause for the failure is another service, possibly another instance of Forefront Identity Manager Service, has already bound to the endpoint.  Another, less likely cause, is that the account under which the service runs does not have permission to bind to endpoints.

Ensure that no other processes have bound to that endpoint and that the service account has permission to bind endpoints.  Further, check the application configuration file to ensure the Forefront Identity Manager Service is binding to the correct endpoints.</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Forefront Identity Manager
Date:          7/21/2015 7:00:00 AM
Event ID:      3
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SERV013.FQDN.com
Description:
.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: Could not find stored procedure 'RegisterService'.
   at Microsoft.ResourceManagement.Utilities.ExceptionManager.ThrowException(Exception exception)
   at Microsoft.ResourceManagement.Data.Exception.DataAccessExceptionManager.ThrowException(SqlException innerException)
   at Microsoft.ResourceManagement.Data.DataAccess.RegisterService(String hostName)
   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.RegisterService(String hostName)
   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.Initialize()
   at Microsoft.ResourceManagement.WebServices.ResourceManagementServiceHostFactory.CreateServiceHost(String constructorString, Uri[] baseAddresses)
   at Microsoft.ResourceManagement.WindowsHostService.OnStart(String[] args)
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Forefront Identity Manager" />
    <EventID Qualifiers="0">3</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-07-21T11:00:00.000000000Z" />
    <EventRecordID>36944</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SERV013.FQDN.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: Could not find stored procedure 'RegisterService'.
   at Microsoft.ResourceManagement.Utilities.ExceptionManager.ThrowException(Exception exception)
   at Microsoft.ResourceManagement.Data.Exception.DataAccessExceptionManager.ThrowException(SqlException innerException)
   at Microsoft.ResourceManagement.Data.DataAccess.RegisterService(String hostName)
   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.RegisterService(String hostName)
   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.Initialize()
   at Microsoft.ResourceManagement.WebServices.ResourceManagementServiceHostFactory.CreateServiceHost(String constructorString, Uri[] baseAddresses)
   at Microsoft.ResourceManagement.WindowsHostService.OnStart(String[] args)</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Microsoft Resource Management Service
Date:          7/21/2015 7:00:00 AM
Event ID:      0
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SERV013.FQDN.com
Description:
Service cannot be started. System.Data.SqlClient.SqlException: Could not find stored procedure 'RegisterService'.
   at Microsoft.ResourceManagement.WindowsHostService.OnStart(String[] args)
   at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft Resource Management Service" />
    <EventID Qualifiers="0">0</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-07-21T11:00:00.000000000Z" />
    <EventRecordID>36945</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SERV013.FQDN.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Service cannot be started. System.Data.SqlClient.SqlException: Could not find stored procedure 'RegisterService'.
   at Microsoft.ResourceManagement.WindowsHostService.OnStart(String[] args)
   at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)</Data>
  </EventData>
</Event>


If we fix this then the User Profile Sync will start I think.



Thoughts
0
martushaProduct managerCommented:
Hello Tom,

From your other question I know you installed some patches for SharePoint. After installing updates the User Profile Service need to be reprovisioned. If you start service using not farm account, 99% it will be stucked on starting.
Now you need to run script to stop this service:

Get-SPServiceInstance | Where-Object {$_.TypeName -eq "User Profile Synchronization Service"}

(here you get the ID, for example "12cfac84-4f2a-457e-a8fe-0950ce7dd69d" and then run next one)

Stop-SPServiceInstance 12cfac84-4f2a-457e-a8fe-0950ce7dd69d

After this check in central admin services on server if UPS is stopped.

Is stopped, you need to add spfarm account to local admins on server;
Add spFarm to „Allow log on locally“;
Add spFarm  to „Log on as service“.

Then log on with spFarm, then log of, then log on again (for user policies applied correctly) and in Central Administration on services on server start:
User Profile Service, then started,
User Profile Synchronization Service (wait about 2-10 min while starting), then started,
IISRESET
Log off
Log on with your admin account
Remove domain\spFarm from local admins

Hope this procedure helps to start UPS.
0
Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

Thomas GrassiSystems AdministratorAuthor Commented:
Martusha

Hope you had a nice vacation.

UPS was stopped.

SPFARM account is a member of Local Admins on the server.
Added SPFARM to Allow Log on Locally in the Default Domain Policy
SPFARM was already in Log on as service

did a gpupdate /force

1. Log on to server as Mydom\ SPFARM
2. Logoff from server
3. Log on to server again as Mydom\SPFARM
4. in Central Administration Started UPS

It has been over 30 minutes and the UPS still shows starting.

ran "Get-SPServiceInstance " 

PS C:\Users\spfarm> get-spserviceinstance

TypeName                         Status   Id
--------                         ------   --
User Profile Synchronization ... Provi... a4f565c9-e68e-4fcd-9a4e-001b0ce591e4


Note:

Event ID 3  then Event Id 26,  Event Id 2, then Event Id 3 again and then Event Id 0 did not appear this time

Both FIM services are now started.

So now we just need to get UPS started


Thoughts?
0
martushaProduct managerCommented:
On my head your UPS becomes corrupted. Maybe the best approach to recreate it.
The procedure would be this:

I. Copy and save those settings:

1. AD connection settings

2. connection filters, DBC

3. preffered search center

4. Mysite host (and personal path)

5. Audiencies (all other added manualy)

6. Profile, Social and Sync DB names (to give the same name while recreating)



II. Delete UPS and Sync DB:

1. CA->Manage secive Applications ->User Profile Service Application -> delete (DO NOT check "delete data association)

2. MS SQL:
there are 3 UPS related DBs:

    Profile DB - stores user info ad activity

    Social DB - stores tags and notes

    Sync DB - staging between AD and SP while it syncing

Delete only Sync DB (check "close existing connections)

 

III. Recreate UPS application:

1. CA ->Manage Serice Application -> create new UPS Application

2. Enter DB names the same as was, my site host and path

3. Highlight created UPS Application and click "Administrators", add search crawler account with permission to "Retrieve People data for Search crawlers"

4. Recreate connections and other settings

5. Start full sync

Hope this would help.
0
Thomas GrassiSystems AdministratorAuthor Commented:
Martusha

That's what I did already

I deleted the entire UPS application and databases and started new again.

I do not believe it is corrupted since it has not been able to be used since I recreated it

Has to be a permissions problem or something like that


Thoughts?
0
martushaProduct managerCommented:
Check these Harbar's blog posts (it is bible for UPS):
http://www.harbar.net/articles/sp2010ups.aspx
http://www.harbar.net/articles/sp2010ups2.aspx
Maybe you will found something what helps you to fix the problem.
0
Thomas GrassiSystems AdministratorAuthor Commented:
Martusha,

Thanks Yes I have seen both of these before as I was doing my research also on this issue.

I will review them closer later today.

The second one says like I do it is a permission problem.

Will post results

If you can think of anything to check first that would be great.
0
Thomas GrassiSystems AdministratorAuthor Commented:
Martusha

Is it possible that the error Event ID 14 I am also getting can cause the Profile SYNC to have issues?

I have this issue open also
http://www.experts-exchange.com/questions/28700109/Sharepoint-2010-Event-Id-14-Every-Hour.html

if you can take a look
0
Thomas GrassiSystems AdministratorAuthor Commented:
Martusha


I followed your instructions above to delete the UPS and rebuild it.

The UPs is still stuck on starting

I did an iisreset /noforce  but no change.

What is wrong here?

Need help

Thanks
0
Thomas GrassiSystems AdministratorAuthor Commented:
Mohit/Martusha

Today was getting Event Id 5555 every hour on the server.

So I disabled all USER Profile Service Applications Job Definitions

The Job
User Profile Service Application - System Job to Manage User Profile Synchronization  is paused at 0%

I can delete and recreate again if needed but I need detailed instructions on the recreate side.

The Harbor site is good reference but it shows building the entire farm I do not want to do that just need to rebuild the UPS service correctly.

I am obviously missing a step if the job is sitting at 0%

Anyone with a suggestion?

Thanks
0
Thomas GrassiSystems AdministratorAuthor Commented:
Closing this out as unsolved but still awarding point because of the good effort.

I decided to Completed rebuild the Farm from the beginning.

Hopefully this will help

Thanks for all your help
1
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SharePoint

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.