Link to home
Start Free TrialLog in
Avatar of Member_2_6492660_1
Member_2_6492660_1Flag for United States of America

asked on

Sharepoint 2010 User Profile Sync Stuck on Starting

Windows 2012 R2 Server
IIS 8.5
SharePoint 2010 Farm SP2

Late yesterday I noticed event Id error 6398 on my Server log then after further checking I found that the User Profile Sync Service was stuck on starting.

Stopped it started it still stuck on starting.

FIM services were started but now today they will not start either  event id 18456
logon failed for user mydom\spfarm reason failed to open the explicitly specified database client <named pipe>


All this was working two days ago  

Restarted two days ago after several windows updates. but after the restart the sites all came back up

The sites are working but functions are not 100%

Restarted the server earlier today but no change

I can not start
Forefront identity Manager Service
Forefront Identity Manger Synchronization Service  

The User Profile Synchronization Service is stopped and will not start


Please help


Thanks
ASKER CERTIFIED SOLUTION
Avatar of Mohit Nair
Mohit Nair
Flag of India image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Member_2_6492660_1

ASKER

Mohit

Did you change the permission setting for farm account ?
No changes to the SPFARM account,

In Active directory, check if you have Replicate directory changes permission for Spfarm.

I found a script that checks this and it was not set so I found instructions to set Replicate directory Changes and added SPFARM to those.

PS C:\util> .\checkrdc our\spfarm
User 'our\spfarm':
        has a 'Replicating Directory Changes' permission on 'DC=our,DC=network,D
C=tgcsnet,DC=com'
        has a 'Replicating Directory Changes' permission on 'CN=Configuration,DC
=our,DC=network,DC=tgcsnet,DC=com'
PS C:\util>

Also share the event id error in detail.

Starts off with Event ID 3  then Event Id 26,  Event Id 2, then Event Id 3 again and then Event Id 0


This happens when I try to start Forefront Identity Manager Service

Log Name:      Application
Source:        Forefront Identity Manager
Date:          7/21/2015 7:00:00 AM
Event ID:      3
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SERV013.FQDN.com
Description:
.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: Could not find stored procedure 'RegisterService'.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Microsoft.ResourceManagement.Data.DataAccess.RegisterService(String hostName)
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Forefront Identity Manager" />
    <EventID Qualifiers="0">3</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-07-21T11:00:00.000000000Z" />
    <EventRecordID>36941</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SERV013.FQDN.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: Could not find stored procedure 'RegisterService'.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Microsoft.ResourceManagement.Data.DataAccess.RegisterService(String hostName)</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Microsoft.ResourceManagement.ServiceHealthSource
Date:          7/21/2015 7:00:00 AM
Event ID:      26
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SERV013.FQDN.com
Description:
The Forefront Identity Manager Service was not able to initialize a timer necessary for supporting the execution of workflows.

Upon startup, the Forefront Identity Manager Service must initialize and set a timer to support workflow execution.  If this timer fails to get created, workflows will not run successfully and there is no recovery other than to stop and start the Forefront Identity Manager Service.

Restart the Forefront Identity Manager Service.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft.ResourceManagement.ServiceHealthSource" />
    <EventID Qualifiers="0">26</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-07-21T11:00:00.000000000Z" />
    <EventRecordID>36942</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SERV013.FQDN.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>The Forefront Identity Manager Service was not able to initialize a timer necessary for supporting the execution of workflows.

Upon startup, the Forefront Identity Manager Service must initialize and set a timer to support workflow execution.  If this timer fails to get created, workflows will not run successfully and there is no recovery other than to stop and start the Forefront Identity Manager Service.

Restart the Forefront Identity Manager Service.</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Microsoft.ResourceManagement.ServiceHealthSource
Date:          7/21/2015 7:00:00 AM
Event ID:      2
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SERV013.FQDN.com
Description:
The Forefront Identity Manager Service could not bind to its endpoints.  This failure prevents clients from communicating with the Web services.

A most likely cause for the failure is another service, possibly another instance of Forefront Identity Manager Service, has already bound to the endpoint.  Another, less likely cause, is that the account under which the service runs does not have permission to bind to endpoints.

Ensure that no other processes have bound to that endpoint and that the service account has permission to bind endpoints.  Further, check the application configuration file to ensure the Forefront Identity Manager Service is binding to the correct endpoints.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft.ResourceManagement.ServiceHealthSource" />
    <EventID Qualifiers="0">2</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-07-21T11:00:00.000000000Z" />
    <EventRecordID>36943</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SERV013.FQDN.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>The Forefront Identity Manager Service could not bind to its endpoints.  This failure prevents clients from communicating with the Web services.

A most likely cause for the failure is another service, possibly another instance of Forefront Identity Manager Service, has already bound to the endpoint.  Another, less likely cause, is that the account under which the service runs does not have permission to bind to endpoints.

Ensure that no other processes have bound to that endpoint and that the service account has permission to bind endpoints.  Further, check the application configuration file to ensure the Forefront Identity Manager Service is binding to the correct endpoints.</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Forefront Identity Manager
Date:          7/21/2015 7:00:00 AM
Event ID:      3
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SERV013.FQDN.com
Description:
.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: Could not find stored procedure 'RegisterService'.
   at Microsoft.ResourceManagement.Utilities.ExceptionManager.ThrowException(Exception exception)
   at Microsoft.ResourceManagement.Data.Exception.DataAccessExceptionManager.ThrowException(SqlException innerException)
   at Microsoft.ResourceManagement.Data.DataAccess.RegisterService(String hostName)
   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.RegisterService(String hostName)
   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.Initialize()
   at Microsoft.ResourceManagement.WebServices.ResourceManagementServiceHostFactory.CreateServiceHost(String constructorString, Uri[] baseAddresses)
   at Microsoft.ResourceManagement.WindowsHostService.OnStart(String[] args)
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Forefront Identity Manager" />
    <EventID Qualifiers="0">3</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-07-21T11:00:00.000000000Z" />
    <EventRecordID>36944</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SERV013.FQDN.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: Could not find stored procedure 'RegisterService'.
   at Microsoft.ResourceManagement.Utilities.ExceptionManager.ThrowException(Exception exception)
   at Microsoft.ResourceManagement.Data.Exception.DataAccessExceptionManager.ThrowException(SqlException innerException)
   at Microsoft.ResourceManagement.Data.DataAccess.RegisterService(String hostName)
   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.RegisterService(String hostName)
   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.Initialize()
   at Microsoft.ResourceManagement.WebServices.ResourceManagementServiceHostFactory.CreateServiceHost(String constructorString, Uri[] baseAddresses)
   at Microsoft.ResourceManagement.WindowsHostService.OnStart(String[] args)</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Microsoft Resource Management Service
Date:          7/21/2015 7:00:00 AM
Event ID:      0
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SERV013.FQDN.com
Description:
Service cannot be started. System.Data.SqlClient.SqlException: Could not find stored procedure 'RegisterService'.
   at Microsoft.ResourceManagement.WindowsHostService.OnStart(String[] args)
   at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft Resource Management Service" />
    <EventID Qualifiers="0">0</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-07-21T11:00:00.000000000Z" />
    <EventRecordID>36945</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SERV013.FQDN.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Service cannot be started. System.Data.SqlClient.SqlException: Could not find stored procedure 'RegisterService'.
   at Microsoft.ResourceManagement.WindowsHostService.OnStart(String[] args)
   at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)</Data>
  </EventData>
</Event>


If we fix this then the User Profile Sync will start I think.



Thoughts
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Martusha

Hope you had a nice vacation.

UPS was stopped.

SPFARM account is a member of Local Admins on the server.
Added SPFARM to Allow Log on Locally in the Default Domain Policy
SPFARM was already in Log on as service

did a gpupdate /force

1. Log on to server as Mydom\ SPFARM
2. Logoff from server
3. Log on to server again as Mydom\SPFARM
4. in Central Administration Started UPS

It has been over 30 minutes and the UPS still shows starting.

ran "Get-SPServiceInstance " 

PS C:\Users\spfarm> get-spserviceinstance

TypeName                         Status   Id
--------                         ------   --
User Profile Synchronization ... Provi... a4f565c9-e68e-4fcd-9a4e-001b0ce591e4


Note:

Event ID 3  then Event Id 26,  Event Id 2, then Event Id 3 again and then Event Id 0 did not appear this time

Both FIM services are now started.

So now we just need to get UPS started


Thoughts?
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Martusha

That's what I did already

I deleted the entire UPS application and databases and started new again.

I do not believe it is corrupted since it has not been able to be used since I recreated it

Has to be a permissions problem or something like that


Thoughts?
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Martusha,

Thanks Yes I have seen both of these before as I was doing my research also on this issue.

I will review them closer later today.

The second one says like I do it is a permission problem.

Will post results

If you can think of anything to check first that would be great.
Martusha

Is it possible that the error Event ID 14 I am also getting can cause the Profile SYNC to have issues?

I have this issue open also
https://www.experts-exchange.com/questions/28700109/Sharepoint-2010-Event-Id-14-Every-Hour.html

if you can take a look
Martusha


I followed your instructions above to delete the UPS and rebuild it.

The UPs is still stuck on starting

I did an iisreset /noforce  but no change.

What is wrong here?

Need help

Thanks
Mohit/Martusha

Today was getting Event Id 5555 every hour on the server.

So I disabled all USER Profile Service Applications Job Definitions

The Job
User Profile Service Application - System Job to Manage User Profile Synchronization  is paused at 0%

I can delete and recreate again if needed but I need detailed instructions on the recreate side.

The Harbor site is good reference but it shows building the entire farm I do not want to do that just need to rebuild the UPS service correctly.

I am obviously missing a step if the job is sitting at 0%

Anyone with a suggestion?

Thanks
Closing this out as unsolved but still awarding point because of the good effort.

I decided to Completed rebuild the Farm from the beginning.

Hopefully this will help

Thanks for all your help