SCOM 2012 R2 Data Access Service Starts and then Stops

Here is the error seen by SCOM Management Console:

Microsoft.EnterpriseManagement.Common.ServiceNotRunningException: The Data Access service is either not running or not yet initialized. Check the event log for more information. ---> System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://v-scom-1:5724/DispatcherService. The connection attempt lasted for a time span of 00:00:01.0008624. TCP error code 10061: No connection could be made because the target machine actively refused it 172.16.77.162:5724.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 172.16.77.162:5724
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.EnterpriseManagement.Common.Internal.IDispatcherService.Connect(SdkClientConnectionOptions connectionOptions)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Initialize(EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.CreateEndpoint[T](EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
   --- End of inner exception stack trace ---
   at Microsoft.EnterpriseManagement.Common.Internal.ExceptionHandlers.HandleChannelExceptions(Exception ex)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.CreateEndpoint[T](EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.ConstructEnterpriseManagementGroupInternal[T,P](EnterpriseManagementConnectionSettings connectionSettings, ClientDataAccessCore clientCallback)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.RetrieveEnterpriseManagementGroupInternal[T,P](EnterpriseManagementConnectionSettings connectionSettings, ClientDataAccessCore callbackDispatcherService)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Connect[T,P](EnterpriseManagementConnectionSettings connectionSettings, ClientDataAccessCore callbackDispatcherService)
   at Microsoft.EnterpriseManagement.ManagementGroup.InternalInitialize(EnterpriseManagementConnectionSettings connectionSettings, ManagementGroupInternal internals)
   at Microsoft.EnterpriseManagement.Mom.Internal.UI.Common.ManagementGroupSessionManager.Connect(String server)
   at Microsoft.EnterpriseManagement.Monitoring.Console.Internal.ConsoleWindowBase.TryConnectToManagementGroupJob(Object sender, ConsoleJobEventArgs args)
System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://v-scom-1:5724/DispatcherService. The connection attempt lasted for a time span of 00:00:01.0008624. TCP error code 10061: No connection could be made because the target machine actively refused it 172.16.77.162:5724.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 172.16.77.162:5724
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.EnterpriseManagement.Common.Internal.IDispatcherService.Connect(SdkClientConnectionOptions connectionOptions)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Initialize(EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.CreateEndpoint[T](EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
System.Net.Sockets.SocketException (0x80004005): No connection could be made because the target machine actively refused it 172.16.77.162:5724
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
 


From the Event Logs:

Log Name:      System
Source:        Service Control Manager
Date:          3/5/2015 9:29:07 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      v-scom-1.sjsd.local
Description:
The System Center Data Access Service service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 60000 milliseconds: Restart the service.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="49152">7031</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2015-03-05T16:29:07.459017900Z" />
    <EventRecordID>248030</EventRecordID>
    <Correlation />
    <Execution ProcessID="592" ThreadID="11172" />
    <Channel>System</Channel>
    <Computer>v-scom-1.sjsd.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="param1">System Center Data Access Service</Data>
    <Data Name="param2">1</Data>
    <Data Name="param3">60000</Data>
    <Data Name="param4">1</Data>
    <Data Name="param5">Restart the service</Data>
    <Binary>4F004D00530044004B000000</Binary>
  </EventData>
</Event>
Log Name:      Application
Source:        Windows Error Reporting
Date:          3/5/2015 9:32:21 AM
Event ID:      1001
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      v-scom-1.sjsd.local
Description:
Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0

Problem signature:
P1: Microsoft.Mom.Sdk.ServiceHost
P2: 7.5.3070.0
P3: 520d7e93
P4: System.Data
P5: 4.0.30319.33440
P6: 52003c3b
P7: 229b
P8: 5e
P9: System.Data.SqlClient.Sql
P10:

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_Microsoft.Mom.Sd_17c7fa124229d734e18e5224d11a218826c98_914c31ed_19eecfdc

Analysis symbol:
Rechecking for solution: 0
Report Id: 31939b2f-c355-11e4-80d6-00155d4b8731
Report Status: 0
Hashed bucket:
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Windows Error Reporting" />
    <EventID Qualifiers="0">1001</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-03-05T16:32:21.000000000Z" />
    <EventRecordID>202257</EventRecordID>
    <Channel>Application</Channel>
    <Computer>v-scom-1.sjsd.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>CLR20r3</Data>
    <Data>Not available</Data>
    <Data>0</Data>
    <Data>Microsoft.Mom.Sdk.ServiceHost</Data>
    <Data>7.5.3070.0</Data>
    <Data>520d7e93</Data>
    <Data>System.Data</Data>
    <Data>4.0.30319.33440</Data>
    <Data>52003c3b</Data>
    <Data>229b</Data>
    <Data>5e</Data>
    <Data>System.Data.SqlClient.Sql</Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_Microsoft.Mom.Sd_17c7fa124229d734e18e5224d11a218826c98_914c31ed_19eecfdc</Data>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>31939b2f-c355-11e4-80d6-00155d4b8731</Data>
    <Data>0</Data>
    <Data>
    </Data>
  </EventData>
</Event>
Jared_BrownAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Jared_BrownAuthor Commented:
More Event Log Info

Log Name:      Application
Source:        Application Error
Date:          3/5/2015 9:32:20 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      v-scom-1.sjsd.local
Description:
Faulting application name: Microsoft.Mom.Sdk.ServiceHost.exe, version: 7.5.3070.0, time stamp: 0x520d7e93
Faulting module name: KERNELBASE.dll, version: 6.3.9600.17278, time stamp: 0x53eebf2e
Exception code: 0xe0434352
Fault offset: 0x000000000000606c
Faulting process id: 0xa4c
Faulting application start time: 0x01d05761f0e4cd3e
Faulting application path: C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Microsoft.Mom.Sdk.ServiceHost.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 31939b2f-c355-11e4-80d6-00155d4b8731
Faulting package full name:
Faulting package-relative application ID:
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-03-05T16:32:20.000000000Z" />
    <EventRecordID>202255</EventRecordID>
    <Channel>Application</Channel>
    <Computer>v-scom-1.sjsd.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Microsoft.Mom.Sdk.ServiceHost.exe</Data>
    <Data>7.5.3070.0</Data>
    <Data>520d7e93</Data>
    <Data>KERNELBASE.dll</Data>
    <Data>6.3.9600.17278</Data>
    <Data>53eebf2e</Data>
    <Data>e0434352</Data>
    <Data>000000000000606c</Data>
    <Data>a4c</Data>
    <Data>01d05761f0e4cd3e</Data>
    <Data>C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Microsoft.Mom.Sdk.ServiceHost.exe</Data>
    <Data>C:\Windows\system32\KERNELBASE.dll</Data>
    <Data>31939b2f-c355-11e4-80d6-00155d4b8731</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>

Log Name:      Application
Source:        .NET Runtime
Date:          3/5/2015 9:32:20 AM
Event ID:      1026
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      v-scom-1.sjsd.local
Description:
Application: Microsoft.Mom.Sdk.ServiceHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.EnterpriseManagement.ConfigurationReaderException
Stack:
   at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.Initialize(Microsoft.EnterpriseManagement.InProcEnterpriseManagementConnectionSettings)
   at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.InitializeRunner(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart(System.Object)

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name=".NET Runtime" />
    <EventID Qualifiers="0">1026</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-03-05T16:32:20.000000000Z" />
    <EventRecordID>202254</EventRecordID>
    <Channel>Application</Channel>
    <Computer>v-scom-1.sjsd.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Application: Microsoft.Mom.Sdk.ServiceHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.EnterpriseManagement.ConfigurationReaderException
Stack:
   at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.Initialize(Microsoft.EnterpriseManagement.InProcEnterpriseManagementConnectionSettings)
   at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.InitializeRunner(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart(System.Object)
</Data>
  </EventData>
</Event>
0
Rich WeisslerProfessional Troublemaker^h^h^h^h^hshooterCommented:
If I'm interpreting what you've posted:  Management Console fails because ServiceHost.exe has stopped on the server (v-scom-1.sjsd.local).  I assume you've already attempted to restart the stopped service, and have already rebooted v-scom-1.

Just to make certain - has this implementation of SCOM ever worked correctly?
Is the SQL Database available and functional?  
Are there any errors in the SQL Server Errorlogs?
Is the account under which the ServiceHost service runs locked out?
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Rich WeisslerProfessional Troublemaker^h^h^h^h^hshooterCommented:
From what little I can glean from what you've posted thus far, it looks like an exception error in CLR20r3, which is being passed up through Kernelbase.dll.  Unfortunately as far as I can translate, it comes out meaning the equivalent to: "An Error occurred, it occurred in shared code, and I can't tell you any more about it"

Have any patches been applied on the server recently?
(Or is this a new implementation, and it isn't patched yet?)
0
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

Jared_BrownAuthor Commented:
It was working, I did run Windows Update around the time I noticed the problem.  I am running Windows update again now and will let you know what happens after that
0
Jared_BrownAuthor Commented:
Machine is fully updated but the problem persists.  Will begin looking at SQL
0
Jared_BrownAuthor Commented:
Now I feel dumb,  The SQL Transaction Log was full.  Backed it up and all is well.
0
Jared_BrownAuthor Commented:
The suggestion to check the SQL Logs put us on the right track.  Thanks
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Server Apps

From novice to tech pro — start learning today.