Avatar of Jared_Brown
Jared_Brown
 asked on

SCOM 2012 R2 Data Access Service Starts and then Stops

Here is the error seen by SCOM Management Console:

Microsoft.EnterpriseManagement.Common.ServiceNotRunningException: The Data Access service is either not running or not yet initialized. Check the event log for more information. ---> System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://v-scom-1:5724/DispatcherService. The connection attempt lasted for a time span of 00:00:01.0008624. TCP error code 10061: No connection could be made because the target machine actively refused it 172.16.77.162:5724.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 172.16.77.162:5724
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.EnterpriseManagement.Common.Internal.IDispatcherService.Connect(SdkClientConnectionOptions connectionOptions)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Initialize(EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.CreateEndpoint[T](EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
   --- End of inner exception stack trace ---
   at Microsoft.EnterpriseManagement.Common.Internal.ExceptionHandlers.HandleChannelExceptions(Exception ex)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.CreateEndpoint[T](EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.ConstructEnterpriseManagementGroupInternal[T,P](EnterpriseManagementConnectionSettings connectionSettings, ClientDataAccessCore clientCallback)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.RetrieveEnterpriseManagementGroupInternal[T,P](EnterpriseManagementConnectionSettings connectionSettings, ClientDataAccessCore callbackDispatcherService)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Connect[T,P](EnterpriseManagementConnectionSettings connectionSettings, ClientDataAccessCore callbackDispatcherService)
   at Microsoft.EnterpriseManagement.ManagementGroup.InternalInitialize(EnterpriseManagementConnectionSettings connectionSettings, ManagementGroupInternal internals)
   at Microsoft.EnterpriseManagement.Mom.Internal.UI.Common.ManagementGroupSessionManager.Connect(String server)
   at Microsoft.EnterpriseManagement.Monitoring.Console.Internal.ConsoleWindowBase.TryConnectToManagementGroupJob(Object sender, ConsoleJobEventArgs args)
System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://v-scom-1:5724/DispatcherService. The connection attempt lasted for a time span of 00:00:01.0008624. TCP error code 10061: No connection could be made because the target machine actively refused it 172.16.77.162:5724.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 172.16.77.162:5724
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.EnterpriseManagement.Common.Internal.IDispatcherService.Connect(SdkClientConnectionOptions connectionOptions)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Initialize(EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.CreateEndpoint[T](EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
System.Net.Sockets.SocketException (0x80004005): No connection could be made because the target machine actively refused it 172.16.77.162:5724
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
 


From the Event Logs:

Log Name:      System
Source:        Service Control Manager
Date:          3/5/2015 9:29:07 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      v-scom-1.sjsd.local
Description:
The System Center Data Access Service service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 60000 milliseconds: Restart the service.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="49152">7031</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2015-03-05T16:29:07.459017900Z" />
    <EventRecordID>248030</EventRecordID>
    <Correlation />
    <Execution ProcessID="592" ThreadID="11172" />
    <Channel>System</Channel>
    <Computer>v-scom-1.sjsd.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="param1">System Center Data Access Service</Data>
    <Data Name="param2">1</Data>
    <Data Name="param3">60000</Data>
    <Data Name="param4">1</Data>
    <Data Name="param5">Restart the service</Data>
    <Binary>4F004D00530044004B000000</Binary>
  </EventData>
</Event>
Log Name:      Application
Source:        Windows Error Reporting
Date:          3/5/2015 9:32:21 AM
Event ID:      1001
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      v-scom-1.sjsd.local
Description:
Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0

Problem signature:
P1: Microsoft.Mom.Sdk.ServiceHost
P2: 7.5.3070.0
P3: 520d7e93
P4: System.Data
P5: 4.0.30319.33440
P6: 52003c3b
P7: 229b
P8: 5e
P9: System.Data.SqlClient.Sql
P10:

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_Microsoft.Mom.Sd_17c7fa124229d734e18e5224d11a218826c98_914c31ed_19eecfdc

Analysis symbol:
Rechecking for solution: 0
Report Id: 31939b2f-c355-11e4-80d6-00155d4b8731
Report Status: 0
Hashed bucket:
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Windows Error Reporting" />
    <EventID Qualifiers="0">1001</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-03-05T16:32:21.000000000Z" />
    <EventRecordID>202257</EventRecordID>
    <Channel>Application</Channel>
    <Computer>v-scom-1.sjsd.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>CLR20r3</Data>
    <Data>Not available</Data>
    <Data>0</Data>
    <Data>Microsoft.Mom.Sdk.ServiceHost</Data>
    <Data>7.5.3070.0</Data>
    <Data>520d7e93</Data>
    <Data>System.Data</Data>
    <Data>4.0.30319.33440</Data>
    <Data>52003c3b</Data>
    <Data>229b</Data>
    <Data>5e</Data>
    <Data>System.Data.SqlClient.Sql</Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_Microsoft.Mom.Sd_17c7fa124229d734e18e5224d11a218826c98_914c31ed_19eecfdc</Data>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>31939b2f-c355-11e4-80d6-00155d4b8731</Data>
    <Data>0</Data>
    <Data>
    </Data>
  </EventData>
</Event>
Microsoft Server AppsSystem UtilitiesMicrosoft SQL Server

Avatar of undefined
Last Comment
Jared_Brown

8/22/2022 - Mon
Jared_Brown

ASKER
More Event Log Info

Log Name:      Application
Source:        Application Error
Date:          3/5/2015 9:32:20 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      v-scom-1.sjsd.local
Description:
Faulting application name: Microsoft.Mom.Sdk.ServiceHost.exe, version: 7.5.3070.0, time stamp: 0x520d7e93
Faulting module name: KERNELBASE.dll, version: 6.3.9600.17278, time stamp: 0x53eebf2e
Exception code: 0xe0434352
Fault offset: 0x000000000000606c
Faulting process id: 0xa4c
Faulting application start time: 0x01d05761f0e4cd3e
Faulting application path: C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Microsoft.Mom.Sdk.ServiceHost.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 31939b2f-c355-11e4-80d6-00155d4b8731
Faulting package full name:
Faulting package-relative application ID:
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-03-05T16:32:20.000000000Z" />
    <EventRecordID>202255</EventRecordID>
    <Channel>Application</Channel>
    <Computer>v-scom-1.sjsd.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Microsoft.Mom.Sdk.ServiceHost.exe</Data>
    <Data>7.5.3070.0</Data>
    <Data>520d7e93</Data>
    <Data>KERNELBASE.dll</Data>
    <Data>6.3.9600.17278</Data>
    <Data>53eebf2e</Data>
    <Data>e0434352</Data>
    <Data>000000000000606c</Data>
    <Data>a4c</Data>
    <Data>01d05761f0e4cd3e</Data>
    <Data>C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Microsoft.Mom.Sdk.ServiceHost.exe</Data>
    <Data>C:\Windows\system32\KERNELBASE.dll</Data>
    <Data>31939b2f-c355-11e4-80d6-00155d4b8731</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>

Log Name:      Application
Source:        .NET Runtime
Date:          3/5/2015 9:32:20 AM
Event ID:      1026
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      v-scom-1.sjsd.local
Description:
Application: Microsoft.Mom.Sdk.ServiceHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.EnterpriseManagement.ConfigurationReaderException
Stack:
   at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.Initialize(Microsoft.EnterpriseManagement.InProcEnterpriseManagementConnectionSettings)
   at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.InitializeRunner(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart(System.Object)

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name=".NET Runtime" />
    <EventID Qualifiers="0">1026</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-03-05T16:32:20.000000000Z" />
    <EventRecordID>202254</EventRecordID>
    <Channel>Application</Channel>
    <Computer>v-scom-1.sjsd.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Application: Microsoft.Mom.Sdk.ServiceHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.EnterpriseManagement.ConfigurationReaderException
Stack:
   at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.Initialize(Microsoft.EnterpriseManagement.InProcEnterpriseManagementConnectionSettings)
   at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.InitializeRunner(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart(System.Object)
</Data>
  </EventData>
</Event>
ASKER CERTIFIED SOLUTION
Rich Weissler

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
Rich Weissler

From what little I can glean from what you've posted thus far, it looks like an exception error in CLR20r3, which is being passed up through Kernelbase.dll.  Unfortunately as far as I can translate, it comes out meaning the equivalent to: "An Error occurred, it occurred in shared code, and I can't tell you any more about it"

Have any patches been applied on the server recently?
(Or is this a new implementation, and it isn't patched yet?)
Jared_Brown

ASKER
It was working, I did run Windows Update around the time I noticed the problem.  I am running Windows update again now and will let you know what happens after that
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
Jared_Brown

ASKER
Machine is fully updated but the problem persists.  Will begin looking at SQL
SOLUTION
Jared_Brown

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Jared_Brown

ASKER
The suggestion to check the SQL Logs put us on the right track.  Thanks