Link to home
Start Free TrialLog in
Avatar of Jared_Brown
Jared_Brown

asked on

SCOM 2012 R2 Data Access Service Starts and then Stops

Here is the error seen by SCOM Management Console:

Microsoft.EnterpriseManagement.Common.ServiceNotRunningException: The Data Access service is either not running or not yet initialized. Check the event log for more information. ---> System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://v-scom-1:5724/DispatcherService. The connection attempt lasted for a time span of 00:00:01.0008624. TCP error code 10061: No connection could be made because the target machine actively refused it 172.16.77.162:5724.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 172.16.77.162:5724
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.EnterpriseManagement.Common.Internal.IDispatcherService.Connect(SdkClientConnectionOptions connectionOptions)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Initialize(EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.CreateEndpoint[T](EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
   --- End of inner exception stack trace ---
   at Microsoft.EnterpriseManagement.Common.Internal.ExceptionHandlers.HandleChannelExceptions(Exception ex)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.CreateEndpoint[T](EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.ConstructEnterpriseManagementGroupInternal[T,P](EnterpriseManagementConnectionSettings connectionSettings, ClientDataAccessCore clientCallback)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.RetrieveEnterpriseManagementGroupInternal[T,P](EnterpriseManagementConnectionSettings connectionSettings, ClientDataAccessCore callbackDispatcherService)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Connect[T,P](EnterpriseManagementConnectionSettings connectionSettings, ClientDataAccessCore callbackDispatcherService)
   at Microsoft.EnterpriseManagement.ManagementGroup.InternalInitialize(EnterpriseManagementConnectionSettings connectionSettings, ManagementGroupInternal internals)
   at Microsoft.EnterpriseManagement.Mom.Internal.UI.Common.ManagementGroupSessionManager.Connect(String server)
   at Microsoft.EnterpriseManagement.Monitoring.Console.Internal.ConsoleWindowBase.TryConnectToManagementGroupJob(Object sender, ConsoleJobEventArgs args)
System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://v-scom-1:5724/DispatcherService. The connection attempt lasted for a time span of 00:00:01.0008624. TCP error code 10061: No connection could be made because the target machine actively refused it 172.16.77.162:5724.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 172.16.77.162:5724
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.EnterpriseManagement.Common.Internal.IDispatcherService.Connect(SdkClientConnectionOptions connectionOptions)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Initialize(EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.CreateEndpoint[T](EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
System.Net.Sockets.SocketException (0x80004005): No connection could be made because the target machine actively refused it 172.16.77.162:5724
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
 


From the Event Logs:

Log Name:      System
Source:        Service Control Manager
Date:          3/5/2015 9:29:07 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      v-scom-1.sjsd.local
Description:
The System Center Data Access Service service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 60000 milliseconds: Restart the service.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="49152">7031</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2015-03-05T16:29:07.459017900Z" />
    <EventRecordID>248030</EventRecordID>
    <Correlation />
    <Execution ProcessID="592" ThreadID="11172" />
    <Channel>System</Channel>
    <Computer>v-scom-1.sjsd.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="param1">System Center Data Access Service</Data>
    <Data Name="param2">1</Data>
    <Data Name="param3">60000</Data>
    <Data Name="param4">1</Data>
    <Data Name="param5">Restart the service</Data>
    <Binary>4F004D00530044004B000000</Binary>
  </EventData>
</Event>
Log Name:      Application
Source:        Windows Error Reporting
Date:          3/5/2015 9:32:21 AM
Event ID:      1001
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      v-scom-1.sjsd.local
Description:
Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0

Problem signature:
P1: Microsoft.Mom.Sdk.ServiceHost
P2: 7.5.3070.0
P3: 520d7e93
P4: System.Data
P5: 4.0.30319.33440
P6: 52003c3b
P7: 229b
P8: 5e
P9: System.Data.SqlClient.Sql
P10:

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_Microsoft.Mom.Sd_17c7fa124229d734e18e5224d11a218826c98_914c31ed_19eecfdc

Analysis symbol:
Rechecking for solution: 0
Report Id: 31939b2f-c355-11e4-80d6-00155d4b8731
Report Status: 0
Hashed bucket:
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Windows Error Reporting" />
    <EventID Qualifiers="0">1001</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-03-05T16:32:21.000000000Z" />
    <EventRecordID>202257</EventRecordID>
    <Channel>Application</Channel>
    <Computer>v-scom-1.sjsd.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>CLR20r3</Data>
    <Data>Not available</Data>
    <Data>0</Data>
    <Data>Microsoft.Mom.Sdk.ServiceHost</Data>
    <Data>7.5.3070.0</Data>
    <Data>520d7e93</Data>
    <Data>System.Data</Data>
    <Data>4.0.30319.33440</Data>
    <Data>52003c3b</Data>
    <Data>229b</Data>
    <Data>5e</Data>
    <Data>System.Data.SqlClient.Sql</Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_Microsoft.Mom.Sd_17c7fa124229d734e18e5224d11a218826c98_914c31ed_19eecfdc</Data>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>31939b2f-c355-11e4-80d6-00155d4b8731</Data>
    <Data>0</Data>
    <Data>
    </Data>
  </EventData>
</Event>
Avatar of Jared_Brown
Jared_Brown

ASKER

More Event Log Info

Log Name:      Application
Source:        Application Error
Date:          3/5/2015 9:32:20 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      v-scom-1.sjsd.local
Description:
Faulting application name: Microsoft.Mom.Sdk.ServiceHost.exe, version: 7.5.3070.0, time stamp: 0x520d7e93
Faulting module name: KERNELBASE.dll, version: 6.3.9600.17278, time stamp: 0x53eebf2e
Exception code: 0xe0434352
Fault offset: 0x000000000000606c
Faulting process id: 0xa4c
Faulting application start time: 0x01d05761f0e4cd3e
Faulting application path: C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Microsoft.Mom.Sdk.ServiceHost.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 31939b2f-c355-11e4-80d6-00155d4b8731
Faulting package full name:
Faulting package-relative application ID:
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-03-05T16:32:20.000000000Z" />
    <EventRecordID>202255</EventRecordID>
    <Channel>Application</Channel>
    <Computer>v-scom-1.sjsd.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Microsoft.Mom.Sdk.ServiceHost.exe</Data>
    <Data>7.5.3070.0</Data>
    <Data>520d7e93</Data>
    <Data>KERNELBASE.dll</Data>
    <Data>6.3.9600.17278</Data>
    <Data>53eebf2e</Data>
    <Data>e0434352</Data>
    <Data>000000000000606c</Data>
    <Data>a4c</Data>
    <Data>01d05761f0e4cd3e</Data>
    <Data>C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Microsoft.Mom.Sdk.ServiceHost.exe</Data>
    <Data>C:\Windows\system32\KERNELBASE.dll</Data>
    <Data>31939b2f-c355-11e4-80d6-00155d4b8731</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>

Log Name:      Application
Source:        .NET Runtime
Date:          3/5/2015 9:32:20 AM
Event ID:      1026
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      v-scom-1.sjsd.local
Description:
Application: Microsoft.Mom.Sdk.ServiceHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.EnterpriseManagement.ConfigurationReaderException
Stack:
   at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.Initialize(Microsoft.EnterpriseManagement.InProcEnterpriseManagementConnectionSettings)
   at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.InitializeRunner(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart(System.Object)

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name=".NET Runtime" />
    <EventID Qualifiers="0">1026</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-03-05T16:32:20.000000000Z" />
    <EventRecordID>202254</EventRecordID>
    <Channel>Application</Channel>
    <Computer>v-scom-1.sjsd.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Application: Microsoft.Mom.Sdk.ServiceHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.EnterpriseManagement.ConfigurationReaderException
Stack:
   at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.Initialize(Microsoft.EnterpriseManagement.InProcEnterpriseManagementConnectionSettings)
   at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.InitializeRunner(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart(System.Object)
</Data>
  </EventData>
</Event>
ASKER CERTIFIED SOLUTION
Avatar of Rich Weissler
Rich Weissler

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
From what little I can glean from what you've posted thus far, it looks like an exception error in CLR20r3, which is being passed up through Kernelbase.dll.  Unfortunately as far as I can translate, it comes out meaning the equivalent to: "An Error occurred, it occurred in shared code, and I can't tell you any more about it"

Have any patches been applied on the server recently?
(Or is this a new implementation, and it isn't patched yet?)
It was working, I did run Windows Update around the time I noticed the problem.  I am running Windows update again now and will let you know what happens after that
Machine is fully updated but the problem persists.  Will begin looking at SQL
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
The suggestion to check the SQL Logs put us on the right track.  Thanks