Windows service stopping - WINDOWS 2012 -

Running Windows 2012 with a service running under Window Services. For some reason, this at certain times stop and fails to restart even though service is set to Recovery to Restart. Looking at the Windows event log the following information has been written. BUt,  it makes no sense as to where I would be pointing the finger to as to the real issue i.e. Windows AD or there is a problem with the application or a GP problem.
See extract from event logs below :-


Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0

Problem signature:
P1: intamacactivationhandlerservice
P2: 1.0.0.0
P3: 54f43942
P4: DAL
P5: 1.0.0.0
P6: 54f43938
P7: fd0
P8: 2a
P9: System.Data.StrongTyping
P10:

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_intamacactivatio_6bc1839796871265e4a46367ca64c74af0a2ab_0325fadd

Analysis symbol:
Rechecking for solution: 0
Report Id: bbf0f759-cc1a-11e4-9449-0e071ace88f5
Report Status: 4
Hashed bucket:
ccfcfcAsked:
Who is Participating?
 
Cliff GaliherCommented:
Deductive reasoning.  First, there is your event log. CLR20r3 is .Net.   So that tells you that the code is doing something .Net doesn't allow, hence the crash.

So, is the OS corrupt, is .Net corrupt, or is it bad code?

So first, I discounted the OS because usually OS corruption exhibits in far worse ways. You'd be seeing blue screens. Or server freezes. So it was easy to discount the OS.

Note that you didn't mention the credentials disappearing until one of your comments. That was NOT in your initial post. So while I had discounted the OS, my initial comment said it could be 50/50 .Net or the app.  Your subsequent post brought up the credentials.

So is it .Net that is corrupt?  Well, usually if .Net is corrupt, something else will show up.  Many of the MS GUI tools now rely on .Net. There is a reason 3.5 is now a required component in 2008 R2 and 4.5 is included in 2012 R2. And why even server core added support for core .Net services. Windows now really needs it.  But...it is always *possible* that you just hadn't gotten .Net to crash...

Then you mentioned the credentials. That helped solidify my opinion on the OS. If the OS was corrupt, *all* services would exhibit this behavior. But you aren't saying that you've had to reset the credentials on every automatic service in the OS.  So we can say that the OS components that handle starting services and store credentials is working (yes even local system and network system have credentials.)

It also steered me away from .Net.  If .Net was corrupt, those components would simply...crash. They wouldn't take their service credentials with them. In fact, because .Net is managed code and the OS does sandbox most of this stuff, even a *bad* crash should simply let you restart the service. This made me believe that the code itself is doing something it shouldn't. Maybe there is some unmanaged DLL. Maybe there is malware. Truth is, I *still* don't know the details there, and I am not in a position to speculate.

But then you said one last thing that really illuminated the app (for whatever reason) as the culprit:  "We have probably 10 or more different .Net services and only a few of these exhibit this."  You have other services running .Net!  Not just other services running C++ or low-level code!  Just as the service crashing solidified to me that the OS was not the culprit, *THIS* solidified that .Net was not the culprit. IF it were, *all* of your .Net services would behave this way.   But no, only a few do!  And dollars to donuts, they were either all written by the same developer, or at the very least, have the same dependencies. And whenever that code gets called, you have the app running that code go nuclear.

Like I said, deductive reasoning.
0
 
Cliff GaliherCommented:
Well, that's .Net crashing. It could be a poorly coded service. Or it could be another app managed to replace some critical .Net dll's with incompatible versions. I've seen both variants.
0
 
ccfcfcAuthor Commented:
It gets a bit weirder. Every now and again, the .NET app which runs as a service stop and you have to re-enter its password and a dialogue box tells you it is re-assigning rights to run as a service. See extract from the event log. BUt, this is not every time.  So I re-enter the password in the service and all is fine. Nothing is locked out and the user account is set in the GP to be able to run as a batch/service.

From event log :-

An account failed to log on.

Subject:
      Security ID:            SYSTEM
      Account Name:            AMI01-APR-EON01$
      Account Domain:            AMI01
      Logon ID:            0x3E7

Logon Type:                  5

Account For Which Logon Failed:
      Security ID:            NULL SID
      Account Name:            srv_eonopenfire
      Account Domain:            AMI01

Failure Information:
      Failure Reason:            The user has not been granted the requested logon type at this machine.
      Status:                  0xC000015B
      Sub Status:            0x0

Process Information:
      Caller Process ID:      0x204
      Caller Process Name:      C:\Windows\System32\services.exe

Network Information:
      Workstation Name:      AMI01-APR-EON01
      Source Network Address:      -
      Source Port:            -

Detailed Authentication Information:
      Logon Process:            Advapi  
      Authentication Package:      Negotiate
      Transited Services:      -
      Package Name (NTLM only):      -
      Key Length:            0

This event is generated when a logon request fails. It is generated on the computer where access was attempted.

The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).

The Process Information fields indicate which account and process on the system requested the logon.

The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
      - Transited services indicate which intermediate services have participated in this logon request.
      - Package name indicates which sub-protocol was used among the NTLM protocols.
      - Key length indicates the length of the generated session key. This will be 0 if no
0
Problems using Powershell and Active Directory?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

 
Cliff GaliherCommented:
Increasingly sounds like a code issue with the service/app itself. You'll likely need to work with the vendor/developer to solve the issue.
0
 
ccfcfcAuthor Commented:
Not entirely sure on that one as, why does when you try to restart the service windows reports you do not have the settings to run s a service. To resolve the password needs to re-entered and then it reports that the account is now able to run as a service. Surely thats an OS/AD issue ? Yet within the Group Policy this is how it is set already
0
 
Cliff GaliherCommented:
Windows  credential management doesn't mean on .Net. But a .Net app, after starting, could easily change (or request to change) its security context and that can cause credentials to be expired. The OS is very likely behaving exactly as it is supposed to.
0
 
ccfcfcAuthor Commented:
So is that a setting in .Net or something in the .Net app ? We have probably 10 or more different .Net services and only a few of these exhibit this. The worrying or annoying part is, running a 24/7 environment, this can happen over night and needs the password re-entering.
0
 
Cliff GaliherCommented:
Something in the .Net app, which is why I said you"\l need to work with the developer. If it were a setting or a corrupt OS, the issue would be more pervasive, or the OS itself would show significant signs of instability.
0
 
ccfcfcAuthor Commented:
From looking through the code, there is nothing that is explicitly doing anything with the credentials.
Whichever account the service is being run as is used for everything within the service application itself.
0
 
Cliff GaliherCommented:
Well, I'm not going to try and convince you. I have better things to do with my time. I've stated what I believe the problem to be and have provided the appropriate evidence to back up that opinion and discount the OS or a widespread .Net issue. You can choose to accept that or not. You came here to ask for help, after all. Not the other way around.
0
 
ccfcfcAuthor Commented:
We are not dismissing your reply, but just asking for details on how you have come to this conclusion in previous situations. e.g. examples of what the code did to force this.
We would be grateful for evidence as many hours have been spent looking hence using this forum to pull upon experience of others.
So if you have details I would be ever so grateful othewise, I am no wiser and feel I am pointing the finger without any real evidence or experience which, in IT is a given.
0
 
ccfcfcAuthor Commented:
Well thanks for that, but trust me I have been through all that logic at the OS/AD level and with the developers.
I was hoping that you may have "real life" experience of perhaps a procedure or function within .Net that you have seen in the past.
I have spent many hours of "deductive" reasoning and before, I can confidently point the point at anything I need evidence of the issue hence by posting of this on this forum hoping someone comes forwrd with documental evidence of "real life" experience as, I also have done "deductive" steps.
I appreciate your time to reply and conents but you must understand, I have a team of developers whom will ask for evidence of issue which is what I am looking for.
0
 
Cliff GaliherCommented:
Hah. And there's the catch-22.  I, in fact, said above that yes, the OS will react to certain events. I *know* this because of "real life experience."  I didn't pull that out of thin air. But if pointing to this thread won't convince you (or your developers) then why would my proclamation that I've seen it before suddenly change your mind??  It wouldn't!  So why should I bother?

Now, if you want me to review your code explicitly, c'mon...you know developers get paid good money.  As far as the rest, any developer worth their salt can add debug routines.  Take the whole credential thing out of it.  Your initial post was about a crash.  The credentials aren't playing a part in that. THAT you can take to your developers without speculation, without finger pointing, and say "here is a .Net crash" and they should be able to debug that. And if they don't...well...you have bigger problems than your services losing credentials.  

I've been doing DevOps before DevOps was even a term, so I wear the dual hat of developer and IT Pro more often than not. I know how things work and I know when it is an IT issue and when it is a developer issue. I'm not sure what you expected here, but I now believe you won't find it from *any* expert because you seem to be wanting more specifics than can be provided, given the environment. Unless you want to post your code and a full OS image of an existing machine with the issue so some expert who is more desparate than I, or who has WAY more time on their hands, can boot up the image, run checksums on every file, reproduce the bug, and/or scour code, I just don't know what you expected to happen here....
0
 
Bob LearnedCommented:
It is difficult to find that "needle-in-a-haystack" without more information.  

1) Is the .NET service something internal, or a 3rd-party service?

2) What versions of .NET framework are installed?

3) Are any service packs installed for Windows Server 2012?
0
 
it_saigeDeveloperCommented:
The problem here is really as Cliff has described it, .NET has encountered an unhandled exception.  The most common causes for an unhandled exception are:

1.  Errors in coding.
2.  Corrupt or missing dependencies.
3.  Unregistered dependencies.

The exception in this case is System.Data.StrongTyping which is happening in the DAL Assembly (DAL is commonly referred to as the Data Access Layer).  A StrongTypingException is caused because one of the properties in a data query is returning a null value and the code that accesses this property is not checking if the property (or it's parent object) is null before attempting to retrieve the value of the property.

I can easily replicate this issue by building a project with a Data Access Layer of my own; consider the following:

Project named EE_Q28638011 -
using System;
using EE_Q28638011.Common;

namespace EE_Q28638011
{
	class Program
	{
		static void Main(string[] args)
		{
			PatientTable table = new PatientTable();
			table.PatientRowChanged += OnPatientRowChanged;
			PatientRow row = table.GetNewRow();
			row.ID = null;
			row.FirstName = default(string);
			table.Add(row);

			foreach (PatientRow patient in table.Rows)
				Console.WriteLine(patient.ID);

			Console.ReadLine();
		}

		private static void OnPatientRowChanged(PatientTable sender, PatientRowChangedEventArgs args)
		{
			if (args.Row.LastName != string.Empty)
				throw new ApplicationException("The row did not initialize to an empty string for the LastName field.");
		}
	}
}

Open in new window

Project named EE_Q28638011.Common -
using System;
using System.Data;

namespace EE_Q28638011.Common
{
	public class PatientTable : DataTable
	{
		public event PatientRowChanged PatientRowChanged;

		public PatientRow this[int index] { get { return (PatientRow)Rows[index]; } }

		public PatientTable()
		{
			Columns.Add(new DataColumn("ID", typeof(int)));
			Columns.Add(new DataColumn("LastName", typeof(string)));
			Columns.Add(new DataColumn("FirstName", typeof(string)));
		}

		public void Add(PatientRow row)
		{
			Rows.Add(row);
		}

		public void Remove(PatientRow row)
		{
			Rows.Remove(row);
		}

		public PatientRow GetNewRow()
		{
			return (PatientRow)NewRow();
		}

		protected override Type GetRowType()
		{
			return typeof(PatientRow);
		}

		protected override DataRow NewRowFromBuilder(DataRowBuilder builder)
		{
			return new PatientRow(builder);
		}

		protected override void OnRowChanged(DataRowChangeEventArgs e)
		{
			base.OnRowChanged(e);
			OnPatientRowChanged(new PatientRowChangedEventArgs(e.Action, (PatientRow)e.Row));
		}

		protected virtual void OnPatientRowChanged(PatientRowChangedEventArgs args)
		{
			if (PatientRowChanged != null)
				PatientRowChanged(this, args);
		}
	}

	public class PatientRow : DataRow
	{
		public int? ID
		{
			get
			{
				try
				{
					return (int?)base["ID"];
				}
				catch (InvalidCastException e)
				{
					throw new StrongTypingException("The value for column 'ID' in table is DBNull", e);
				}
			}
			set { base["ID"] = value.HasValue ? (object)value.Value : DBNull.Value; }
		}

		public string LastName
		{
			get
			{
				try
				{
					return (string)base["LastName"];
				}
				catch (InvalidCastException e)
				{
					throw new StrongTypingException("The value for column 'LastName' in table is DBNull", e);
				}
			}
			set { base["LastName"] = value; }
		}

		public string FirstName
		{
			get
			{
				try
				{
					return (string)base["FirstName"];
				}
				catch (InvalidCastException e)
				{
					throw new StrongTypingException("The value for column 'FirstName' in table is DBNull", e);
				}
			}
			set { base["FirstName"] = value; }
		}

		internal PatientRow(DataRowBuilder builder) : base(builder) { ;}
	}

	public delegate void PatientRowChanged(PatientTable sender, PatientRowChangedEventArgs args);

	public class PatientRowChangedEventArgs
	{
		protected DataRowAction action;
		protected PatientRow row;

		public DataRowAction Action { get { return action; } }
		public PatientRow Row { get { return row; } }

		public PatientRowChangedEventArgs(DataRowAction action, PatientRow row)
		{
			this.action = action;
			this.row = row;
		}
	}
}

Open in new window

Produces the following error in the event log -Capture.JPGTo combat this problem I can apply the following fix in the code that represents the Data Access Layer -
public class PatientRow : DataRow
{
	public int? ID
	{
		get
		{
			try
			{
				return base["ID"] != DBNull.Value ? (int)base["ID"] : default(int);
			}
			catch (InvalidCastException e)
			{
				throw new StrongTypingException("The value for column 'ID' in table is DBNull", e);
			}
		}
		set { base["ID"] = value.HasValue ? (object)value.Value : DBNull.Value; }
	}

	public string LastName
	{
		get
		{
			try
			{
				return base["LastName"] != DBNull.Value ? (string)base["LastName"] : default(string);
			}
			catch (InvalidCastException e)
			{
				throw new StrongTypingException("The value for column 'LastName' in table is DBNull", e);
			}
		}
		set { base["LastName"] = value; }
	}

	public string FirstName
	{
		get
		{
			try
			{
				return base["FirstName"] != DBNull.Value ? (string)base["FirstName"] : default(string);
			}
			catch (InvalidCastException e)
			{
				throw new StrongTypingException("The value for column 'FirstName' in table is DBNull", e);
			}
		}
		set { base["FirstName"] = value; }
	}

	internal PatientRow(DataRowBuilder builder) : base(builder) { ;}
}

Open in new window

Which now no longer produces the exception.

NOTE:  The resolution here is not meant to be a fix that the development team for your application would use.  They need to debug and determine the root cause of the problem.

-saige-
0
 
ccfcfcAuthor Commented:
The resolution to the actual crash is with our Dev team to look at, but in the meantime we need to restart the service to get it back up and running again.

We have it configured to automatically restart on failure, but this isn't working.
When we try to restart the service manually, it requires us to re-enter the service user's password at which point it says that the user has been granted the logon as a service right (http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/Windows_Server_2012/Q_28638011.html#a40671186)
0
 
it_saigeDeveloperCommented:
Is there a specific reason that this service is bound to a user as opposed to Local Service, Network Service or Local System (which are the three standard Service accounts)?  When you introduce a user into the service mix, then you end up with the potential of an outside party exercising it's ability to potentially change the properties associated with said user (in other words, there could be an audit policy in place to prevent the user from having the LogonAs Service right).

-saige-
0
 
ccfcfcAuthor Commented:
The said user account is set within the Domain Policy to have the required permissions (logon as a service and batch) .
The reason specific user accounts are used is due to having a mulit-tenent platform with different services running different programs on a per customer basis. We can and need to control access to DB's etc.
The account WAS set to have the permissions to start as a service/batch already - domain policy based.
The fix was to re-type the service password and it restarted - as a dialogue box reported it was not set as this requirement.
This is a strange fix as the re-tying the password resolved the issue nothing was changed in the policy and looking at the GP this was still set as required. i,e LogonAs a Service  and Batch

Hope that makes sense
0
 
it_saigeDeveloperCommented:
Does the user have *any* other policies applied to him/her (to be certain on this point, you can run an RSoP [Resultant Set of Policy] in logging mode to determine which policies affect the user)?

Is the user account actively used in any other process (meaning; for example, is the user used to login to a workstation and perform non-service related tasks or is the user used to authenticate other service related tasks)?

Is the user affected by the password auditing policy (assuming you have one)?

Is the server that this service is attached to a DC?  If not, have you attempted to create the user locally, assign them the appropriate rights and run the service?

-saige-
0
 
ccfcfcAuthor Commented:
The service account does not logon to a member server but is set to only run as a service on a member server.

How do I run thr RSoP against that service account ?
0
 
it_saigeDeveloperCommented:
You will need to log into the server that the service runs on with the account in question, once a profile has been generated for the service account, you should then be able to run the RSoP in logging mode against the server for that service account.

-saige-
0
 
David Johnson, CD, MVPOwnerCommented:
you have 3 problems
1. unhandled exception which can be coded around using try/catch routines.
2. service not restarting which is related to #3
3. service account password requires re-entry..

problems 2 and 3 can be probably be pointed to #1.  a probable hang in the program itself with the exception. In a test environment one codes for expected and unexpected data responses. So this points to your DEV and QA testing areas.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.