Application Pool Recycle causes failure

Thomas Grassi
Thomas Grassi used Ask the Experts™
on
Windows 2012 R2 Web Server
IIS 8.5
SharePoint 2010 SP2 November CU 2015


Everyday at 2:49 AM I get the following errors one in the system log and one in the application log

Log Name:      Application
Source:        ASP.NET 2.0.50727.0
Date:          11/13/2015 2:49:03 AM
Event ID:      1334
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SERV013.FQDN.com
Description:
An unhandled exception occurred and the process was terminated.

Application ID: /LM/W3SVC/1068268973/ROOT

Process ID: 7540

Exception: System.Security.Cryptography.CryptographicException

Message: Keyset does not exist


StackTrace:    at System.Security.Cryptography.CryptographicException.ThrowCryptogaphicException(Int32 hr)
   at System.Security.Cryptography.SafeProvHandle._FreeCSP(IntPtr pProvCtx)
   at System.Security.Cryptography.SafeProvHandle.ReleaseHandle()
   at System.Runtime.InteropServices.SafeHandle.InternalFinalize()
   at System.Runtime.InteropServices.SafeHandle.Dispose(Boolean disposing)
   at System.Runtime.InteropServices.SafeHandle.Finalize()
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="ASP.NET 2.0.50727.0" />
    <EventID Qualifiers="49152">1334</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-11-13T07:49:03.000000000Z" />
    <EventRecordID>90691</EventRecordID>
    <Channel>Application</Channel>
    <Computer>TGCS013.our.network.tgcsnet.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>An unhandled exception occurred and the process was terminated.

Application ID: /LM/W3SVC/1068268973/ROOT

Process ID: 7540

Exception: System.Security.Cryptography.CryptographicException

Message: Keyset does not exist


StackTrace:    at System.Security.Cryptography.CryptographicException.ThrowCryptogaphicException(Int32 hr)
   at System.Security.Cryptography.SafeProvHandle._FreeCSP(IntPtr pProvCtx)
   at System.Security.Cryptography.SafeProvHandle.ReleaseHandle()
   at System.Runtime.InteropServices.SafeHandle.InternalFinalize()
   at System.Runtime.InteropServices.SafeHandle.Dispose(Boolean disposing)
   at System.Runtime.InteropServices.SafeHandle.Finalize()</Data>
  </EventData>
</Event>



Log Name:      System
Source:        Microsoft-Windows-WAS
Date:          11/13/2015 2:49:04 AM
Event ID:      5009
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      SERV013.FQDN.com
Description:
A process serving application pool 'SharePoint - Default App Pool' terminated unexpectedly. The process id was '7540'. The process exit code was '0xe0434f4d'.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-WAS" Guid="{524B5D04-133C-4A62-8362-64E8EDB9CE40}" EventSourceName="WAS" />
    <EventID Qualifiers="32768">5009</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-11-13T07:49:04.000000000Z" />
    <EventRecordID>74547</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>System</Channel>
    <Computer>SERV013.FQDN.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="AppPoolID">SharePoint - Default App Pool</Data>
    <Data Name="ProcessID">7540</Data>
    <Data Name="ExitCode">e0434f4d</Data>
  </EventData>
</Event>

My research found these

From the Event ID 5009 ran err.exe   on 0xe0434f4d

C:\Util\Err>err 0xe0434f4d
# as an HRESULT: Severity: FAILURE (1), Facility: 0x43, Code 0x4f4d
# NOT FOUND: 0xe0434f4d

Do not understand what is NOT FOUND

This seems to be COM permissions?????????
Command prompt->dcomcnfg->component services->computers->right click on my computer->properties.
Go to COM security
Under launch and activation permissions
Add the SID S-1-5-82-3006700770-424185619-1745488364-794895919-4004696415 and provide local launch permissions.

then try and check if IISRESET still throws same error in event log.

I added spapppool account

No change

I spent many hours trying to figure out why everyday at 2:49 AM then by dumb luck I found it.

From within IIS I expanded the Application Pools list after scrolling thru each entry I found that

Application Pool SharePoint - Default App Pool is started  v2.0 Integrated identity mydom\spapppool

when I clicked on Recycling Settings  I see that it is set for Specific times at 2:49 AM


I also installed DebugDiag 2 Analysis and I have a dump  but I a, not sure what to look for


Any thoughts

Thanks in advance

Tom
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
What I interpret is that the error is older than NOV 2015 CU, right?
Or did it come up with NOV 2015 CU?

What pointed you to the DCOM description, did you have this error in combination with the other error?

Does the pool recycle?

I would expect that this may be a permission problem. If you see a DCOM error, it can even come up,
if a component tries to start too early. Seen it more often with local accounts as local service or network service. So interesting in this combination is the farm count and the identity of the app pool.
In which local groups are they a member?

Also keep in mind that changing permissions need sometimes a logoff. So you have to shut down al the services running under this account and restart them again (or just reboot the machine.

So you said the error comes also up, if you run IISReset, right? Are you farm admin? In which local groups is your account?

What maybe a fast try is to change the App Pool (via the SharePoint service accounts ) to a temporary account, IISRESET, change it back, run the SharePoint config wizard and see, if it makes any change. Also you may reboot the machine and see, if the behavior change.
This procedure may reset all needed permissions for the pool account.
Thomas GrassiSystems Administrator

Author

Commented:
Bembi

First thanks for responding

What I interpret is that the error is older than NOV 2015 CU, right?

Yes this error has been happing for a long time I even rebuilt my entire SharePoint Farm 3 times and still continue to get this error

Or did it come up with NOV 2015 CU?   NO see above reply

I would expect that this may be a permission problem. If you see a DCOM error, it can even come up,
 if a component tries to start too early. Seen it more often with local accounts as local service or network service. So interesting in this combination is the farm count and the identity of the app pool.
 In which local groups are they a member?


No Dcom error in my logs I just came across an article in my research of the above errors
I wish I had the link  to show but my computer at work my browser died and I lost the link that day.

Also keep in mind that changing permissions need sometimes a logoff. So you have to shut down al the services running under this account and restart them again (or just reboot the machine.


Yes I know I restart/reboot my server at least once a month to apply updates.
I also changed my login account to use the sharepoint farm account  spfarm

So you said the error comes also up, if you run IISReset, right?
Yes they do even when I modify the application pool as I did today to remove the scheduled time of 2:49 AM

Are you farm admin?  yes I am

which local groups is your account?  on the server local Users and Groups  "Administrators"
On the domain
PS C:\util> Get-ADPrincipalGroupMembership spfarm | select name

name
----
Domain Users
WSS_WPG
WSS_ADMIN_WPG
SharePointAdmins

What maybe a fast try is to change the App Pool (via the SharePoint service accounts ) to a temporary account, IISRESET, change it back, run the SharePoint config wizard and see, if it makes any change. Also you may reboot the machine and see, if the behavior change.
 This procedure may reset all needed permissions for the pool account.



I will try this and report back
No problem, ...
You may also post the group membership of the pool account identity.

Another check is just, to put the farm account into the local admins groups. (Reboot or restart of the affected services running under farm account).

The farm account (timer service) runs the job  to recycle. So if there is a permission problem with the farm account, the error may move away with admin permissions. But also the pool account is under inspection, and need not necessarily only the web site. Check the service accounts in SharePoint, which services are using the pool account as well.

Next step would be (but for this I need my systems) to check the permissions in the web application.

Assigning the service accounts in SharePoint should at least put them into the correct groups. Configuration wizard then resets the permissions on files, registry and the database.

I see also in you logs errors in he shut down....
So my question would be, if you use IISReset..., do you see only errors in the event log or so also errors in the command line?

Have you tried to disable the virus scan during IISReset? Maybe the scanner blocs files.

Some other items what is a common procedure (for me) just to exclude such possibilities....
C:\Windows\Microsoft.NET\Framework64\xxx\Temporary ASP.NET Files
You may clean them  up....

And also the search engine may involve, so recreating the index may be just a try....
Amazon Web Services

Are you thinking about creating an Amazon Web Services account for your business? Not sure where to start? In this course you’ll get an overview of the history of AWS and take a tour of their user interface.

Thomas GrassiSystems Administrator

Author

Commented:
Bembi

Current App Pool account groups

PS C:\util> Get-ADPrincipalGroupMembership spapppool | select name

name
----
Domain Users
WSS_WPG


SPFARM account is in the local group administrators and has been since I started



The account SPAPPPOOL is set on the services in SharePoint CA Service Accounts Credential Management



Sorry

Does not happen when I do an IISRESET  only when I do maint on the app pool entry or at the recycle time

one other time I see this is when in sharepoint I delete a service then recreate it
sharepoint automatically updates IIS and sometimes I see it then


Should I just change it to use the SPFARM account then?
> The account SPAPPPOOL is set on the services in SharePoint CA Service Accounts Credential Management
This I'm not quite clear about...
I meant SharePoint CA - Security - Configure Service accounts...
My question was, which services are assigned to SPAPPPOOL as account.

A second question, SharePoint CA - Security - Configure managed accounts...
Can you check if any of the accounts have set "Enable automatic password change"?
This is due to the crypto messages in the event log.

The crypto message also points me to certificates in general.
Do you use SLL on the IIS sites?
If yes, how have you got the certs?

Last but not least this blog I found....
http://blogs.msdn.com/b/tess/archive/2007/10/31/asp-net-crash-system-security-cryptography-cryptographicexception.aspx
It shows at least similar errors..., nevertheless the article doesn't point out a solution, but in the comment is also the hint to change the pool account.
Thomas GrassiSystems Administrator

Author

Commented:
Thanks

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial