Application Pool Recycle causes failure

Windows 2012 R2 Web Server
IIS 8.5
SharePoint 2010 SP2 November CU 2015


Everyday at 2:49 AM I get the following errors one in the system log and one in the application log

Log Name:      Application
Source:        ASP.NET 2.0.50727.0
Date:          11/13/2015 2:49:03 AM
Event ID:      1334
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SERV013.FQDN.com
Description:
An unhandled exception occurred and the process was terminated.

Application ID: /LM/W3SVC/1068268973/ROOT

Process ID: 7540

Exception: System.Security.Cryptography.CryptographicException

Message: Keyset does not exist


StackTrace:    at System.Security.Cryptography.CryptographicException.ThrowCryptogaphicException(Int32 hr)
   at System.Security.Cryptography.SafeProvHandle._FreeCSP(IntPtr pProvCtx)
   at System.Security.Cryptography.SafeProvHandle.ReleaseHandle()
   at System.Runtime.InteropServices.SafeHandle.InternalFinalize()
   at System.Runtime.InteropServices.SafeHandle.Dispose(Boolean disposing)
   at System.Runtime.InteropServices.SafeHandle.Finalize()
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="ASP.NET 2.0.50727.0" />
    <EventID Qualifiers="49152">1334</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-11-13T07:49:03.000000000Z" />
    <EventRecordID>90691</EventRecordID>
    <Channel>Application</Channel>
    <Computer>TGCS013.our.network.tgcsnet.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>An unhandled exception occurred and the process was terminated.

Application ID: /LM/W3SVC/1068268973/ROOT

Process ID: 7540

Exception: System.Security.Cryptography.CryptographicException

Message: Keyset does not exist


StackTrace:    at System.Security.Cryptography.CryptographicException.ThrowCryptogaphicException(Int32 hr)
   at System.Security.Cryptography.SafeProvHandle._FreeCSP(IntPtr pProvCtx)
   at System.Security.Cryptography.SafeProvHandle.ReleaseHandle()
   at System.Runtime.InteropServices.SafeHandle.InternalFinalize()
   at System.Runtime.InteropServices.SafeHandle.Dispose(Boolean disposing)
   at System.Runtime.InteropServices.SafeHandle.Finalize()</Data>
  </EventData>
</Event>



Log Name:      System
Source:        Microsoft-Windows-WAS
Date:          11/13/2015 2:49:04 AM
Event ID:      5009
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      SERV013.FQDN.com
Description:
A process serving application pool 'SharePoint - Default App Pool' terminated unexpectedly. The process id was '7540'. The process exit code was '0xe0434f4d'.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-WAS" Guid="{524B5D04-133C-4A62-8362-64E8EDB9CE40}" EventSourceName="WAS" />
    <EventID Qualifiers="32768">5009</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-11-13T07:49:04.000000000Z" />
    <EventRecordID>74547</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>System</Channel>
    <Computer>SERV013.FQDN.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="AppPoolID">SharePoint - Default App Pool</Data>
    <Data Name="ProcessID">7540</Data>
    <Data Name="ExitCode">e0434f4d</Data>
  </EventData>
</Event>

My research found these

From the Event ID 5009 ran err.exe   on 0xe0434f4d

C:\Util\Err>err 0xe0434f4d
# as an HRESULT: Severity: FAILURE (1), Facility: 0x43, Code 0x4f4d
# NOT FOUND: 0xe0434f4d

Do not understand what is NOT FOUND

This seems to be COM permissions?????????
Command prompt->dcomcnfg->component services->computers->right click on my computer->properties.
Go to COM security
Under launch and activation permissions
Add the SID S-1-5-82-3006700770-424185619-1745488364-794895919-4004696415 and provide local launch permissions.

then try and check if IISRESET still throws same error in event log.

I added spapppool account

No change

I spent many hours trying to figure out why everyday at 2:49 AM then by dumb luck I found it.

From within IIS I expanded the Application Pools list after scrolling thru each entry I found that

Application Pool SharePoint - Default App Pool is started  v2.0 Integrated identity mydom\spapppool

when I clicked on Recycling Settings  I see that it is set for Specific times at 2:49 AM


I also installed DebugDiag 2 Analysis and I have a dump  but I a, not sure what to look for


Any thoughts

Thanks in advance

Tom
LVL 23
Thomas GrassiSystems AdministratorAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

BembiCEOCommented:
What I interpret is that the error is older than NOV 2015 CU, right?
Or did it come up with NOV 2015 CU?

What pointed you to the DCOM description, did you have this error in combination with the other error?

Does the pool recycle?

I would expect that this may be a permission problem. If you see a DCOM error, it can even come up,
if a component tries to start too early. Seen it more often with local accounts as local service or network service. So interesting in this combination is the farm count and the identity of the app pool.
In which local groups are they a member?

Also keep in mind that changing permissions need sometimes a logoff. So you have to shut down al the services running under this account and restart them again (or just reboot the machine.

So you said the error comes also up, if you run IISReset, right? Are you farm admin? In which local groups is your account?

What maybe a fast try is to change the App Pool (via the SharePoint service accounts ) to a temporary account, IISRESET, change it back, run the SharePoint config wizard and see, if it makes any change. Also you may reboot the machine and see, if the behavior change.
This procedure may reset all needed permissions for the pool account.
Thomas GrassiSystems AdministratorAuthor Commented:
Bembi

First thanks for responding

What I interpret is that the error is older than NOV 2015 CU, right?

Yes this error has been happing for a long time I even rebuilt my entire SharePoint Farm 3 times and still continue to get this error

Or did it come up with NOV 2015 CU?   NO see above reply

I would expect that this may be a permission problem. If you see a DCOM error, it can even come up,
 if a component tries to start too early. Seen it more often with local accounts as local service or network service. So interesting in this combination is the farm count and the identity of the app pool.
 In which local groups are they a member?


No Dcom error in my logs I just came across an article in my research of the above errors
I wish I had the link  to show but my computer at work my browser died and I lost the link that day.

Also keep in mind that changing permissions need sometimes a logoff. So you have to shut down al the services running under this account and restart them again (or just reboot the machine.


Yes I know I restart/reboot my server at least once a month to apply updates.
I also changed my login account to use the sharepoint farm account  spfarm

So you said the error comes also up, if you run IISReset, right?
Yes they do even when I modify the application pool as I did today to remove the scheduled time of 2:49 AM

Are you farm admin?  yes I am

which local groups is your account?  on the server local Users and Groups  "Administrators"
On the domain
PS C:\util> Get-ADPrincipalGroupMembership spfarm | select name

name
----
Domain Users
WSS_WPG
WSS_ADMIN_WPG
SharePointAdmins

What maybe a fast try is to change the App Pool (via the SharePoint service accounts ) to a temporary account, IISRESET, change it back, run the SharePoint config wizard and see, if it makes any change. Also you may reboot the machine and see, if the behavior change.
 This procedure may reset all needed permissions for the pool account.



I will try this and report back
BembiCEOCommented:
No problem, ...
You may also post the group membership of the pool account identity.

Another check is just, to put the farm account into the local admins groups. (Reboot or restart of the affected services running under farm account).

The farm account (timer service) runs the job  to recycle. So if there is a permission problem with the farm account, the error may move away with admin permissions. But also the pool account is under inspection, and need not necessarily only the web site. Check the service accounts in SharePoint, which services are using the pool account as well.

Next step would be (but for this I need my systems) to check the permissions in the web application.

Assigning the service accounts in SharePoint should at least put them into the correct groups. Configuration wizard then resets the permissions on files, registry and the database.

I see also in you logs errors in he shut down....
So my question would be, if you use IISReset..., do you see only errors in the event log or so also errors in the command line?

Have you tried to disable the virus scan during IISReset? Maybe the scanner blocs files.

Some other items what is a common procedure (for me) just to exclude such possibilities....
C:\Windows\Microsoft.NET\Framework64\xxx\Temporary ASP.NET Files
You may clean them  up....

And also the search engine may involve, so recreating the index may be just a try....
Expert Spotlight: Joe Anderson (DatabaseMX)

We’ve posted a new Expert Spotlight!  Joe Anderson (DatabaseMX) has been on Experts Exchange since 2006. Learn more about this database architect, guitar aficionado, and Microsoft MVP.

Thomas GrassiSystems AdministratorAuthor Commented:
Bembi

Current App Pool account groups

PS C:\util> Get-ADPrincipalGroupMembership spapppool | select name

name
----
Domain Users
WSS_WPG


SPFARM account is in the local group administrators and has been since I started



The account SPAPPPOOL is set on the services in SharePoint CA Service Accounts Credential Management



Sorry

Does not happen when I do an IISRESET  only when I do maint on the app pool entry or at the recycle time

one other time I see this is when in sharepoint I delete a service then recreate it
sharepoint automatically updates IIS and sometimes I see it then


Should I just change it to use the SPFARM account then?
BembiCEOCommented:
> The account SPAPPPOOL is set on the services in SharePoint CA Service Accounts Credential Management
This I'm not quite clear about...
I meant SharePoint CA - Security - Configure Service accounts...
My question was, which services are assigned to SPAPPPOOL as account.

A second question, SharePoint CA - Security - Configure managed accounts...
Can you check if any of the accounts have set "Enable automatic password change"?
This is due to the crypto messages in the event log.

The crypto message also points me to certificates in general.
Do you use SLL on the IIS sites?
If yes, how have you got the certs?

Last but not least this blog I found....
http://blogs.msdn.com/b/tess/archive/2007/10/31/asp-net-crash-system-security-cryptography-cryptographicexception.aspx
It shows at least similar errors..., nevertheless the article doesn't point out a solution, but in the comment is also the hint to change the pool account.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Thomas GrassiSystems AdministratorAuthor Commented:
Thanks
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft IIS Web Server

From novice to tech pro — start learning today.