Need help troubleshooting IIS 6 error code 1000, Faulting application w3wp.exe

Event Type:      Error
Event Source:      Application Error
Event Category:      (100)
Event ID:      1000
Date:            5/5/2011
Time:            9:20:26 AM
User:            N/A
Computer:      NJ1ESP47V
Description:
Faulting application w3wp.exe, version 6.0.3790.3959, faulting module ntdll.dll, version 5.2.3790.4789, fault address 0x0000000000035f13.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74   Applicat
0008: 69 6f 6e 20 46 61 69 6c   ion Fail
0010: 75 72 65 20 20 77 33 77   ure  w3w
0018: 70 2e 65 78 65 20 36 2e   p.exe 6.
0020: 30 2e 33 37 39 30 2e 33   0.3790.3
0028: 39 35 39 20 69 6e 20 6e   959 in n
0030: 74 64 6c 6c 2e 64 6c 6c   tdll.dll
0038: 20 35 2e 32 2e 33 37 39    5.2.379
0040: 30 2e 34 37 38 39 20 61   0.4789 a
0048: 74 20 6f 66 66 73 65 74   t offset
0050: 20 30 30 30 30 30 30 30    0000000
0058: 30 30 30 30 33 35 66 31   000035f1
0060: 33                        3      
esphelpdeskAsked:
Who is Participating?
 
contactrobolCommented:
It can be many things.

Application causing your your iis to crash.
any permission issue with your network service account.
bottleneck.

Allocate seperate pools to each website . Are you using any php based website ?
Please give lil details about your setup.
0
 
esphelpdeskAuthor Commented:
Each website has its own application pool.  The website is created in Cold Fusion.  It runs Coldfusion Server on top of IIS.  Operating system is Windows 2003 R2 64bit.  There's an SSO(single sign on) application called OpenAM agent that is installed in IIS (this is most likely the cause of the problem, but not sure how?).  I get a lot of these warnings...
Event Type:      Warning
Event Source:      W3SVC
Event Category:      None
Event ID:      1009
Date:            5/6/2011
Time:            11:25:36 AM
User:            N/A
Computer:      NJ1ESP47V
Description:
A process serving application pool 'SSOAppPool' terminated unexpectedly. The process id was '300'. The process exit code was '0x80'.

...with different process ID's.

I turned on Failure Audit logging, and I'm seeing these events...

Event Type:      Failure Audit
Event Source:      Security
Event Category:      Object Access
Event ID:      560
Date:            5/6/2011
Time:            12:38:15 PM
User:            NT AUTHORITY\NETWORK SERVICE
Computer:      NJ1ESP47V
Description:
Object Open:
       Object Server:      Security
       Object Type:      File
       Object Name:      C:\WINDOWS\Debug\UserMode\ChkAcc.log
       Handle ID:      -
       Operation ID:      {0,34536284}
       Process ID:      2880
       Image File Name:      C:\WINDOWS\system32\inetsrv\w3wp.exe
       Primary User Name:      NETWORK SERVICE
       Primary Domain:      NT AUTHORITY
       Primary Logon ID:      (0x0,0x3E4)
       Client User Name:      -
       Client Domain:      -
       Client Logon ID:      -
       Accesses:      READ_CONTROL
                  SYNCHRONIZE
                  WriteData (or AddFile)
                  AppendData (or AddSubdirectory or CreatePipeInstance)
                  WriteEA
                  ReadAttributes
                  WriteAttributes
                  
       Privileges:      -
       Restricted Sid Count:      0
       Access Mask:      0x120196


Any ideas?  Let me know if you need more info.
0
 
contactrobolCommented:
check event log of coldfusion.
0
Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

 
esphelpdeskAuthor Commented:
After opening a ticket with Microsoft to chase down the error, it looks like a java agent (amiis6.dll) is the culprit.  Using debugdiag to log the crashes to a dump file, it is showing that this dll is causing access violations.  

I just added Java as a keyword to this thread.
0
 
contactrobolCommented:
What is the network stats when a crash happens. Is there a sudden spike in concurrent users ?
0
 
esphelpdeskAuthor Commented:
yes.  however this application is not stressing the server by any means.... perhaps a 100 users.  A very low load.
0
 
contactrobolCommented:
looks like coldfusion is the culprit.
0
 
contactrobolCommented:
what is the frequency of the crash ?
0
 
esphelpdeskAuthor Commented:
Approximately 20-40 minutes.  The crash is hardly noticeable, but my goal is to stop the hangs and pauses in the application.  

When looking at the crash dumps...
- The first one shows crashing thread as : w3wp__PID__816__Date__05_10_2011__Time_01_44_44PM__909__Second_Chance_Exception_C0000005.dmp


0:006> kL99
Child-SP          RetAddr           Call Site
00000000`0199f8a0 00000000`77ef6040 ntdll!RtlTimeFieldsToTime+0x864
00000000`0199f8e0 00000000`02c6f6b0 ntdll!RtlTimeFieldsToTime+0x8e0
00000000`0199f9f0 00000000`02bd5165 amsdk!decode_base64+0x437e0
00000000`0199fa20 000007ff`5e4421a1 amiis6!HttpExtensionProc+0xfb5
00000000`0199fbd0 000007ff`5e5118d4 w3isapi!ProcessIsapiRequest+0x39d
00000000`0199fc60 000007ff`5e4d757a w3core!W3_ISAPI_HANDLER::IsapiDoWork+0x601
00000000`0199fcf0 000007ff`5e501dd7 w3core!W3_HANDLER::MainDoWork+0x2ba
00000000`0199fd60 000007ff`5e51b262 w3core!W3_CONTEXT::ExecuteHandler+0x177
00000000`0199fda0 000007ff`5e508716 w3core!W3_STATE_HANDLE_REQUEST::DoWork+0xe2
00000000`0199fe10 000007ff`5e4915fb w3core!W3_MAIN_CONTEXT::OnNewRequest+0xc9
00000000`0199fe50 000007ff`5e491538 w3dt!UL_NATIVE_REQUEST::DoStateProcess+0x7b
00000000`0199fe80 000007ff`5e49148c w3dt!UL_NATIVE_REQUEST::DoWork+0x157
00000000`0199fec0 000007ff`5e4127e1 w3dt!OverlappedCompletionRoutine+0x1c
00000000`0199fef0 000007ff`5e4116eb w3tp!THREAD_POOL_DATA::ThreadPoolThread+0xd1
00000000`0199ff50 00000000`77d6b71a w3tp!THREAD_MANAGER::ThreadManagerThread+0x6b
00000000`0199ff80 00000000`00000000 kernel32!BaseThreadStart+0x3a


- We have  amiis6 dll configured as ISAPI. Based on above call stack, the thread 13 encountered Access Violation exception and the web request went through IIS modules and got to amiis6.dll.
- The amiis6 module called one function of amsdk.dll to free heap memory. And the Access Violation exception happened when the amsdk function called kernel system function ntdll!RtlpLowFragHeapFree to free heap.

Sun just released a new version of the amiis6.dll, so I'm going to try upgrading and see if that clears up the issue.
0
 
QlemoBatchelor, Developer and EE Topic AdvisorCommented:
I've requested that this question be deleted for the following reason:

This question has been classified as abandoned and is closed as part of the Cleanup Program. See the recommendation for more details.
0
 
esphelpdeskAuthor Commented:
The Sun Microsystems amiis6.dll module was determined to be the cause of this crash.  A new version was released, but did not fix the issue.  My next step is to move this particular website to a new Operating System (Windows 2008 R2) and try that version of the Sun agent.

Perhaps keep this question as CLOSED instead of ABANDONED?
0
 
esphelpdeskAuthor Commented:
This question came up in many web searches, but seems to chronically have no solution.  I ended up calling Microsoft and it was determined that a 3rd party dll (from Sun Microsystems) was causing IIS to crash.  This is an unfixable incompatibility between amiis6.dll and the Windows 2003 64 bit version of IIS.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.