[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Need help troubleshooting IIS 6 error code 1000, Faulting application w3wp.exe

Posted on 2011-05-05
13
Medium Priority
?
2,171 Views
Last Modified: 2012-05-11
Event Type:      Error
Event Source:      Application Error
Event Category:      (100)
Event ID:      1000
Date:            5/5/2011
Time:            9:20:26 AM
User:            N/A
Computer:      NJ1ESP47V
Description:
Faulting application w3wp.exe, version 6.0.3790.3959, faulting module ntdll.dll, version 5.2.3790.4789, fault address 0x0000000000035f13.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74   Applicat
0008: 69 6f 6e 20 46 61 69 6c   ion Fail
0010: 75 72 65 20 20 77 33 77   ure  w3w
0018: 70 2e 65 78 65 20 36 2e   p.exe 6.
0020: 30 2e 33 37 39 30 2e 33   0.3790.3
0028: 39 35 39 20 69 6e 20 6e   959 in n
0030: 74 64 6c 6c 2e 64 6c 6c   tdll.dll
0038: 20 35 2e 32 2e 33 37 39    5.2.379
0040: 30 2e 34 37 38 39 20 61   0.4789 a
0048: 74 20 6f 66 66 73 65 74   t offset
0050: 20 30 30 30 30 30 30 30    0000000
0058: 30 30 30 30 33 35 66 31   000035f1
0060: 33                        3      
0
Comment
Question by:esphelpdesk
  • 6
  • 5
12 Comments
 
LVL 4

Accepted Solution

by:
contactrobol earned 2000 total points
ID: 35702965
It can be many things.

Application causing your your iis to crash.
any permission issue with your network service account.
bottleneck.

Allocate seperate pools to each website . Are you using any php based website ?
Please give lil details about your setup.
0
 

Author Comment

by:esphelpdesk
ID: 35707663
Each website has its own application pool.  The website is created in Cold Fusion.  It runs Coldfusion Server on top of IIS.  Operating system is Windows 2003 R2 64bit.  There's an SSO(single sign on) application called OpenAM agent that is installed in IIS (this is most likely the cause of the problem, but not sure how?).  I get a lot of these warnings...
Event Type:      Warning
Event Source:      W3SVC
Event Category:      None
Event ID:      1009
Date:            5/6/2011
Time:            11:25:36 AM
User:            N/A
Computer:      NJ1ESP47V
Description:
A process serving application pool 'SSOAppPool' terminated unexpectedly. The process id was '300'. The process exit code was '0x80'.

...with different process ID's.

I turned on Failure Audit logging, and I'm seeing these events...

Event Type:      Failure Audit
Event Source:      Security
Event Category:      Object Access
Event ID:      560
Date:            5/6/2011
Time:            12:38:15 PM
User:            NT AUTHORITY\NETWORK SERVICE
Computer:      NJ1ESP47V
Description:
Object Open:
       Object Server:      Security
       Object Type:      File
       Object Name:      C:\WINDOWS\Debug\UserMode\ChkAcc.log
       Handle ID:      -
       Operation ID:      {0,34536284}
       Process ID:      2880
       Image File Name:      C:\WINDOWS\system32\inetsrv\w3wp.exe
       Primary User Name:      NETWORK SERVICE
       Primary Domain:      NT AUTHORITY
       Primary Logon ID:      (0x0,0x3E4)
       Client User Name:      -
       Client Domain:      -
       Client Logon ID:      -
       Accesses:      READ_CONTROL
                  SYNCHRONIZE
                  WriteData (or AddFile)
                  AppendData (or AddSubdirectory or CreatePipeInstance)
                  WriteEA
                  ReadAttributes
                  WriteAttributes
                  
       Privileges:      -
       Restricted Sid Count:      0
       Access Mask:      0x120196


Any ideas?  Let me know if you need more info.
0
 
LVL 4

Expert Comment

by:contactrobol
ID: 35716968
check event log of coldfusion.
0
NEW Veeam Backup for Microsoft Office 365 1.5

With Office 365, it’s your data and your responsibility to protect it. NEW Veeam Backup for Microsoft Office 365 eliminates the risk of losing access to your Office 365 data.

 

Author Comment

by:esphelpdesk
ID: 35741649
After opening a ticket with Microsoft to chase down the error, it looks like a java agent (amiis6.dll) is the culprit.  Using debugdiag to log the crashes to a dump file, it is showing that this dll is causing access violations.  

I just added Java as a keyword to this thread.
0
 
LVL 4

Assisted Solution

by:contactrobol
contactrobol earned 2000 total points
ID: 35741983
What is the network stats when a crash happens. Is there a sudden spike in concurrent users ?
0
 

Author Comment

by:esphelpdesk
ID: 35742005
yes.  however this application is not stressing the server by any means.... perhaps a 100 users.  A very low load.
0
 
LVL 4

Expert Comment

by:contactrobol
ID: 35742031
looks like coldfusion is the culprit.
0
 
LVL 4

Expert Comment

by:contactrobol
ID: 35742041
what is the frequency of the crash ?
0
 

Author Comment

by:esphelpdesk
ID: 35742112
Approximately 20-40 minutes.  The crash is hardly noticeable, but my goal is to stop the hangs and pauses in the application.  

When looking at the crash dumps...
- The first one shows crashing thread as : w3wp__PID__816__Date__05_10_2011__Time_01_44_44PM__909__Second_Chance_Exception_C0000005.dmp


0:006> kL99
Child-SP          RetAddr           Call Site
00000000`0199f8a0 00000000`77ef6040 ntdll!RtlTimeFieldsToTime+0x864
00000000`0199f8e0 00000000`02c6f6b0 ntdll!RtlTimeFieldsToTime+0x8e0
00000000`0199f9f0 00000000`02bd5165 amsdk!decode_base64+0x437e0
00000000`0199fa20 000007ff`5e4421a1 amiis6!HttpExtensionProc+0xfb5
00000000`0199fbd0 000007ff`5e5118d4 w3isapi!ProcessIsapiRequest+0x39d
00000000`0199fc60 000007ff`5e4d757a w3core!W3_ISAPI_HANDLER::IsapiDoWork+0x601
00000000`0199fcf0 000007ff`5e501dd7 w3core!W3_HANDLER::MainDoWork+0x2ba
00000000`0199fd60 000007ff`5e51b262 w3core!W3_CONTEXT::ExecuteHandler+0x177
00000000`0199fda0 000007ff`5e508716 w3core!W3_STATE_HANDLE_REQUEST::DoWork+0xe2
00000000`0199fe10 000007ff`5e4915fb w3core!W3_MAIN_CONTEXT::OnNewRequest+0xc9
00000000`0199fe50 000007ff`5e491538 w3dt!UL_NATIVE_REQUEST::DoStateProcess+0x7b
00000000`0199fe80 000007ff`5e49148c w3dt!UL_NATIVE_REQUEST::DoWork+0x157
00000000`0199fec0 000007ff`5e4127e1 w3dt!OverlappedCompletionRoutine+0x1c
00000000`0199fef0 000007ff`5e4116eb w3tp!THREAD_POOL_DATA::ThreadPoolThread+0xd1
00000000`0199ff50 00000000`77d6b71a w3tp!THREAD_MANAGER::ThreadManagerThread+0x6b
00000000`0199ff80 00000000`00000000 kernel32!BaseThreadStart+0x3a


- We have  amiis6 dll configured as ISAPI. Based on above call stack, the thread 13 encountered Access Violation exception and the web request went through IIS modules and got to amiis6.dll.
- The amiis6 module called one function of amsdk.dll to free heap memory. And the Access Violation exception happened when the amsdk function called kernel system function ntdll!RtlpLowFragHeapFree to free heap.

Sun just released a new version of the amiis6.dll, so I'm going to try upgrading and see if that clears up the issue.
0
 
LVL 71

Expert Comment

by:Qlemo
ID: 35905213
I've requested that this question be deleted for the following reason:

This question has been classified as abandoned and is closed as part of the Cleanup Program. See the recommendation for more details.
0
 

Assisted Solution

by:esphelpdesk
esphelpdesk earned 0 total points
ID: 35905214
The Sun Microsystems amiis6.dll module was determined to be the cause of this crash.  A new version was released, but did not fix the issue.  My next step is to move this particular website to a new Operating System (Windows 2008 R2) and try that version of the Sun agent.

Perhaps keep this question as CLOSED instead of ABANDONED?
0
 

Author Closing Comment

by:esphelpdesk
ID: 35950889
This question came up in many web searches, but seems to chronically have no solution.  I ended up calling Microsoft and it was determined that a 3rd party dll (from Sun Microsystems) was causing IIS to crash.  This is an unfixable incompatibility between amiis6.dll and the Windows 2003 64 bit version of IIS.
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Debug Tools to analyse IIS process: This article focus on taking memory dumps from IIS to determine which code is taking more time and to analyse which calls hangs/causes more CPU usage. To take dumps,download the following. Install1: To st…
If you are a web developer, you would be aware of the <iframe> tag in HTML. The <iframe> stands for inline frame and is used to embed another document within the current HTML document. The embedded document could be even another website.
We’ve all felt that sense of false security before—locking down external access to a database or component and feeling like we’ve done all we need to do to secure company data. But that feeling is fleeting. Attacks these days can happen in many w…
Whether it be Exchange Server Crash Issues, Dirty Shutdown Errors or Failed to mount error, Stellar Phoenix Mailbox Exchange Recovery has always got your back. With the help of its easy to understand user interface and 3 simple steps recovery proced…

873 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question