Need help troubleshooting IIS 6 error code 1000, Faulting application w3wp.exe

Posted on 2011-05-05
Last Modified: 2012-05-11
Event Type:      Error
Event Source:      Application Error
Event Category:      (100)
Event ID:      1000
Date:            5/5/2011
Time:            9:20:26 AM
User:            N/A
Computer:      NJ1ESP47V
Faulting application w3wp.exe, version 6.0.3790.3959, faulting module ntdll.dll, version 5.2.3790.4789, fault address 0x0000000000035f13.

For more information, see Help and Support Center at
0000: 41 70 70 6c 69 63 61 74   Applicat
0008: 69 6f 6e 20 46 61 69 6c   ion Fail
0010: 75 72 65 20 20 77 33 77   ure  w3w
0018: 70 2e 65 78 65 20 36 2e   p.exe 6.
0020: 30 2e 33 37 39 30 2e 33   0.3790.3
0028: 39 35 39 20 69 6e 20 6e   959 in n
0030: 74 64 6c 6c 2e 64 6c 6c   tdll.dll
0038: 20 35 2e 32 2e 33 37 39    5.2.379
0040: 30 2e 34 37 38 39 20 61   0.4789 a
0048: 74 20 6f 66 66 73 65 74   t offset
0050: 20 30 30 30 30 30 30 30    0000000
0058: 30 30 30 30 33 35 66 31   000035f1
0060: 33                        3      
Question by:esphelpdesk
    LVL 4

    Accepted Solution

    It can be many things.

    Application causing your your iis to crash.
    any permission issue with your network service account.

    Allocate seperate pools to each website . Are you using any php based website ?
    Please give lil details about your setup.

    Author Comment

    Each website has its own application pool.  The website is created in Cold Fusion.  It runs Coldfusion Server on top of IIS.  Operating system is Windows 2003 R2 64bit.  There's an SSO(single sign on) application called OpenAM agent that is installed in IIS (this is most likely the cause of the problem, but not sure how?).  I get a lot of these warnings...
    Event Type:      Warning
    Event Source:      W3SVC
    Event Category:      None
    Event ID:      1009
    Date:            5/6/2011
    Time:            11:25:36 AM
    User:            N/A
    Computer:      NJ1ESP47V
    A process serving application pool 'SSOAppPool' terminated unexpectedly. The process id was '300'. The process exit code was '0x80'.

    ...with different process ID's.

    I turned on Failure Audit logging, and I'm seeing these events...

    Event Type:      Failure Audit
    Event Source:      Security
    Event Category:      Object Access
    Event ID:      560
    Date:            5/6/2011
    Time:            12:38:15 PM
    Computer:      NJ1ESP47V
    Object Open:
           Object Server:      Security
           Object Type:      File
           Object Name:      C:\WINDOWS\Debug\UserMode\ChkAcc.log
           Handle ID:      -
           Operation ID:      {0,34536284}
           Process ID:      2880
           Image File Name:      C:\WINDOWS\system32\inetsrv\w3wp.exe
           Primary User Name:      NETWORK SERVICE
           Primary Domain:      NT AUTHORITY
           Primary Logon ID:      (0x0,0x3E4)
           Client User Name:      -
           Client Domain:      -
           Client Logon ID:      -
           Accesses:      READ_CONTROL
                      WriteData (or AddFile)
                      AppendData (or AddSubdirectory or CreatePipeInstance)
           Privileges:      -
           Restricted Sid Count:      0
           Access Mask:      0x120196

    Any ideas?  Let me know if you need more info.
    LVL 4

    Expert Comment

    check event log of coldfusion.

    Author Comment

    After opening a ticket with Microsoft to chase down the error, it looks like a java agent (amiis6.dll) is the culprit.  Using debugdiag to log the crashes to a dump file, it is showing that this dll is causing access violations.  

    I just added Java as a keyword to this thread.
    LVL 4

    Assisted Solution

    What is the network stats when a crash happens. Is there a sudden spike in concurrent users ?

    Author Comment

    yes.  however this application is not stressing the server by any means.... perhaps a 100 users.  A very low load.
    LVL 4

    Expert Comment

    looks like coldfusion is the culprit.
    LVL 4

    Expert Comment

    what is the frequency of the crash ?

    Author Comment

    Approximately 20-40 minutes.  The crash is hardly noticeable, but my goal is to stop the hangs and pauses in the application.  

    When looking at the crash dumps...
    - The first one shows crashing thread as : w3wp__PID__816__Date__05_10_2011__Time_01_44_44PM__909__Second_Chance_Exception_C0000005.dmp

    0:006> kL99
    Child-SP          RetAddr           Call Site
    00000000`0199f8a0 00000000`77ef6040 ntdll!RtlTimeFieldsToTime+0x864
    00000000`0199f8e0 00000000`02c6f6b0 ntdll!RtlTimeFieldsToTime+0x8e0
    00000000`0199f9f0 00000000`02bd5165 amsdk!decode_base64+0x437e0
    00000000`0199fa20 000007ff`5e4421a1 amiis6!HttpExtensionProc+0xfb5
    00000000`0199fbd0 000007ff`5e5118d4 w3isapi!ProcessIsapiRequest+0x39d
    00000000`0199fc60 000007ff`5e4d757a w3core!W3_ISAPI_HANDLER::IsapiDoWork+0x601
    00000000`0199fcf0 000007ff`5e501dd7 w3core!W3_HANDLER::MainDoWork+0x2ba
    00000000`0199fd60 000007ff`5e51b262 w3core!W3_CONTEXT::ExecuteHandler+0x177
    00000000`0199fda0 000007ff`5e508716 w3core!W3_STATE_HANDLE_REQUEST::DoWork+0xe2
    00000000`0199fe10 000007ff`5e4915fb w3core!W3_MAIN_CONTEXT::OnNewRequest+0xc9
    00000000`0199fe50 000007ff`5e491538 w3dt!UL_NATIVE_REQUEST::DoStateProcess+0x7b
    00000000`0199fe80 000007ff`5e49148c w3dt!UL_NATIVE_REQUEST::DoWork+0x157
    00000000`0199fec0 000007ff`5e4127e1 w3dt!OverlappedCompletionRoutine+0x1c
    00000000`0199fef0 000007ff`5e4116eb w3tp!THREAD_POOL_DATA::ThreadPoolThread+0xd1
    00000000`0199ff50 00000000`77d6b71a w3tp!THREAD_MANAGER::ThreadManagerThread+0x6b
    00000000`0199ff80 00000000`00000000 kernel32!BaseThreadStart+0x3a

    - We have  amiis6 dll configured as ISAPI. Based on above call stack, the thread 13 encountered Access Violation exception and the web request went through IIS modules and got to amiis6.dll.
    - The amiis6 module called one function of amsdk.dll to free heap memory. And the Access Violation exception happened when the amsdk function called kernel system function ntdll!RtlpLowFragHeapFree to free heap.

    Sun just released a new version of the amiis6.dll, so I'm going to try upgrading and see if that clears up the issue.
    LVL 67

    Expert Comment

    I've requested that this question be deleted for the following reason:

    This question has been classified as abandoned and is closed as part of the Cleanup Program. See the recommendation for more details.

    Assisted Solution

    The Sun Microsystems amiis6.dll module was determined to be the cause of this crash.  A new version was released, but did not fix the issue.  My next step is to move this particular website to a new Operating System (Windows 2008 R2) and try that version of the Sun agent.

    Perhaps keep this question as CLOSED instead of ABANDONED?

    Author Closing Comment

    This question came up in many web searches, but seems to chronically have no solution.  I ended up calling Microsoft and it was determined that a 3rd party dll (from Sun Microsystems) was causing IIS to crash.  This is an unfixable incompatibility between amiis6.dll and the Windows 2003 64 bit version of IIS.

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    How your wiki can always stay up-to-date

    Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
    - Increase transparency
    - Onboard new hires faster
    - Access from mobile/offline

    Prologue It is often required to host multiple websites on a single instance of IIS, mostly in development environments instead of on production servers. I am sure it is not much a preferred solution on production servers but this is at least a pos…
    This tutorial will go through the steps required to write a script that will back up the configuration settings of a HP-ProCurve switch. You will need to get the following things to follow this tutorial: Telnet Scripting Tool e.g. TST10.exe …
    It is a freely distributed piece of software for such tasks as photo retouching, image composition and image authoring. It works on many operating systems, in many languages.
    Illustrator's Shape Builder tool will let you combine shapes visually and interactively. This video shows the Mac version, but the tool works the same way in Windows. To follow along with this video, you can draw your own shapes or download the file…

    759 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    14 Experts available now in Live!

    Get 1:1 Help Now