Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

IIS - Threads in Wait State. IISSTATE Log included

Posted on 2005-04-22
13
Medium Priority
?
795 Views
Last Modified: 2012-06-21
I have included a log from IISSTATE below.  My site is bogging down with very little activity.  It is IIS 6.0 on Windows 2003 running in IIS 5.0 mode.    Site is ASP with ACCESS DB.  Plan to go to SQL Server, but not there yet.
The DLLHOST.EXE that this is a trace of has User Name IWAM_machinename.   This DLLHOST.EXE will be at 0% CPU and about 40MB Mem Usage, but will shoot up to 50 to 70% CPU and upwards of 300MB Mem Usage.   I got this log at one of those times.  It is consistenly doing it about every minute or so.  I have no idea how to read this log.  I have the dump also if anyone wants to take a shot.  Site is averaging about 2 to 4 users at any given time.  Practically every page hits 3 different ACCESS databases.

Terry Roller

Kernel Time: 0:0:55.984
User Time: 0:0:59.218
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
 # ChildEBP RetAddr
00 08f0fdcc 77f4372d SharedUserData!SystemCallStub+0x4
01 08f0fdd0 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
02 08f0fe78 77d02695 kernel32!WaitForMultipleObjectsEx+0xcb
03 08f0fed4 77d037f0 USER32!RealMsgWaitForMultipleObjectsEx+0x13f
04 08f0fef0 75614e57 USER32!MsgWaitForMultipleObjects+0x1d
05 08f0ff84 77bc91ed COMSVCS!CSTAThread::WorkerLoop+0x1e3
06 08f0ffb8 77e4a990 msvcrt!_endthreadex+0x95
07 08f0ffec 00000000 kernel32!FlsSetValue+0x779




Thread ID: 64
System Thread ID: f28
Kernel Time: 0:0:41.453
User Time: 0:0:45.531
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
 # ChildEBP RetAddr
00 08f4fdcc 77f4372d SharedUserData!SystemCallStub+0x4
01 08f4fdd0 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
02 08f4fe78 77d02695 kernel32!WaitForMultipleObjectsEx+0xcb
03 08f4fed4 77d037f0 USER32!RealMsgWaitForMultipleObjectsEx+0x13f
04 08f4fef0 75614e57 USER32!MsgWaitForMultipleObjects+0x1d
05 08f4ff84 77bc91ed COMSVCS!CSTAThread::WorkerLoop+0x1e3
06 08f4ffb8 77e4a990 msvcrt!_endthreadex+0x95
07 08f4ffec 00000000 kernel32!FlsSetValue+0x779




Thread ID: 65
System Thread ID: f88
Kernel Time: 0:1:5.703
User Time: 0:1:11.734
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
 # ChildEBP RetAddr
00 006ffdcc 77f4372d SharedUserData!SystemCallStub+0x4
01 006ffdd0 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
02 006ffe78 77d02695 kernel32!WaitForMultipleObjectsEx+0xcb
03 006ffed4 77d037f0 USER32!RealMsgWaitForMultipleObjectsEx+0x13f
04 006ffef0 75614e57 USER32!MsgWaitForMultipleObjects+0x1d
05 006fff84 77bc91ed COMSVCS!CSTAThread::WorkerLoop+0x1e3
06 006fffb8 77e4a990 msvcrt!_endthreadex+0x95
07 006fffec 00000000 kernel32!FlsSetValue+0x779




Thread ID: 66
System Thread ID: f90
Kernel Time: 0:0:39.453
User Time: 0:0:42.0
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
 # ChildEBP RetAddr
00 08f8fdcc 77f4372d SharedUserData!SystemCallStub+0x4
01 08f8fdd0 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
02 08f8fe78 77d02695 kernel32!WaitForMultipleObjectsEx+0xcb
03 08f8fed4 77d037f0 USER32!RealMsgWaitForMultipleObjectsEx+0x13f
04 08f8fef0 75614e57 USER32!MsgWaitForMultipleObjects+0x1d
05 08f8ff84 77bc91ed COMSVCS!CSTAThread::WorkerLoop+0x1e3
06 08f8ffb8 77e4a990 msvcrt!_endthreadex+0x95
07 08f8ffec 00000000 kernel32!FlsSetValue+0x779




Thread ID: 67
System Thread ID: fa0
Kernel Time: 0:0:48.718
User Time: 0:0:53.31
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
 # ChildEBP RetAddr
00 0900fdcc 77f4372d SharedUserData!SystemCallStub+0x4
01 0900fdd0 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
02 0900fe78 77d02695 kernel32!WaitForMultipleObjectsEx+0xcb
03 0900fed4 77d037f0 USER32!RealMsgWaitForMultipleObjectsEx+0x13f
04 0900fef0 75614e57 USER32!MsgWaitForMultipleObjects+0x1d
05 0900ff84 77bc91ed COMSVCS!CSTAThread::WorkerLoop+0x1e3
06 0900ffb8 77e4a990 msvcrt!_endthreadex+0x95
07 0900ffec 00000000 kernel32!FlsSetValue+0x779




Thread ID: 68
System Thread ID: 280
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
 # ChildEBP RetAddr
00 19cdfb04 77f43741 SharedUserData!SystemCallStub+0x4
01 19cdfb08 71b23ac3 ntdll!ZwWaitForSingleObject+0xc
02 19cdfb44 71b239d1 mswsock!SockWaitForSingleObject+0x19b
03 19cdfc08 71c016c9 mswsock!WSPSelect+0x229
04 19cdfc58 760f4c0d WS2_32!select+0xb9
05 19cdffb4 7610d7b0 WININET!ICAsyncThread::SelectThread+0x22a
06 19cdffb8 77e4a990 WININET!ICAsyncThread::SelectThreadWrapper+0x9
WARNING: Stack unwind information not available. Following frames may be wrong.
07 19cdffec 00000000 kernel32!FlsSetValue+0x779




Thread ID: 69
System Thread ID: fe8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
 # ChildEBP RetAddr
00 19d6feb0 77f4372d SharedUserData!SystemCallStub+0x4
01 19d6feb4 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
02 19d6ff5c 77e4b0e4 kernel32!WaitForMultipleObjectsEx+0xcb
03 19d6ffb4 7610dd34 kernel32!WaitForMultipleObjects+0x17
04 19d6ffb8 77e4a990 WININET!AutoProxyThreadFunc+0xb
05 19d6fff0 7610dd28 kernel32!FlsSetValue+0x779
06 00000000 00000000 WININET!AUTO_PROXY_DLLS::QueueAsyncAutoProxyRequest+0x1ba




Thread ID: 70
System Thread ID: e24
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
 # ChildEBP RetAddr
00 1a49fcec 77f4372d SharedUserData!SystemCallStub+0x4
01 1a49fcf0 77f75297 ntdll!NtWaitForMultipleObjects+0xc
02 1a49ffb8 77e4a990 ntdll!RtlpWaitThread+0x158
WARNING: Stack unwind information not available. Following frames may be wrong.
03 1a49ffec 00000000 kernel32!FlsSetValue+0x779




Thread ID: 71
System Thread ID: 4bc
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
 # ChildEBP RetAddr
00 1a4dfec4 77f4372d SharedUserData!SystemCallStub+0x4
01 1a4dfec8 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
02 1a4dff70 77e4b0e4 kernel32!WaitForMultipleObjectsEx+0xcb
03 00000000 00000000 kernel32!WaitForMultipleObjects+0x17




Thread ID: 72
System Thread ID: c00
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
 # ChildEBP RetAddr
00 1a1bff10 77f43741 SharedUserData!SystemCallStub+0x4
01 1a1bff14 77e41817 ntdll!ZwWaitForSingleObject+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
02 1a1bff84 77e4168f kernel32!WaitForSingleObjectEx+0x88
03 1a1bffb8 77e4a990 kernel32!WaitForSingleObject+0xf
04 1a1bffec 00000000 kernel32!FlsSetValue+0x779




Thread ID: 73
System Thread ID: bc8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
 # ChildEBP RetAddr
00 1a1fff08 77f43741 SharedUserData!SystemCallStub+0x4
01 1a1fff0c 77e41817 ntdll!ZwWaitForSingleObject+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
02 1a1fff7c 77e4168f kernel32!WaitForSingleObjectEx+0x88
03 1a1fffb8 77e4a990 kernel32!WaitForSingleObject+0xf
04 1a1fffec 00000000 kernel32!FlsSetValue+0x779




Thread ID: 74
System Thread ID: e48
Kernel Time: 0:0:0.93
User Time: 0:0:0.78
Thread Type: Possible ASP page.  Possible DCOM activity
Executing Page: Unable to locate ASP page

OLE32.dll Symbols not found. Unable to proceed with DCOM check.
Continuing other analysis.

 # ChildEBP RetAddr
00 00c0fe20 77f4313f SharedUserData!SystemCallStub+0x4
01 00c0fe24 77c52555 ntdll!NtReplyWaitReceivePortEx+0xc
02 00c0ff8c 77c65159 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x193
03 00c0ff90 77c650a1 RPCRT4!RecvLotsaCallsWrapper+0x9
04 00c0ffb0 77c65187 RPCRT4!BaseCachedThreadRoutine+0x9c
05 00c0ffb8 77e4a990 RPCRT4!ThreadStartRoutine+0x17
WARNING: Stack unwind information not available. Following frames may be wrong.
06 00c0ffec 00000000 kernel32!FlsSetValue+0x779




Thread ID: 75
System Thread ID: b44
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
 # ChildEBP RetAddr
00 0090ff10 77f4262b SharedUserData!SystemCallStub+0x4
01 0090ff14 77e418ea ntdll!NtDelayExecution+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
02 0090ff7c 77e416ee kernel32!SleepEx+0x51
03 00000000 00000000 kernel32!Sleep+0xb




Thread ID: 76
System Thread ID: bb8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
 # ChildEBP RetAddr
00 00ccfea4 77f43741 SharedUserData!SystemCallStub+0x4
01 00ccfea8 77e41817 ntdll!ZwWaitForSingleObject+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
02 00ccff18 77e4168f kernel32!WaitForSingleObjectEx+0x88
03 00000000 00000000 kernel32!WaitForSingleObject+0xf




Thread ID: 77
System Thread ID: f48
Kernel Time: 0:0:0.0
User Time: 0:0:0.15
Thread Status: Thread is in a WAIT state.
Thread Type: Other
 # ChildEBP RetAddr
00 00fefea4 77f43741 SharedUserData!SystemCallStub+0x4
01 00fefea8 77e41817 ntdll!ZwWaitForSingleObject+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
02 00feff18 77e4168f kernel32!WaitForSingleObjectEx+0x88
03 00000000 00000000 kernel32!WaitForSingleObject+0xf




Thread ID: 78
System Thread ID: 534
Kernel Time: 0:0:0.0
User Time: 0:0:0.15
Thread Status: Thread is in a WAIT state.
Thread Type: Other
 # ChildEBP RetAddr
00 19ebfea4 77f43741 SharedUserData!SystemCallStub+0x4
01 19ebfea8 77e41817 ntdll!ZwWaitForSingleObject+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
02 19ebff18 77e4168f kernel32!WaitForSingleObjectEx+0x88
03 00000000 00000000 kernel32!WaitForSingleObject+0xf

*****

Dump name is formatted as: PID-Timestamp.dmp

Creating C:\Program Files\IIS Resources\IISState\output\1180-1114212582.dmp - us
er full dump

*****

Closing open log file C:\Program Files\IIS Resources\IISState\output\IISState.lo
g

Finished
C:\Program Files\IIS Resources\IISState>
0
Comment
Question by:tkrepr
  • 6
  • 4
  • 2
  • +1
13 Comments
 
LVL 6

Expert Comment

by:masirof
ID: 13849200
Hi,

Close db connections and destroy objects after page execution.

Apply this hotfix, restart IIS. It may help.
http://support.microsoft.com/default.aspx?scid=kb;en-us;838306
0
 

Author Comment

by:tkrepr
ID: 13849442
Hi. Originally, I was having terrible hangs which were unrecoverable and would require a restart of IIS.  It helped considerably by setting IIS in the 5.0 compatibility mode, but they still occurred.  About a week ago, I did obtain the hotfix from Microsoft and it fixed the unrecoverable hang problem.  However, I still have this particular problem in which everything seems to stall for periods of time, but it does recover on its own.  I see these threads in wait state, but I just don't understand what the IISSTATE log is telling me.

Terry
0
 
LVL 6

Expert Comment

by:masirof
ID: 13853765
Did you apply the hotfix provided above?
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 

Author Comment

by:tkrepr
ID: 13853979
I had applied the hotfix about a week ago and it fixed problem where hang was unrecoverable.  Before that, I would have to recycle IIS.  The most recent hangs, one of which I have posted the IISSTATE log for, last for 15 seconds to a minute, but
eventually, the ram and CPU consumption come back down and it seems to recover.
0
 
LVL 34

Expert Comment

by:Dave_Dietz
ID: 13854270
The threads in a wait state aren't the problem.  They are simply threads that have nothing to do and are waiting to be used.

This thread looks questionable:
Thread ID: 75
System Thread ID: b44
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
 # ChildEBP RetAddr
00 0090ff10 77f4262b SharedUserData!SystemCallStub+0x4
01 0090ff14 77e418ea ntdll!NtDelayExecution+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
02 0090ff7c 77e416ee kernel32!SleepEx+0x51
03 00000000 00000000 kernel32!Sleep+0xb

We shouldn't be sleeping.  This is where a thread stops for a period of time (doing nothing) and then continues.

It is possible that the sleeping thread is holding a critical setion that other threads are waiting on, causing them to spin waiting for the lock to be released.

I am also curious what the first 63 threads are doing since they're not listed here.

I'd be happy to take a crak at the dump if you'd care to make it available.....  :-)

Dave Dietz

0
 

Author Comment

by:tkrepr
ID: 13854306
Hi Dave,

When I copied the log, I did it with MARK/COPY/PASTE inside the command prompt.  I bet the other threads had already scrolled out of the buffer.  Sorry about that.  I have posted the entire log file and associated dump file here:

http://208.180.177.128/dump/dump_20050422.zip
It's about 24 meg.  If you just want the log file, download this:
http://208.180.177.128/dump/dump_20050422_log.zip (only 15k)

Thanks,

Terry Roller
0
 
LVL 34

Expert Comment

by:Dave_Dietz
ID: 13854490
Hate to break it to you but the fix for 838306 has *not* been installed on this machine, or it has been overwritten by something else.

The version of MSJET40 in the dump is 4.0.8618.0 and is dated 18-Feb-2004:

0:017> lmvm msjet40
start    end        module name
1b000000 1b170000   msjet40    (private pdb symbols)  c:\symsrv\msjet40.pdb\4035303E1\msjet40.pdb
    Loaded symbol image file: msjet40.dll
    Mapped memory image file: c:\symsrv\msjet40.dll\403430AC170000\msjet40.dll
    Image path: C:\WINDOWS\system32\msjet40.dll
    Image name: msjet40.dll
    Timestamp:        Wed Feb 18 22:42:36 2004 (403430AC)
    CheckSum:         0017605C
    ImageSize:        00170000
    File version:     4.0.8618.0
    Product version:  4.0.8618.0
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0000.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft (R) Jet
    InternalName:     MSJET40.DLL
    OriginalFilename: MSJET40.DLL
    ProductVersion:   4.00.8618.0
    FileVersion:      4.00.8618.0
    FileDescription:  Microsoft Jet Engine Library
    LegalCopyright:   Copyright (C) Microsoft Corp. 1993-1999

The version in the Hotfix is 4.0.8704.0 and is dated 17-May-2004.

Reinstall the fix and verify the version of the C:\WINDOWS\system32\msjet40.dll after installation.

The dump shows the classic issues associated with this bug.

Dave Dietz
0
 

Author Comment

by:tkrepr
ID: 13855156
Dave,

Darn it!  I thought you were on to something and got my hopes up.  The DLL in question is MSJTES40.DLL and the one I have is the 4.0.8704.0 version dated 17-May-2004.

Terry
0
 
LVL 34

Expert Comment

by:Dave_Dietz
ID: 13855917
Hmmm, the version of the DLL in the dump is showing the older version.

Try doing a search on msjet40 on your machine and see if there is more than one copy that shows up.

If there are multiple copies it may be necessary to register the correct version with the following command:

   regsvr32 c:\windows\system32\msjet40.dll

I assume the dump was collected *after* installing the fix?  Also, silly but important question, was IIS restarted after applying the fix?

Dave Dietz
0
 

Author Comment

by:tkrepr
ID: 13855963
Hi Dave,

I think you're confusing two different files.
MSJET40.DLL is different than MSJTES40.DLL.  The hotfix only mentions MSJTES40.DLL, which I have the correct copy of.

Regarding cycling IIS...yes.  The hotfix forced me to reboot.

Thanks,

Terry
0
 
LVL 34

Expert Comment

by:Dave_Dietz
ID: 13856279
Oops!  I did make a mistake on that.

I'll take another look at it tomorrow and see if I can pull anything else out of the dump.....

Dave Dietz
0
 
LVL 2

Accepted Solution

by:
KenSchaefer earned 2000 total points
ID: 13907136
Hi Terry,

Was this the same IISState log that was posted to the IIS public newsgroups? I looks kinda familiar but my memory isn't that great. I haven't downloaded the dump file (it's coming down a bit slow at the moment), but it seems that you have a deadlock of some kind. Both threads 16 and 19 (with IDs 154 and 184)  are both waiting to enter critical sections that the other thread is in. One appears to be processing code in your global.asa file that involves a VB component. What is that component doing?

Cheers
Ken
0
 

Author Comment

by:tkrepr
ID: 13909322
Oops. I didn't do that right. I should have used the "Split Points" instead of Accept.  Dave, if you want your split, could you contact the moderator? (-;  Sorry about that.
0

Featured Post

NEW Veeam Backup for Microsoft Office 365 1.5

With Office 365, it’s your data and your responsibility to protect it. NEW Veeam Backup for Microsoft Office 365 eliminates the risk of losing access to your Office 365 data.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

First of all, clustering IIS is something you should rarely consider doing. In almost all cases, Microsoft Network Load Balancing (NLB) (http://technet.microsoft.com/en-us/library/cc758834(WS.10).aspx) is a much better solution when you need to p…
Debug Tools to analyse IIS process: This article focus on taking memory dumps from IIS to determine which code is taking more time and to analyse which calls hangs/causes more CPU usage. To take dumps,download the following. Install1: To st…
Please read the paragraph below before following the instructions in the video — there are important caveats in the paragraph that I did not mention in the video. If your PaperPort 12 or PaperPort 14 is failing to start, or crashing, or hanging, …
As many of you are aware about Scanpst.exe utility which is owned by Microsoft itself to repair inaccessible or damaged PST files, but the question is do you really think Scanpst.exe is capable to repair all sorts of PST related corruption issues?
Suggested Courses

810 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question