Link to home
Start Free TrialLog in
Avatar of johnb6767
johnb6767Flag for United States of America

asked on

Poolmon Usage? Need a poolmon expert.....

Havent used Poolmon a heck of a lot, but in troubleshooting a BSOD, bugcheck 0xF4, where csrss.exe is terminating and crashing, I am finding a problem with interpreting the Tags in poolmon. There is supposed to be a localtags.txt file that when used with the appropriate switch, should add a Mapped Driver view in the app. Only problem is, using poolmon, I cannot get any of the documented switches to work.

poolmon /c
poolmon /g

Tey are unknown switches. I have enclosed a screenshot of poolmon, sorted with paged and non paged pool entries, sorted by most bytes.......

If anyone has, I guess you could say, "general" localflags.txt for comm apps (mainly I need Symantec, and everything else in the screenshot that has high values), that might allow me to move further in my research.......

Any questions, please ask......
poolmon.jpg
Avatar of Qlemo
Qlemo
Flag of Germany image

Did you try the brute force method:
   findstr /s /m /L "SavE xns2 MmCm" %SystemRoot%\*.sys
? It will find the driver files containing the top 3 tags.

The /c and /g switches do work with the W2003 Support Tools only. They are located on the W2003 setup CD.
Avatar of johnb6767

ASKER

Ok quick update.

I am aware of the findstr, ust didnt try it with these switches. I also thought I had the most recent poolmon. I should have known better to verify.....

OK. Finding some more info on the Tags, and trying to match them now.

C:\>poolmon /c
Poolmon: No localtag.txt in current directory
Poolmon: Unable to load required dlls, cannot create local tag file

Had to get msdis160.dll from another PC for the above command to work (after creating a localtag.txt file).

Set the Buffer Size properly to see the new column.....

 Memory: 2074820K Avail: 1506032K  PageFlts:   329   InRam Krnl: 3652K P:116424K
 Commit: 437352K Limit:3992012K Peak: 555416K            Pool N:119320K P:116656K
 System pool information
 Tag  Type     Allocs            Frees            Diff   Bytes      Per Alloc  Mapped_Driver

 SavE Paged   1155886 (   0)   1155251 (   0)      635 73749816 (     0) 116141 Unknown Driver
 MmSt Paged      7547 (   2)      2347 (   2)     5200 9400400 (     0)   1807 Unknown Driver
 R600 Paged       253 (   0)        47 (   0)      206 5694440 (     0)  27642 Unknown Driver
 Gh05 Paged      7922 (   3)      7424 (   3)      498 4528120 (     0)   9092 Unknown Driver
 UlHT Paged         1 (   0)         0 (   0)        1 4198400 (     0) 4198400 Unknown Driver
 LDfb Paged         3 (   0)         0 (   0)        3 3530976 (     0) 1176992 Unknown Driver
 Ntff Paged      2954 (   0)       155 (   0)     2799 2328768 (     0)    832 Unknown Driver
 NtfF Paged      4027 (   0)      2802 (   0)     1225 1156400 (     0)    944 Unknown Driver
 SACC Paged       250 (   0)         0 (   0)      250 1008968 (     0)   4035 Unknown Driver
 DATI Paged       170 (   0)       106 (   0)       64  899184 (     0)  14049 Unknown Driver
 Toke Paged     37174 (   2)     37008 (   2)      166  730744 (     0)   4402 Unknown Driver
 Ttfd Paged      1827 (   0)      1232 (   0)      595  602816 (     0)   1013 Unknown Driver
 PTr3 Paged      5072 (   0)      3056 (   0)     2016  596736 (     0)    296 Unknown Driver
 IoNm Paged    425194 (  21)    421377 (  21)     3817  561968 (     0)    147 Unknown Driver
 CM35 Paged        32 (   0)         6 (   0)       26  466944 (     0)  17959 Unknown Driver
 CM16 Paged        80 (   0)         3 (   0)       77  446464 (     0)   5798 Unknown Driver
 FSim Paged      3029 (   0)         0 (   0)     3029  387712 (     0)    128 Unknown Driver
 SAV  Paged    287063 (   6)    286519 (   6)      544  320472 (     0)    589 Unknown Driver
 Obtb Paged       295 (   0)       169 (   0)      126  307264 (     0)   2438 Unknown Driver
 NtFs Paged     16665 (   0)     11895 (   0)     4770  307160 (     0)     64 Unknown Driver
 Gla1 Paged       289 (   0)       100 (   0)      189  302400 (     0)   1600 Unknown Driver
 CMAl Paged      2672 (   0)      2606 (   0)       66  270336 (     0)   4096 Unknown Driver
 MmSm Paged      3655 (   0)       248 (   0)     3407  218048 (     0)     64 Unknown Driver
 CMDa Paged    147372 (   0)    145920 (   0)     1452  199176 (     0)    137 Unknown Driver
 CM25 Paged      1110 (   0)      1098 (   0)       12  196608 (     0)  16384 Unknown Driver
 CMVa Paged    518229 (   0)    514211 (   0)     4018  192448 (     0)     47 Unknown Driver
 Gcac Paged       111 (   0)        83 (   0)       28  152936 (     0)   5462 Unknown Driver
 CM39 Paged       600 (   0)        72 (   0)      528  151680 (     0)    287 Unknown Driver
 Gla5 Paged       636 (   0)       250 (   0)      386  151312 (     0)    392 Unknown Driver
 Gdrs Paged      1277 (   0)      1253 (   0)       24  146160 (     0)   6090 Unknown Driver
 Ntfo Paged     10126 (   0)      8906 (   0)     1220  140368 (     0)    115 Unknown Driver
 WmIS Paged         1 (   0)         0 (   0)        1  135168 (     0) 135168 Unknown Driver
 Ntfc Paged      2542 (   0)       713 (   0)     1829  131688 (     0)     72 Unknown Driver
 Key  Paged    810623 ( 488)    809358 ( 488)     1265  131520 (     0)    103 Unknown Driver
 NtFf Paged        76 (   0)        73 (   0)        3  131144 (     0)  43714 Unknown Driver

So I am guessing I need to run a command to generate the localtag.txt file?




Poolmon /c should create the file itself.

C:\>poolmon /c
Poolmon: No localtag.txt in current directory
Poolmon: Unable to load required dlls, cannot create local tag file

I created the file earlier before posting the last extract with teh Unknown Drivers, in hopes it would POPULATE it.....

I went through and found a few of them already by searching individually.



SavE - c:\Program Files\Symantec AntiVirus\savrt.sys
MmSt - 
R600 - c:\WINDOWS\system32\drivers\ati2mtag.sys
Gh05 - 
U1Ht - 
LDfb - 
NtfF - c:\WINDOWS\system32\drivers\ntfs.sys
SACC - c:\Program Files\Symantec AntiVirus\savrt.sys
DATI - c:\WINDOWS\system32\drivers\ATSwpWDF.sys
       c:\WINDOWS\system32\drivers\mf.sys
       c:\WINDOWS\system32\drivers\pcmcia.sys

Open in new window

So the top issue seems to be caused by SAV - surprise, surprise!
Well, we always thought it was... But just because it is using the most Paged Pool memory, doesnt mean it is the one corrupting the pool.....

Shouldnt I be concerned with the Diff values, from 1 capture to another? 1 hour went by, and SavE stayed at 635, where as several others had HUGE diffs.... Just from sitting here idle....

I am also playing with the Driver Verifier, and am solely focusing my next boot on Savrt.sys.

I will get some more screenshots momentarily. If you can assist with anything else that will help me identify these drivers by tags, I think that will be extremely beneficial.

Also, What else can I use from Poolmon to help determine whats corrupting the pool?

Oh, and btw, in case you were wondering, I did use gflags.exe to enable Pool Tagging......
FYI....

If you have the gflags setting turned on for pool tags and if you use the Poolmon utility, you see a higher usage of the MmSt tag. This is the pool tag that is used to map the operating system memory that is used to track shared files.

from

Backup program is unsuccessful when you back up a large system volume
http://support.microsoft.com/kb/304101

I know to watch for increases in Bytes, that could indicate a problem.....
This is the realtime stats on Savrt.sys

No IRQL raises
No failed Pool Allocations
No allocations without tag

This has been up 1 hr. 4miutes and 33 seocnds..... This was opened within 5 minutes of login....

Going to start widening the search on Driver Verifier.
Savrt.sys-Verifier-Stats.jpg
Another update.....

Tag Mmst has almost doubled insize, which will almost lead me towards the server service.

Diffs from 3146 to 4903

Any more thoughts? Keep in mind, this machine is just sitting on my desk, staring at me like I am stupid.....   :^)



OK, so I extracted the XP SP2 symbols, and once I did I started seeing MmSt, and Ntff increase diffs, and bytes used. Ntff actually quadrupled in size, diffs almost quadrupled as well. Once the copy was done, shouldnt it have released some of this Paged Pool memory?
Summarizing all things said and shown, I assume the paged or non-paged pool hits the soft limits stored in registry, or even the hard limits of kernel. CSRSS tries to allocate more of pool memory, can't get any, and crashes.
 
 Alas, that does not help. MmSt is the culprit, you think? File Cache Memory Manager, this belongs to. This can result when using AV software, so this is not completely out of the line of fire. That using symbols adds to the pool usage is not that strange, however, it should not be that much.

On a analysis for a similar bugcheck on a client's system, I remember I had to use live kernel debugger as touched in http://blogs.msdn.com/ntdebugging/archive/2006/12/18/Understanding-Pool-Consumption-and-Event-ID_3A00_--2020-or-2019.aspx.
Scanning the pool memory assigned to the tag, I saw file names for log files, which were not freed by the network layer (a novell driver).

Sorry I cannot go into more detail, it's too long ago. And having found the culprit after weeks (!) ended up in more drinks than necessary ;-)
If you like, I will have a look into Windows Internals 4th Ed. in Office tomorrow - I think it had some info helpful for live debugging.

"Summarizing all things said and shown, I assume the paged or non-paged pool hits the soft limits stored in registry, or even the hard limits of kernel. CSRSS tries to allocate more of pool memory, can't get any, and crashes."

Exactly. It is a 0xC4 (going off memory).

" That using symbols adds to the pool usage is not that strange, however, it should not be that much."

This was just an example of activity that I did at the time to watch the pool values climb. I needed them as Process Explorer wouldnt use the Symbol Server. At the time, I was far from limits. *sigh*

I am fairly dangerous in the debugger, and I can attach to the  Kernel, but doing live debugging I am not sure I know where to start.....
Am I correct in understanding that the two main things to look at, are the sizes of the Pooled/Non Paged Pool values, as well as thier diffs are the two most important things there?



You will have to look for
 the absolute sizes -  whether the limits are about to be hit
diff values of alloc/free/bytes, those in brackets - to see big allocators per interval

The thing to hunt for is whatever tag has an constant high rate of diff bytes per interval. It is helpful to export the output once a day, and compare the results, to get the "big figure".


Yea, I am a good way from the P/NP limits.

I think I am going to continue on the Verifier route for now, to see if I can get some failures. I did last week, but no dump was left on the 0xC5 bugcheck, like it was supposed to.....

Thanks for the info so far. If you can think of anything useful in the Live Kernel Debug, I would be more than happy to play around some with it......
First, you need a hint which tag or even driver is the culprit. Driver Verifier will (hopefully) guide you on that track. It should reveal the stuff stored in that pool memory, but I'm not certain about it. Live kernel debugging does make sense only when you have found leaks, which you can observe than, and before crash, of course.
On the other hand, kd or windbg is useful on the (kernel) crash dump of BSOD.

My suggestion is indeed to follow the Verifier thread first, maybe mixed with poolmon snapshots.
CRITICAL_OBJECT_TERMINATION (f4)
A process or thread crucial to system operation has unexpectedly exited or been
terminated.
Several processes and threads are necessary for the operation of the
system; when they are terminated (for any reason), the system can no
longer function.
Arguments:
Arg1: 00000003, Process
Arg2: 864ee650, Terminating object
Arg3: 864ee7c4, Process image file name
Arg4: 805d1160, Explanatory message (ascii)

This is why I started looking at poolmon to see what was happening......



0: kd> !vm
 
*** Virtual Memory Usage ***
GetUlongFromAddress: unable to read from 80561108
	Physical Memory:           0 (         0 Kb)
GetUlongFromAddress: unable to read from 80560c40
 
************ NO PAGING FILE *********************
 
80560b60: Unable to get paged pool info
GetUlongPtrFromAddress: unable to read from 80550990
GetUlongPtrFromAddress: unable to read from 80560f2c
GetPointerFromAddress: unable to read from 80560c04
GetPointerFromAddress: unable to read from 80554c48
GetUlongFromAddress: unable to read from 8055c780
GetPointerFromAddress: unable to read from 8055c6f4
GetUlongFromAddress: unable to read from 8055c4d4
GetUlongFromAddress: unable to read from 80550918
GetUlongFromAddress: unable to read from 80550928
GetUlongFromAddress: unable to read from 805610fc
GetUlongFromAddress: unable to read from 805610bc
GetUlongFromAddress: unable to read from 8055c330
GetUlongFromAddress: unable to read from 8055c180
GetUlongFromAddress: unable to read from 8055c17c
GetUlongFromAddress: unable to read from 8055c184
GetUlongFromAddress: unable to read from 8055c180
GetUlongFromAddress: unable to read from 8055c17c
GetUlongFromAddress: unable to read from 8055c3dc
GetUlongPtrFromAddress: unable to read from 80553280
GetUlongPtrFromAddress: unable to read from 80554cc0
GetUlongFromAddress: unable to read from 8055c398
GetUlongFromAddress: unable to read from 8055c380
	Error reading free nonpaged PTEs 8055c334
GetUlongFromAddress: unable to read from 8055c390
	Available Pages:           0 (         0 Kb)
	ResAvail Pages:            0 (         0 Kb)
 
	********** Running out of physical memory **********
 
	Locked IO Pages:           0 (         0 Kb)
	Free System PTEs:          0 (         0 Kb)
 
	********** Running out of system PTEs **************
 
GetUlongFromAddress: unable to read from 8055c318
GetUlongFromAddress: unable to read from 8055c530
	Free NP PTEs:              0 (         0 Kb)
	Free Special NP:           0 (         0 Kb)
	Modified Pages:            0 (         0 Kb)
	Modified PF Pages:         0 (         0 Kb)
80563c20: Unable to get pool descriptor
GetUlongFromAddress: unable to read from 805512b8
	NonPagedPool Usage:        0 (         0 Kb)
	NonPagedPool Max:          0 (         0 Kb)
GetUlongFromAddress: unable to read from 805512b4
	PagedPool Usage:           0 (         0 Kb)
	PagedPool Maximum:         0 (         0 Kb)
GetUlongFromAddress: unable to read from 80564c48
	Shared Commit:             0 (         0 Kb)
	Special Pool:              0 (         0 Kb)
	Shared Process:            0 (         0 Kb)
	PagedPool Commit:          0 (         0 Kb)
	Driver Commit:             0 (         0 Kb)
	Committed pages:      118503 (    474012 Kb)
	Commit limit:              0 (         0 Kb)
 
	********** Number of committed pages is near limit ********
GetUlongFromAddress: unable to read from 8055c3f8
GetUlongFromAddress: unable to read from 8055c3fc
 
Unable to read/NULL value _LIST_ENTRY @ 805627b8
 
ProcessCommitUsage could not be calculated

Open in new window

I would still like to find out what these three tags represent......

Gh05
U1Ht
LDfb

:^)
Extract of Debugging Tools for Windows\triage\pooltag.txt:

Gh?5 - win32k.sys                           - GDITAG_HMGR_SURF_TYPE

The other two are not known ...

To the crash dump: Did you !anaylze -v, as suggested by windbg?
Do you have a full dump (does not look like, no VM info)?
Otherwise, the !vm output is not helpful.


No, just memory dumps. Didnt know that about the !vm command actually. Analyze -v, shows the process is csrss.exe. I can post one on Monday if you would like to work with it....... Like I said, I am about half dangerous when it comes to Windbg. I have learned alot about it, but without programming skills, and understanding alot of the commands, it make sit tough to really dig......
ASKER CERTIFIED SOLUTION
Avatar of Qlemo
Qlemo
Flag of Germany image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
I havent been able to get another crash. I am going to close this out, as you helped me with the original question, regarding poolmon. After learning more about it, it will prove to be a great tool to use in the future.

Thanks!!