ThaneVcr
asked on
Analysis of Memory dump debug output
Hello, Can anyone offer me insight based on this !analyze bug check finding. I will be happy to share the WEW debug workspace if anyone thinks it will help.
This is Q-Xeon 2.26 12GB white box. Running 2008 64Bit It is DC, Hyper-V and SQL (honestly I didnt do it.) Nothing serious going on in hyper space, rather a 2003 and 2000 running with no load.
In the past 7 days it has had two crashes and rebooted. Different times\days. No application or system event action logged, coinciding before the crashes, Everything comes back normally. No device error messages.
No changes have been made since routine updates late July. No SQL management plan running. No backup action going on. But it did leave behind a dmp file that may offer clues.
"DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT" this is what has me wondering? A incompatible\uncertified driver? If so how do I find out which? Before I got the debug done I was leaning to expect hardware?
The watchdog timeout?
I surely appreciate any assistance.
Thane C
0: kd> !analyze -v
**************************
* *
* Bugcheck Analysis *
* *
**************************
CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
Arguments:
Arg1: 0000000000000019, Clock interrupt time out interval in nominal clock ticks.
Arg2: 0000000000000000, 0.
Arg3: fffff88001e5d180, The PRCB address of the hung processor.
Arg4: 0000000000000002, 0.
Debugging Details:
------------------
BUGCHECK_STR: CLOCK_WATCHDOG_TIMEOUT_8_P
DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT
PROCESS_NAME: System
CURRENT_IRQL: d
STACK_TEXT:
fffff800`0151aa28 fffff800`01685443 : 00000000`00000101 00000000`00000019 00000000`00000000 fffff880`01e5d180 : nt!KeBugCheckEx
fffff800`0151aa30 fffff800`016dfcf7 : 00000000`00000000 fffff800`00000002 00000000`00002710 fffffa80`0a6ff278 : nt! ?? ::FNODOBFM::`string'+0x4df
fffff800`0151aac0 fffff800`01628895 : fffff800`0164d460 fffff800`0151ac70 fffff800`0164d460 fffff800`00000000 : nt!KeUpdateSystemTime+0x37
fffff800`0151abc0 fffff800`016d3af3 : 00000000`00000000 fffff880`03392328 fffff800`01855300 00000000`00000004 : hal!HalpHpetClockInterrupt
fffff800`0151abf0 fffff800`016dfdd2 : fffff800`01851e80 fffff800`00000002 00000000`00000002 fffff880`00000000 : nt!KiInterruptDispatchNoLo
fffff800`0151ad80 00000000`00000000 : fffff800`0151b000 fffff800`01515000 fffff800`0151ad40 00000000`00000000 : nt!KiIdleLoop+0x32
STACK_COMMAND: kb
SYMBOL_NAME: ANALYSIS_INCONCLUSIVE
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: Unknown_Module
IMAGE_NAME: Unknown_Image
DEBUG_FLR_IMAGE_TIMESTAMP:
FAILURE_BUCKET_ID: X64_CLOCK_WATCHDOG_TIMEOUT
BUCKET_ID: X64_CLOCK_WATCHDOG_TIMEOUT
This is Q-Xeon 2.26 12GB white box. Running 2008 64Bit It is DC, Hyper-V and SQL (honestly I didnt do it.) Nothing serious going on in hyper space, rather a 2003 and 2000 running with no load.
In the past 7 days it has had two crashes and rebooted. Different times\days. No application or system event action logged, coinciding before the crashes, Everything comes back normally. No device error messages.
No changes have been made since routine updates late July. No SQL management plan running. No backup action going on. But it did leave behind a dmp file that may offer clues.
"DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT" this is what has me wondering? A incompatible\uncertified driver? If so how do I find out which? Before I got the debug done I was leaning to expect hardware?
The watchdog timeout?
I surely appreciate any assistance.
Thane C
0: kd> !analyze -v
**************************
* *
* Bugcheck Analysis *
* *
**************************
CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
Arguments:
Arg1: 0000000000000019, Clock interrupt time out interval in nominal clock ticks.
Arg2: 0000000000000000, 0.
Arg3: fffff88001e5d180, The PRCB address of the hung processor.
Arg4: 0000000000000002, 0.
Debugging Details:
------------------
BUGCHECK_STR: CLOCK_WATCHDOG_TIMEOUT_8_P
DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT
PROCESS_NAME: System
CURRENT_IRQL: d
STACK_TEXT:
fffff800`0151aa28 fffff800`01685443 : 00000000`00000101 00000000`00000019 00000000`00000000 fffff880`01e5d180 : nt!KeBugCheckEx
fffff800`0151aa30 fffff800`016dfcf7 : 00000000`00000000 fffff800`00000002 00000000`00002710 fffffa80`0a6ff278 : nt! ?? ::FNODOBFM::`string'+0x4df
fffff800`0151aac0 fffff800`01628895 : fffff800`0164d460 fffff800`0151ac70 fffff800`0164d460 fffff800`00000000 : nt!KeUpdateSystemTime+0x37
fffff800`0151abc0 fffff800`016d3af3 : 00000000`00000000 fffff880`03392328 fffff800`01855300 00000000`00000004 : hal!HalpHpetClockInterrupt
fffff800`0151abf0 fffff800`016dfdd2 : fffff800`01851e80 fffff800`00000002 00000000`00000002 fffff880`00000000 : nt!KiInterruptDispatchNoLo
fffff800`0151ad80 00000000`00000000 : fffff800`0151b000 fffff800`01515000 fffff800`0151ad40 00000000`00000000 : nt!KiIdleLoop+0x32
STACK_COMMAND: kb
SYMBOL_NAME: ANALYSIS_INCONCLUSIVE
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: Unknown_Module
IMAGE_NAME: Unknown_Image
DEBUG_FLR_IMAGE_TIMESTAMP:
FAILURE_BUCKET_ID: X64_CLOCK_WATCHDOG_TIMEOUT
BUCKET_ID: X64_CLOCK_WATCHDOG_TIMEOUT
ASKER
Yes it was Windows 7 professional Debugger
WinDbg 6.11.0001.404 x86 Build 7600
Symbols path set
SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
WinDbg 6.11.0001.404 x86 Build 7600
Symbols path set
SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
it looks like one of your processors does not work :
>>>
CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
>>>
CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
ASKER
nobus - OK so that is the direction you would think. It certainly behaves like a hardware problem.
What if there was an uncertified driver that somehow blocked the interrupt
Is "DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT" anything to run down?
This server doesnt look unhappy in device manager, not a lot of device drivers that look questionable.
What if there was an uncertified driver that somehow blocked the interrupt
Is "DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT" anything to run down?
This server doesnt look unhappy in device manager, not a lot of device drivers that look questionable.
ASKER
justinyeung: _ failed to mention this is debug from an actual 500 MB kernel dump not a small memory dump. So therefore it got created c:\windows\memory.dmp as per default path.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
jcgriff - Thanks for the direction. Please find a txt output of the debug per your commands. Also the nfo you requested.
I will post the findings of the driver verifier pronto. I had to append an acceptable extension to the nfo remove it and you should have what you requested.
Thanks for the help.
srv002-debug.txt
srv002-msinfo.nfo.txt
I will post the findings of the driver verifier pronto. I had to append an acceptable extension to the nfo remove it and you should have what you requested.
Thanks for the help.
srv002-debug.txt
srv002-msinfo.nfo.txt
ASKER
jcgriff - sorry read post ..zip it you asked...my apology. I will try and follow instructions better.
While I am reading about how to run DV, please confirm this says it can BSOD. As this is a production server without remote lights out is this likely to hang needing a power off if it finds a non-compatible driver.
I can deal with a reset as long as it comes back live. It is set to Auto -restart on the startup and recovery options.
T
While I am reading about how to run DV, please confirm this says it can BSOD. As this is a production server without remote lights out is this likely to hang needing a power off if it finds a non-compatible driver.
I can deal with a reset as long as it comes back live. It is set to Auto -restart on the startup and recovery options.
T
if it points to the cpu - don't ask "what if" - test with another one ..
ASKER
ASKER
Please see this KB.
http://support.microsoft.com/kb/975530
So far it looks good. It is that processor, that role (hyper-v) and it doesnt currently have this fix. It has only one of the referenced HV* files and it is older than what is required.
I have checked it is not on the list of updates waiting either.
T
http://support.microsoft.com/kb/975530
So far it looks good. It is that processor, that role (hyper-v) and it doesnt currently have this fix. It has only one of the referenced HV* files and it is older than what is required.
I have checked it is not on the list of updates waiting either.
T
ASKER
I have gone ahead and installed KB975530. Things are fine, we will observe for a a couple of weeks and see if anything changes. So I am suggesting keeping the question open.
Thanks for your help. Very grateful.
Thanks for your help. Very grateful.
Hi -
Nice find on the kb.
I would advise updating your Intel(R) 82574L Gigabit Network Connection driver -
e1q60x64.sys Fri Apr 03 16:03:40 2009 (49D695CC)
http://downloadcenter.inte
Thank you for the files.
The only other item I noticed was Symantec Backup. There are 2 DLL's - 2006 & 2007 that are older than the others. This may be nothing; just wanted to note it.\
Regards. . .
jcgriff2
Nice find on the kb.
I would advise updating your Intel(R) 82574L Gigabit Network Connection driver -
e1q60x64.sys Fri Apr 03 16:03:40 2009 (49D695CC)
http://downloadcenter.inte
Thank you for the files.
The only other item I noticed was Symantec Backup. There are 2 DLL's - 2006 & 2007 that are older than the others. This may be nothing; just wanted to note it.\
Regards. . .
jcgriff2
vxcrypto 3.1.1.6 1.25 MB (1,314,360 bytes) 06/09/2006 10:32 PM Redistributed by Symantec c:\program files\symantec\backup exec\raws\vxcrypto.dll
vxace_3i 3.1.1.29 1.87 MB (1,955,640 bytes) 26/11/2007 4:27 AM Redistributed by Symantec c:\program files\symantec\backup exec\raws\vxace_3i.dll
beremote 12.5.2213.158 1.68 MB (1,760,584 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\beremote.exe
bestdutl 12.5.2213.103 162.32 KB (166,216 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bestdutl.dll
becrypto 12.5.2213.0 51.32 KB (52,552 bytes) 08/09/2008 11:12 AM Symantec Corporation c:\program files\symantec\backup exec\raws\becrypto.dll
beclass 12.5.2213.154 936.32 KB (958,792 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\beclass.dll
serdll 12.5.2213.133 224.82 KB (230,216 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\serdll.dll
besocket 12.5.2213.0 267.32 KB (273,736 bytes) 08/09/2008 11:12 AM Symantec Corporation c:\program files\symantec\backup exec\raws\besocket.dll
becluster 12.5.2213.146 478.32 KB (489,800 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\becluster.dll
bevs 12.5.2213.0 278.82 KB (285,512 bytes) 08/09/2008 11:12 AM Symantec Corporation c:\program files\symantec\backup exec\raws\bevs.dll
benetapi 12.5.2213.0 24.82 KB (25,416 bytes) 08/09/2008 11:13 AM Symantec Corporation c:\program files\symantec\backup exec\raws\benetapi.dll
benetutl 12.5.2213.0 24.32 KB (24,904 bytes) 08/09/2008 11:13 AM Symantec Corporation c:\program files\symantec\backup exec\raws\benetutl.dll
ndmpcomm 12.5.2213.111 448.32 KB (459,080 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\ndmpcomm.dll
bedscomn 12.5.2213.154 317.82 KB (325,448 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedscomn.dll
bemsdk 12.5.2213.138 3.06 MB (3,203,400 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bemsdk.dll
libvxSigComp2 2.4.60.0 73.32 KB (75,080 bytes) 25/07/2008 9:47 AM Symantec Corporation c:\program files\symantec\backup exec\raws\libvxsigcomp2.dll
bebsdu 12.5.2213.0 340.32 KB (348,488 bytes) 11/09/2008 9:34 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bebsdu.dll
ndmpsrvr 12.5.2213.154 2.81 MB (2,947,400 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\ndmpsrvr.dll
bedssms 12.5.2213.0 96.82 KB (99,144 bytes) 08/09/2008 11:12 AM Symantec Corporation c:\program files\symantec\backup exec\raws\bedssms.dll
bedsnt5 12.5.2213.154 2.44 MB (2,555,208 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedsnt5.dll
bedssql2 12.5.2213.154 750.82 KB (768,840 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedssql2.dll
bedsxese 12.5.2213.154 210.82 KB (215,880 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedsxese.dll
bedsmbox 12.5.2213.142 405.82 KB (415,560 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedsmbox.dll
remprov 12.5.2213.133 75.32 KB (77,128 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\remprov.dll
edbprov 12.5.2213.142 794.32 KB (813,384 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\edbprov.dll
beerrors_EN 12.5.2213.133 235.32 KB (240,968 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\beerrors_en.dll
virtapi 1.0.114.133 114.05 KB (116,784 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\virtapi.dll
bedsnote 12.5.2213.133 759.32 KB (777,544 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedsnote.dll
bedsmdoc 12.5.2213.0 94.82 KB (97,096 bytes) 08/09/2008 11:12 AM Symantec Corporation c:\program files\symantec\backup exec\raws\bedsmdoc.dll
bedssps2 12.5.2213.155 1.46 MB (1,533,256 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedssps2.dll
bedssps3 12.5.2213.155 1.43 MB (1,502,024 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedssps3.dll
bedsupfs 12.5.2213.0 50.32 KB (51,528 bytes) 08/09/2008 11:12 AM Symantec Corporation c:\program files\symantec\backup exec\raws\bedsupfs.dll
bedsshadow 12.5.2213.154 1.49 MB (1,558,344 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedsshadow.dll
engine_EN 12.5.2213.133 624.32 KB (639,304 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\engine_en.dll
bedsoffhost 12.5.2213.0 544.32 KB (557,384 bytes) 11/09/2008 9:34 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedsoffhost.dll
bedscps 12.5.2213.0 267.32 KB (273,736 bytes) 11/09/2008 9:34 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedscps.dll
bedssmsp 12.5.2213.146 22.32 KB (22,856 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedssmsp.dll
bedsra 12.5.2213.0 136.82 KB (140,104 bytes) 08/09/2008 11:13 AM Symantec Corporation c:\program files\symantec\backup exec\raws\bedsra.dll
bedsrman 12.5.2213.137 526.82 KB (539,464 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedsrman.dll
dbsb 12.5.2213.0 31.32 KB (32,072 bytes) 11/09/2008 9:35 PM Symantec Corporation c:\program files\symantec\backup exec\raws\dbsb.dll
bedsdb2 12.5.2213.0 391.32 KB (400,712 bytes) 11/09/2008 9:34 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedsdb2.dll
bedsss 12.5.2213.0 401.82 KB (411,464 bytes) 11/09/2008 9:34 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedsss.dll
bedsadgran 12.5.2213.142 1.36 MB (1,422,152 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedsadgran.dll
bedsev 12.5.2213.0 621.32 KB (636,232 bytes) 11/09/2008 9:34 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedsev.dll
bedsvrtsrv 12.5.2213.0 442.82 KB (453,448 bytes) 11/09/2008 9:34 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedsvrtsrv.dll
bedsvmvcb 12.5.2213.154 987.32 KB (1,011,016 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedsvmvcb.dll
bexml 12.5.2213.0 97.82 KB (100,168 bytes) 08/09/2008 11:12 AM Symantec Corporation c:\program files\symantec\backup exec\raws\bexml.dll
BEDiskFind 12.5.2213.0 58.32 KB (59,720 bytes) 08/09/2008 11:13 AM Symantec Corporation c:\program files\symantec\backup exec\raws\bediskfind.dll
bedisk 12.5.2213.133 960.32 KB (983,368 bytes) 13/11/2009 1:01 PM Symantec Corporation c:\program files\symantec\backup exec\raws\bedisk.dll
dlomaintsvcu 3.1.341.3201 1.37 MB (1,435,000 bytes) 16/09/2008 8:39 PM Symantec Corporation c:\program files (x86)\symantec\backup exec\nt\dlomaintsvcu.exeASKER
jcgriff2awesome knowledge of kernel debug analysis and troubleshooting led to the quick resolution of the problem. Thank you.
Hi -
You are very welcome. Glad to hear of your BSOD-free status and hope it stays that way.
Regards. . .
jcgriff2
You are very welcome. Glad to hear of your BSOD-free status and hope it stays that way.
Regards. . .
jcgriff2
ASKER
jcgriff2, My experience with you has already allowed me to use driver verifier to weed out a problem on my own home theatre PC. Another tool for the kit. I learned something very valuable through the process.
Thanks again
Thanks again
http://www.microsoft.com/whdc/devtools/debugging/default.mspx
it used to tell you which file cause the issue