Solved

Analysis of Memory dump debug output

Posted on 2010-09-08
16
1,037 Views
Last Modified: 2013-11-29
Hello, Can anyone offer me insight based on this  !analyze bug check finding. I will be happy to share the WEW debug workspace if anyone thinks it will help.

This is Q-Xeon 2.26 12GB white box. Running 2008 64Bit It is DC, Hyper-V and SQL (honestly I didnt do it.) Nothing serious going on in hyper space, rather a 2003 and 2000 running with no load.

In the past 7 days it has had two crashes and rebooted. Different times\days. No application or system event action logged, coinciding before the crashes, Everything comes back normally. No device error messages.

No changes have been made since routine updates late July. No SQL management plan running. No backup action going on. But it did leave behind a dmp file that may offer clues.
 
"DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT" this is what has me wondering? A incompatible\uncertified driver? If so how do I find out which? Before I got the debug done I was leaning to expect hardware?
The watchdog timeout?

I surely appreciate any assistance.
Thane C

0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
Arguments:
Arg1: 0000000000000019, Clock interrupt time out interval in nominal clock ticks.
Arg2: 0000000000000000, 0.
Arg3: fffff88001e5d180, The PRCB address of the hung processor.
Arg4: 0000000000000002, 0.

Debugging Details:
------------------


BUGCHECK_STR:  CLOCK_WATCHDOG_TIMEOUT_8_PROC

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

PROCESS_NAME:  System

CURRENT_IRQL:  d

STACK_TEXT:  
fffff800`0151aa28 fffff800`01685443 : 00000000`00000101 00000000`00000019 00000000`00000000 fffff880`01e5d180 : nt!KeBugCheckEx
fffff800`0151aa30 fffff800`016dfcf7 : 00000000`00000000 fffff800`00000002 00000000`00002710 fffffa80`0a6ff278 : nt! ?? ::FNODOBFM::`string'+0x4dfe
fffff800`0151aac0 fffff800`01628895 : fffff800`0164d460 fffff800`0151ac70 fffff800`0164d460 fffff800`00000000 : nt!KeUpdateSystemTime+0x377
fffff800`0151abc0 fffff800`016d3af3 : 00000000`00000000 fffff880`03392328 fffff800`01855300 00000000`00000004 : hal!HalpHpetClockInterrupt+0x8d
fffff800`0151abf0 fffff800`016dfdd2 : fffff800`01851e80 fffff800`00000002 00000000`00000002 fffff880`00000000 : nt!KiInterruptDispatchNoLock+0x163
fffff800`0151ad80 00000000`00000000 : fffff800`0151b000 fffff800`01515000 fffff800`0151ad40 00000000`00000000 : nt!KiIdleLoop+0x32


STACK_COMMAND:  kb

SYMBOL_NAME:  ANALYSIS_INCONCLUSIVE

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: Unknown_Module

IMAGE_NAME:  Unknown_Image

DEBUG_FLR_IMAGE_TIMESTAMP:  0

FAILURE_BUCKET_ID:  X64_CLOCK_WATCHDOG_TIMEOUT_8_PROC_ANALYSIS_INCONCLUSIVE

BUCKET_ID:  X64_CLOCK_WATCHDOG_TIMEOUT_8_PROC_ANALYSIS_INCONCLUSIVE
0
Comment
Question by:ThaneVcr
  • 10
  • 3
  • 2
  • +1
16 Comments
 
LVL 14

Expert Comment

by:Justin Yeung
Comment Utility
did you use the windows debugging tools to debug this dump file that saved c:\windows\minidump

http://www.microsoft.com/whdc/devtools/debugging/default.mspx

it used to tell you which file cause the issue
0
 

Author Comment

by:ThaneVcr
Comment Utility
Yes it was Windows 7 professional Debugger

WinDbg 6.11.0001.404 x86 Build 7600

Symbols path set

SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
0
 
LVL 91

Expert Comment

by:nobus
Comment Utility
it looks like one of your processors does not work :
>>>
CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
0
 

Author Comment

by:ThaneVcr
Comment Utility
nobus - OK so that is the direction you would think. It certainly behaves like a hardware problem.  

What if there was an uncertified driver that somehow blocked the interrupt  
Is "DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT" anything to run down?
This server doesnt look unhappy in device manager, not a lot of device drivers that look questionable.
0
 

Author Comment

by:ThaneVcr
Comment Utility
justinyeung: _ failed to mention this is debug from an actual 500 MB kernel dump not a small memory dump. So therefore it got created c:\windows\memory.dmp as per default path.
0
 
LVL 12

Accepted Solution

by:
John Griffith earned 500 total points
Comment Utility
Hi -
0x101 = CPU hang.  Info on 0x101 --> http://www.sevenforums.com/crashes-debugging/35427-win-7-clock-interrupt-bsod-101-error.html#post356791
"DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT"  = no 3rd party driver named, so probable cause goes to Windows driver; based on stack, hal.dll or NT = Windows Kernel & Executive
Please re-run Windbg and paste the commands in the code box into the kd> command line.
I would also suggest running Driver Verifier.  If 3rd party driver flagged, it will BSOD your system -
http://jcgriff2.com/driver_verifier.htm
Would you also please run msinfo32 and save as an NFO file (default file ext) -
START | type msinfo32 | zip up & attach
This does appear to be hardware related; however, I have seen 0x101 where 3rd party drivers are at fault.
 Regards. . .
jcgriff2



Paste one line at a time; hit enter

___________________________________



!analyze -v; kv; lmnt; lmntsm



!for_each_module .echo @#ModuleIndex : @#Base @#End @#ModuleName @#ImageName  @#LoadedImageName



!for_each_module .echo @#ModuleName fver = @#FileVersion pver = @#ProductVersion 

Open in new window

0
 

Author Comment

by:ThaneVcr
Comment Utility
jcgriff - Thanks for the direction. Please find a txt output of the debug per your commands. Also the nfo you requested.

I will post the findings of the driver verifier pronto. I had to append an acceptable extension to the nfo remove it and you should have what you requested.

Thanks for the help.
 
srv002-debug.txt
srv002-msinfo.nfo.txt
0
 

Author Comment

by:ThaneVcr
Comment Utility
jcgriff - sorry read post ..zip it you asked...my apology. I will try and follow instructions better.

While I am reading about how to run DV, please confirm this says it can BSOD. As this is a production server without remote lights out is this likely to hang needing a power off if it finds a non-compatible driver.

I can deal with a reset as long as it comes back live. It is set to Auto -restart on the startup and recovery options.
T





0
Want to promote your upcoming event?

Is your company attending an event or exhibiting at a trade show soon? Are you speaking at a conference? Spread the word by using a promotional banner in your email signature. This will ensure your organization’s most important contacts are in the know.

 
LVL 91

Expert Comment

by:nobus
Comment Utility
if it points to the cpu - don't ask "what if" - test with another one ..
0
 

Author Comment

by:ThaneVcr
Comment Utility
Here is the driver verifier output

driververifier.txt
0
 

Author Comment

by:ThaneVcr
Comment Utility
Please see this KB.
http://support.microsoft.com/kb/975530

So far it looks good. It is that processor, that role (hyper-v) and it doesnt currently have this fix. It has only one of the referenced HV* files and it is older than what is required.

I have checked it is not on the list of updates waiting either.
T
0
 

Author Comment

by:ThaneVcr
Comment Utility
I have gone ahead and installed KB975530.  Things are fine, we will observe for a a couple of weeks and see if anything changes. So I am suggesting keeping the question open.

Thanks for your help. Very grateful.
0
 
LVL 12

Expert Comment

by:John Griffith
Comment Utility
Hi -
Nice find on the kb.
I would advise updating your Intel(R) 82574L Gigabit Network Connection driver -
e1q60x64.sys Fri Apr 03 16:03:40 2009 (49D695CC)
http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=18725&lang=eng
Thank you for the files.
The only other item I noticed was Symantec Backup.  There are 2 DLL's - 2006 & 2007 that are older than the others.  This may be nothing; just wanted to note it.\
Regards. . .
jcgriff2
 
 







vxcrypto	3.1.1.6	1.25 MB (1,314,360 bytes)	06/09/2006 10:32 PM	Redistributed by Symantec	c:\program files\symantec\backup exec\raws\vxcrypto.dll



vxace_3i	3.1.1.29	1.87 MB (1,955,640 bytes)	26/11/2007 4:27 AM	Redistributed by Symantec	c:\program files\symantec\backup exec\raws\vxace_3i.dll





beremote	12.5.2213.158	1.68 MB (1,760,584 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\beremote.exe

bestdutl	12.5.2213.103	162.32 KB (166,216 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bestdutl.dll

becrypto	12.5.2213.0	51.32 KB (52,552 bytes)	08/09/2008 11:12 AM	Symantec Corporation	c:\program files\symantec\backup exec\raws\becrypto.dll

beclass	12.5.2213.154	936.32 KB (958,792 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\beclass.dll

serdll	12.5.2213.133	224.82 KB (230,216 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\serdll.dll

besocket	12.5.2213.0	267.32 KB (273,736 bytes)	08/09/2008 11:12 AM	Symantec Corporation	c:\program files\symantec\backup exec\raws\besocket.dll

becluster	12.5.2213.146	478.32 KB (489,800 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\becluster.dll

bevs	12.5.2213.0	278.82 KB (285,512 bytes)	08/09/2008 11:12 AM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bevs.dll

benetapi	12.5.2213.0	24.82 KB (25,416 bytes)	08/09/2008 11:13 AM	Symantec Corporation	c:\program files\symantec\backup exec\raws\benetapi.dll

benetutl	12.5.2213.0	24.32 KB (24,904 bytes)	08/09/2008 11:13 AM	Symantec Corporation	c:\program files\symantec\backup exec\raws\benetutl.dll

ndmpcomm	12.5.2213.111	448.32 KB (459,080 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\ndmpcomm.dll

bedscomn	12.5.2213.154	317.82 KB (325,448 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedscomn.dll

bemsdk	12.5.2213.138	3.06 MB (3,203,400 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bemsdk.dll

libvxSigComp2	2.4.60.0	73.32 KB (75,080 bytes)	25/07/2008 9:47 AM	Symantec Corporation	c:\program files\symantec\backup exec\raws\libvxsigcomp2.dll

bebsdu	12.5.2213.0	340.32 KB (348,488 bytes)	11/09/2008 9:34 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bebsdu.dll

ndmpsrvr	12.5.2213.154	2.81 MB (2,947,400 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\ndmpsrvr.dll

bedssms	12.5.2213.0	96.82 KB (99,144 bytes)	08/09/2008 11:12 AM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedssms.dll

bedsnt5	12.5.2213.154	2.44 MB (2,555,208 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedsnt5.dll

bedssql2	12.5.2213.154	750.82 KB (768,840 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedssql2.dll

bedsxese	12.5.2213.154	210.82 KB (215,880 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedsxese.dll

bedsmbox	12.5.2213.142	405.82 KB (415,560 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedsmbox.dll

remprov	12.5.2213.133	75.32 KB (77,128 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\remprov.dll

edbprov	12.5.2213.142	794.32 KB (813,384 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\edbprov.dll

beerrors_EN	12.5.2213.133	235.32 KB (240,968 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\beerrors_en.dll

virtapi	1.0.114.133	114.05 KB (116,784 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\virtapi.dll

bedsnote	12.5.2213.133	759.32 KB (777,544 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedsnote.dll

bedsmdoc	12.5.2213.0	94.82 KB (97,096 bytes)	08/09/2008 11:12 AM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedsmdoc.dll

bedssps2	12.5.2213.155	1.46 MB (1,533,256 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedssps2.dll

bedssps3	12.5.2213.155	1.43 MB (1,502,024 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedssps3.dll

bedsupfs	12.5.2213.0	50.32 KB (51,528 bytes)	08/09/2008 11:12 AM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedsupfs.dll

bedsshadow	12.5.2213.154	1.49 MB (1,558,344 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedsshadow.dll

engine_EN	12.5.2213.133	624.32 KB (639,304 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\engine_en.dll

bedsoffhost	12.5.2213.0	544.32 KB (557,384 bytes)	11/09/2008 9:34 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedsoffhost.dll

bedscps	12.5.2213.0	267.32 KB (273,736 bytes)	11/09/2008 9:34 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedscps.dll

bedssmsp	12.5.2213.146	22.32 KB (22,856 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedssmsp.dll

bedsra	12.5.2213.0	136.82 KB (140,104 bytes)	08/09/2008 11:13 AM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedsra.dll

bedsrman	12.5.2213.137	526.82 KB (539,464 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedsrman.dll

dbsb	12.5.2213.0	31.32 KB (32,072 bytes)	11/09/2008 9:35 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\dbsb.dll

bedsdb2	12.5.2213.0	391.32 KB (400,712 bytes)	11/09/2008 9:34 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedsdb2.dll

bedsss	12.5.2213.0	401.82 KB (411,464 bytes)	11/09/2008 9:34 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedsss.dll

bedsadgran	12.5.2213.142	1.36 MB (1,422,152 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedsadgran.dll

bedsev	12.5.2213.0	621.32 KB (636,232 bytes)	11/09/2008 9:34 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedsev.dll

bedsvrtsrv	12.5.2213.0	442.82 KB (453,448 bytes)	11/09/2008 9:34 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedsvrtsrv.dll

bedsvmvcb	12.5.2213.154	987.32 KB (1,011,016 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedsvmvcb.dll

bexml	12.5.2213.0	97.82 KB (100,168 bytes)	08/09/2008 11:12 AM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bexml.dll

BEDiskFind	12.5.2213.0	58.32 KB (59,720 bytes)	08/09/2008 11:13 AM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bediskfind.dll

bedisk	12.5.2213.133	960.32 KB (983,368 bytes)	13/11/2009 1:01 PM	Symantec Corporation	c:\program files\symantec\backup exec\raws\bedisk.dll

dlomaintsvcu	3.1.341.3201	1.37 MB (1,435,000 bytes)	16/09/2008 8:39 PM	Symantec Corporation	c:\program files (x86)\symantec\backup exec\nt\dlomaintsvcu.exe

Open in new window

0
 

Author Closing Comment

by:ThaneVcr
Comment Utility
jcgriff2awesome knowledge of kernel debug analysis and troubleshooting led to the quick resolution of the problem. Thank you.
0
 
LVL 12

Expert Comment

by:John Griffith
Comment Utility
Hi -
You are very welcome.  Glad to hear of your BSOD-free status and hope it stays that way.
Regards. . .
jcgriff2
 
0
 

Author Comment

by:ThaneVcr
Comment Utility
jcgriff2,  My experience with you has already allowed me to use driver verifier to weed out a problem on my own home theatre PC. Another tool for the kit. I learned something very valuable through the process.

Thanks again
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Windows Mobile Barcode Scanning These days almost every product has a barcode in some way... amongst there are 1D barcodes en 2D barcodes.. From http://www.barcodeman.com/faq/2d.php I found some handy definitions and insights. 1D barcodes …
Several part series to implement Internet Explorer 11 Enterprise Mode
The viewer will learn how to successfully create a multiboot device using the SARDU utility on Windows 7. Start the SARDU utility: Change the image directory to wherever you store your ISOs, this will prevent you from having 2 copies of an ISO wit…
With the advent of Windows 10, Microsoft is pushing a Get Windows 10 icon into the notification area (system tray) of qualifying computers. There are many reasons for wanting to remove this icon. This two-part Experts Exchange video Micro Tutorial s…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now