BSOD Windows 2003 HP Blade to emc san

Had a BSOD last night. server re-booted. ran dump file through debug tool.
Any ideas how to troubleshoot this issue ?

here are results:

Microsoft (R) Windows Debugger Version 6.12.0002.633 X86
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Users\astoner.WLS\Documents\Mini042010-03.dmp]
Mini Kernel Dump File: Only registers and stack trace are available

Symbol search path is: C:\symbols
Executable search path is:
Windows Server 2003 Kernel Version 3790 (Service Pack 2) MP (8 procs) Free x86 compatible
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Built by: 3790.srv03_sp2_gdr.070304-2240
Machine Name:
Kernel base = 0x80800000 PsLoadedModuleList = 0x808a6ea8
Debug session time: Tue Apr 20 21:34:05.797 2010 (UTC - 5:00)
System Uptime: 0 days 10:01:40.941
Loading Kernel Symbols
...............................................................
..................................................
Loading User Symbols
Loading unloaded module list
......
Unable to load image emcmpio.sys, Win32 error 0n2
*** WARNING: Unable to verify timestamp for emcmpio.sys
*** ERROR: Module load completed but symbols could not be loaded for emcmpio.sys
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 100000C5, {4, d0000002, 1, 808933b7}

*** WARNING: Unable to verify timestamp for mpio.sys
*** ERROR: Module load completed but symbols could not be loaded for mpio.sys
Probably caused by : emcmpio.sys ( emcmpio+12a0 )

Followup: MachineOwner
---------

5: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

DRIVER_CORRUPTED_EXPOOL (c5)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is
caused by drivers that have corrupted the system pool.  Run the driver
verifier against any new (or suspect) drivers, and if that doesn't turn up
the culprit, then use gflags to enable special pool.
Arguments:
Arg1: 00000004, memory referenced
Arg2: d0000002, IRQL
Arg3: 00000001, value 0 = read operation, 1 = write operation
Arg4: 808933b7, address which referenced memory

Debugging Details:
------------------


BUGCHECK_STR:  0xC5_D0000002

CURRENT_IRQL:  2

FAULTING_IP:
nt!ExAllocatePoolWithTag+83f
808933b7 897004          mov     dword ptr [eax+4],esi

CUSTOMER_CRASH_COUNT:  3

DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP

PROCESS_NAME:  System

LAST_CONTROL_TRANSFER:  from f725c2a0 to 808933b7

STACK_TEXT:  
f791a91c f725c2a0 00000000 00000000 4d636d45 nt!ExAllocatePoolWithTag+0x83f
WARNING: Stack unwind information not available. Following frames may be wrong.
f791a944 f726c136 000001e0 00000003 4d636d45 emcmpio+0x12a0
f791a958 f726c917 000001e0 00000001 f791a978 emcmpio+0x11136
f791a968 f726cc99 97850980 00000001 f791a98c emcmpio+0x11917
f791a978 f726ced0 97850980 93450ef0 97850980 emcmpio+0x11c99
f791a98c f726d041 97850980 93450ef0 93450f14 emcmpio+0x11ed0
f791a9b0 f7280e7f 97850980 93450ef0 00000000 emcmpio+0x12041
f791a9e0 f726f0c6 96d3f4d0 96d4b000 00000000 emcmpio+0x25e7f
f791ac74 f72783b0 96d3f4d0 96d432d8 96d544e0 emcmpio+0x140c6
f791acc4 f73188f6 978a02f0 96d3f4d0 96d3f4d0 emcmpio+0x1d3b0
f791ad6c 808ec1eb 00d54428 96cda228 808ae5fc mpio+0x98f6
f791ad80 8088043d 91c1f2a0 00000000 978993f0 nt!IopProcessWorkItem+0x13
f791adac 80949b7c 91c1f2a0 00000000 00000000 nt!ExpWorkerThread+0xeb
f791addc 8088e062 80880352 00000001 00000000 nt!PspSystemThreadStartup+0x2e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16


STACK_COMMAND:  kb

FOLLOWUP_IP:
emcmpio+12a0
f725c2a0 ??              ???

SYMBOL_STACK_INDEX:  1

SYMBOL_NAME:  emcmpio+12a0

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: emcmpio

IMAGE_NAME:  emcmpio.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  448e2da6

FAILURE_BUCKET_ID:  0xC5_D0000002_emcmpio+12a0

BUCKET_ID:  0xC5_D0000002_emcmpio+12a0

Followup: MachineOwner
---------

5: kd> lmvm emcmpio
start    end        module name
f725b000 f7292000   emcmpio  T (no symbols)          
    Loaded symbol image file: emcmpio.sys
    Image path: emcmpio.sys
    Image name: emcmpio.sys
    Timestamp:        Mon Jun 12 22:14:46 2006 (448E2DA6)
    CheckSum:         00034E80
    ImageSize:        00037000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4
andrew5499Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

TripyreCommented:
Looks like your PowerPath software had an issue.  Make sure you have the latest version installed.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
DanMerkCommented:
It looks like the emcmpio.sys driver file failed. Do you run EMC PowerPath or antoher product from EMC? Is there any Errors in Event Manager that you could post? Unfortunately, the timestamp is messed up (since it thinks it happened at Mon Jun 12 22:14:46 2006) so I can't give you an exact time of failure.

What I can say for sure is that the driver definately failed base on this: DRIVER_FAULT_SERVER_MINIDUMP. Without more information, I can't say exactly why.
andrew5499Author Commented:
After the reboot (7:33pm tuesday) - This event shows up:

The description for Event ID ( 2 ) in Source ( elxplus ) cannot be found. Either the component that raises this event is not installed on your local computer, or the installation is corrupted. You can install or repair the component on the local computer, or contact the component manufacturer for a newer version.

If the event was saved from another computer or forwarded from a remote computer, you might have to include display information with the events when saving them or when setting up the forwarding s , ScsiPort0, LPe1105-HP, 2.50A8, 10000000c95ea718.
Protecting & Securing Your Critical Data

Considering 93 percent of companies file for bankruptcy within 12 months of a disaster that blocked access to their data for 10 days or more, planning for the worst is just smart business. Learn how Acronis Backup integrates security at every stage

DanMerkCommented:
In device manager, do you notice anything that is flagged as not working correctly? (Remember to look for hidden devices also)
andrew5499Author Commented:
not sure what this
under non-plug and play drivers.
VSCore mferkdk

show error (yellow exclamation)
DanMerkCommented:
Typically speaking, it should be Mcafee, and the file for it should be at c:\program files\mcafee\virusscan enterprise\mferkdk.sys. If it is showing an error, then it might be best to simply uniinstall and reinstall the mcafee software as it is the most likely canidate to cause this issue. The mcafee uninstaller can be found at  http://download.mcafee.com/products/licensed/cust_support_patches/MCPR.exe (I believe that this does enterprise also).

After reinstalling it, I guess the only thing that can be done is to wait and see if the issue occurs again. If it does, then there must be another issue. (I know this doesn't sound helpful, but it is really the only way to see if this resolves the issue :( )
acl-puzzCommented:
you shold also call emc customer care  by sending them this crash dump as it is clearly indicating

 Probably caused by : emcmpio.sys
andrew5499Author Commented:
Update info:
Called Hp, they indicate could be firmware issue with a controller card.
Will be installing tomorrow AM - and update if this is the issue.

Appreciate all your feedback - will update.
andrew5499Author Commented:
here is a link to a very informative HP forum.
Looks like this firmware has caused a number of issues for people. Especially the E200i controller card and BSOD.

http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1271956012141+28353475&threadId=1274967

andrew5499Author Commented:
Awarding point to all that commented.
Looks like cause was HP firmware - but has been ongoing issue with HP controller card.
acl-puzzCommented:
so what is fix you it? did HP support helped you in this?
acl-puzzCommented:
"so what is fix you it?" sorry typo mistake i meant what is the fix you used on it?
andrew5499Author Commented:
Well right now I am 2 days with BSOD - the firmware update looks likely.

Reason is: as soon as I got the HP tech online and he saw the specific Blue Screen errors, he immediately referred me to the firmware update. It may not be the solution, only time will tell - but rarely do you hear support techs from HP come up with a solution so fast.

Most of my bugchecks - referred to drivers for either my backup software, or the last one was our san drivers...HP Indicated that the controller could cause the BSOD - but the bugcheck shows what driver or app. was running at that time.

Still hoping that is it is controller.

andrew5499Author Commented:
Still had a blue screen, so HP controller card did not help.
put a call into EMC.
acl-puzzCommented:
its bad....i wonder what sort of testing these hardware vendors do before putting their product into market...we cant just blame windows for these as you can see emc system file is creating problems BSOD errors mostly caused by third party drivers but poor microsoft usually have to bear all the venoms ...lol

hopefully EMC will solve it!
DanMerkCommented:
Have you addressed the mccaffe driver error in device manager? If not, that is something to try.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Legacy OS

From novice to tech pro — start learning today.