ESXi PSoD Exception 14

Hi All,

Can anyone here please assist me in troubleshooting the problem in random PSoD that affects one of my HP Blade server running ESXi 5.1 as per below screenshot:

PSoD
I'm not sure what else to do to begin troubleshooting this problem ?

Thanks
LVL 8
Senior IT System EngineerIT ProfessionalAsked:
Who is Participating?
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
Most PSODs are caused by a hardware issue.

This could be non compatible hardware, which is not on the HCL.

Check the VMware Hardware Compatability Lists HCL here

The VMware Hardware Compatibility List is the detailed lists showing actual vendor devices that are either physically tested or are similar to the devices tested by VMware or VMware partners. Items on the list are tested with VMware products and are known to operate correctly.Devices which are not on the list may function, but will not be supported by VMware.

http://www.vmware.com/go/hcl

So first checks...

1. What is the HP Blade Server, that's rather generic ? Is your hardware on the HCL ?

2. is your hardware up to date with Firmware, for BIOS, Storage, Network Controllers ?

3. Are you using the OEM HP version of ESXi 5.1 ?

4. Have you checked the memory is seated correctly?

5. Have you checked fans, CPU heatsinks, and fans?

6. Have you tested memory using memtest86+

7. If you have a support contract with HP, log a support request.

8. If you have a support contract with VMware, log a support request.

9. Random faults are difficult to track down.....how many VMs were running at time of crash?

10. Look back at your change database, what changes have been made to the server and environment.

11. Do you have a syslog server, or persistant storage of logs, so you can check back and have a look at the logs /var/logs/vmkernel.log, to check for any errors before the PSOD ?

12. Build version of ESXi - is it the latest?

13. Track down the World ID and the VM?

14. Is that VM OS supported for ESXi 5.1 ?

15. Is the network card in the VM VMXNET3 or E1000, there have been issues with builds of ESXi and VM nics, causing PSOD, e.g. bug in ESXi!

16. Supported CPU microcode, and are both CPUs the same.

17. Memory installed in correct banks.

18. Certified memory installed.

These are the troubleshooting steps you need to start performing.

There is not really a simple answer, of AH the PSOD is caused by that!

We've had issues where servers have been stable for years, and when we started to load them, and more VMs were on them, they used more memory, and we had a memory fault at the TOP of RAM on a server, at 496GB ish, and when the server was heavily loaded with VMs, and used that "memory module" the server would PSOD!
0
 
Senior IT System EngineerIT ProfessionalAuthor Commented:
ok, so in this case what log should I gather and analyze for the root cause analysis ?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I've listed the log in my post!

It may not reveal anything, but it worth a look, I'm also waiting for answers to the questions in my post.
0
Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
Senior IT System EngineerIT ProfessionalAuthor Commented:
SOmehow when I log the case to HP, they recommends me to update the iLO v4 firmware from the existing v1.4.0 into v2.02 (http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=5228286&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3Didx%253D2%257CswItem%253DMTX_8372c55483b9432abd53d91951%257CswEnvOID%253D4115%257CitemLocale%253D%257CswLang%253D%257Cmode%253D4%257Caction%253DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken)

since this is a well known issue according to them... that's rather strange, because how come I can see the one particular VMname in there not the ESXi host ?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
This was Bullet Point 2 in my post, Update and Check Firmware!
1
 
Senior IT System EngineerIT ProfessionalAuthor Commented:
Thanks !
0
 
Senior IT System EngineerIT ProfessionalAuthor Commented:
So in this case why the PSOD shows the VM name ? not the actual host name.

is there something happened caused by that particular VM ?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
It's possible we've seen VMs running unsupported OS, or network interfaces, or using defective memory cause PSODs.

Is it always this vm?
0
 
Senior IT System EngineerIT ProfessionalAuthor Commented:
No it is not always. but just curious as to why that VM name is displayed on the PSoD.

next time when the crashed happened i'll get some more information and post it in here.

My manager doesn't like the idea of upgrading the firmware for all of the Blade components for the time being, unless it is a must to upgrade from ESXi 5.1u1 into ESXi 5.5 and above.
0
All Courses

From novice to tech pro — start learning today.