How to identify component causing "bus fatal error" on Dell PowerEdge T110

David Haycox
David Haycox used Ask the Experts™
on
A Dell PowerEdge T110 server running Server 2009 R2 has been rebooting because of a BugCheck.  The Dell hardware logs show the following (errors are the same each time the server reboots):

	Tue Mar 21 12:07:51 2017	A runtime critical stop occurred.	
	Tue Mar 21 12:07:24 2017	An OEM diagnostic event occurred.
	Tue Mar 21 12:07:24 2017	A bus fatal error was detected on a component at bus 0 device 28 function 5.	
	Tue Mar 21 12:07:24 2017	A bus fatal error was detected on a component at slot 4.

Open in new window

We have remote access to the server only.  Without physical access, how can we determine which component is causing the problem?

Thanks in advance for any assistance.
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Dr. KlahnPrincipal Software Engineer

Commented:
So far as I know there's not a utility to show PCI bus slot information.  PCI-Z comes close to what you need, but it does not report slot numbers.

You can get the information out of the Registry, but it is troublesome and requires exact knowledge of what is installed and where ... in other words, you need to already know the information you're trying to find out.

Open the Registry Editor and drill down to HKLM/System/CurrentControlSet/Enum/PCI.  Under this you'll find many, many keys corresponding to every PCI device which is now or has ever been installed in this system.  The LocationInformation field contains information about where that device was when it was last installed in the system.

PCI enumeration in Registry
Here's the nasty part.  Nothing indicates whether that device is in the system.  You have to know what is installed in that slot to know whether that device is installed in that slot.  This could be workable on a system that has never changed since the original installation, but on most systems things have been swapped back and forth, devices have changed, and each slot might have ten potential candidates that have been in that location.  So on most systems, this is not a workable way to find out what's in the system without seeing what's in that slot.  And then of course you already know what's in the slot.
David HaycoxConsultant Engineer

Author

Commented:
Fantastic, thanks.  By drilling down through the device manager I found this though:

Device Manager extract
So that's not a removable device in any case, it's on the system board I would say.  So if there's a fault with that it'll be either drivers, firmware or a board change?
Principal Software Engineer
Commented:
You're correct.  That's part of the chipset and if it's hardware faulting, motherboard replacement is the only solution that comes to mind.

This particular device is a PCIe controller but a fault in any of the bus controllers could certainly cause problems on the PCI bus, resulting in the next logged error on the PCI bus at slot 4.

Keep in mind though, the problem could possibly go the other way as well -- a faulty device in slot 4 could cause the bus controller to fault.
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

David HaycoxConsultant Engineer

Author

Commented:
Great, thanks.  We'll get slot 4 checked out.
Top Expert 2014

Commented:
Pity this was closed so quickly, was hoping someone would have ppsted the Linux method and also the generic Dell method of identifying the device. A lot of the time you can't even check via the OS as it won't boot with a PCIe error and all you get is POST error or the LCD display to go on.
David HaycoxConsultant Engineer

Author

Commented:
Oops, can ask another question easily enough if you think it will be worth it.
David HaycoxConsultant Engineer

Author

Commented:
Looks like PCI slot 4 is a USB3.0 controller we fitted recently in order to improve backup / restore speed.  We'll investigate it further.
Top Expert 2014

Commented:
PCIe 3.0 card?
David HaycoxConsultant Engineer

Author

Commented:
This one: https://www.startech.com/uk/Cards-Adapters/USB-3.0/Cards/2-Port-USB-3-PCI-Express-Card-with-SATA-Power~PEXUSB3S23

We've fitted many of these to different servers; this will be the first problem.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial