Dell Poweredge R620 continuous OS corruption
Posted on 2014-01-03
We have a brand new Dell Poweredge R620 that was put into production in September 2013 and since then, every 4 weeks the system files become corrupted and we have to reload the OS from scratch or recover from a Backup Exec DR disk. We notice the issue when we try to launch an application on the server, such as Backup Exec, and receive a missing DLL file message and the program won't launch. If the systems is rebooted it goes into recovery mode and can't load the OS. We have opened numerous cases with Dell and Microsoft, ran diagnostics on the server and tape library, updated all the firmware, drivers, etc but no solution. The corruption always happens on a Monday, that is why I think there is some sort of process running on the Dell that is causing the corruption. Yesterday, I discovered the Patrol Read process that appears to run on a Saturday, once a month on the embedded H310 mini controller. I set that to "manual" in case that is causing the corruption on the local RAID 1 array.
The server has a PERC 310 mini embedded controller for 2 local drives in a RAID 1 array.
There is a 6Gbps SAS controller connected to an external tape library (Brand new Dell PV 124t LTO 6 library). There is a PERC h810 controller card attached to an external DAS (brand new Dell MD1200). The server was originally loaded with Windows 2008 R2 x64 and after the first crash we installed Windows 2008 Standard x64. The only software running on the server is Backup Exec 2012 SP3 and EMC Application Xtender for our document management system. This program just stores some configuration settings for the Document storage repository on the MD1200 and isn't running many processes. The same software is installed on numerous Windows 7 PCs in our environment and has never caused any issues.
I'm guessing that there is some sort of Dell process that is causing the corruption or perhaps a bad sector on one of the drives or an issue with the PERC controller. However, none of the Dell diags have shown any H/W errors.
We have many Dell PE servers (r610s), PE 2950s and never experienced this type of issue. This "12th" generation server takes forever to boot and has been a complete nightmare!
Any suggestions are appreciated.