Need help with ESX logs from servers that keep locking up.

Our servers have been locking up from time to time (one or two of them lock up every week) this server locked up last night around 10:23PM and was restarted at 9:11AM this morning.  This is the kernal.1 log file from the time period it locked up.

Jul 13 22:15:42 esxsvr10 vmkernel: 57:09:20:48.991 cpu2:13814)WARNING: NMP: nmp_DeviceRetryCommand: Device "naa.60060160a319170000e81cb579e9e011": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device. 
Jul 13 22:15:43 esxsvr10 vmkernel: 57:09:20:49.994 cpu4:4194)vmw_psp_mru: psp_mruSelectPathToActivateInt: Changing active path from vmhba1:C0:T9:L5 to vmhba2:C0:T11:L5 for device "naa.60060160a319170000e81cb579e9e011". 
Jul 13 22:15:43 esxsvr10 vmkernel: 57:09:20:49.994 cpu2:4232)WARNING: NMP: nmpDeviceAttemptFailover: Retry world failover device "naa.60060160a319170000e81cb579e9e011" - issuing command 0x41027f380540 
Jul 13 22:15:43 esxsvr10 vmkernel: 57:09:20:49.994 cpu6:4102)WARNING: NMP: nmpCompleteRetryForPath: Retry command 0x2a (0x41027f380540) to NMP device "naa.60060160a319170000e81cb579e9e011" failed on physical path "vmhba2:C0:T11:L5" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. 
Jul 13 22:15:43 esxsvr10 vmkernel: 57:09:20:49.994 cpu6:4102)WARNING: NMP: nmpCompleteRetryForPath: Retry world restored device "naa.60060160a319170000e81cb579e9e011" - no more commands to retry 
Jul 13 22:15:43 esxsvr10 vmkernel: 57:09:20:49.994 cpu6:4102)ScsiDeviceIO: 1688: Command 0x2a to device "naa.60060160a319170000e81cb579e9e011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. 
Jul 13 22:15:43 esxsvr10 vmkernel: 57:09:20:49.994 cpu6:4102)WARNING: ScsiCore: 1355: Power-on Reset occurred on naa.60060160a319170000e81cb579e9e011 
Jul 13 22:15:48 esxsvr10 vmkernel: 57:09:20:55.319 cpu1:4233)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027dc0c440) to NMP device "mpx.vmhba32:C0:T0:L0" failed on physical path "vmhba32:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:15:48 esxsvr10 vmkernel: 57:09:20:55.319 cpu1:4233)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:15:48 esxsvr10 vmkernel: 57:09:20:55.325 cpu2:4239)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027dc10540) to NMP device "mpx.vmhba3:C0:T0:L0" failed on physical path "vmhba3:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:15:48 esxsvr10 vmkernel: 57:09:20:55.325 cpu2:4239)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba3:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:18:32 esxsvr10 vmkernel: 57:09:23:38.979 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f390840) to NMP device "naa.60060160a3191700527e52783ee9e011" failed on physical path "vmhba2:C0:T11:L4" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0x3. 
Jul 13 22:18:32 esxsvr10 vmkernel: 57:09:23:38.979 cpu6:4102)WARNING: NMP: nmp_DeviceRetryCommand: Device "naa.60060160a3191700527e52783ee9e011": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device. 
Jul 13 22:18:33 esxsvr10 vmkernel: 57:09:23:39.982 cpu4:4193)vmw_psp_mru: psp_mruSelectPathToActivateInt: Changing active path from vmhba2:C0:T11:L4 to vmhba2:C0:T8:L4 for device "naa.60060160a3191700527e52783ee9e011". 
Jul 13 22:18:33 esxsvr10 vmkernel: 57:09:23:39.982 cpu3:4232)WARNING: NMP: nmpDeviceAttemptFailover: Retry world failover device "naa.60060160a3191700527e52783ee9e011" - issuing command 0x41027f390840 
Jul 13 22:18:33 esxsvr10 vmkernel: 57:09:23:39.982 cpu6:4102)WARNING: NMP: nmpCompleteRetryForPath: Retry command 0x2a (0x41027f390840) to NMP device "naa.60060160a3191700527e52783ee9e011" failed on physical path "vmhba2:C0:T8:L4" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. 
Jul 13 22:18:33 esxsvr10 vmkernel: 57:09:23:39.982 cpu6:4102)WARNING: NMP: nmpCompleteRetryForPath: Retry world restored device "naa.60060160a3191700527e52783ee9e011" - no more commands to retry 
Jul 13 22:18:33 esxsvr10 vmkernel: 57:09:23:39.982 cpu6:4102)ScsiDeviceIO: 1688: Command 0x2a to device "naa.60060160a3191700527e52783ee9e011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. 
Jul 13 22:18:33 esxsvr10 vmkernel: 57:09:23:39.982 cpu6:4102)WARNING: ScsiCore: 1355: Power-on Reset occurred on naa.60060160a3191700527e52783ee9e011 
Jul 13 22:18:33 esxsvr10 vmkernel: 57:09:23:40.057 cpu6:16631)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4bc240) to NMP device "naa.60060160a319170000e81cb579e9e011" failed on physical path "vmhba2:C0:T11:L5" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0x3. 
Jul 13 22:18:33 esxsvr10 vmkernel: 57:09:23:40.057 cpu6:16631)WARNING: NMP: nmp_DeviceRetryCommand: Device "naa.60060160a319170000e81cb579e9e011": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device. 
Jul 13 22:18:34 esxsvr10 vmkernel: 57:09:23:40.984 cpu5:4196)vmw_psp_mru: psp_mruSelectPathToActivateInt: Changing active path from vmhba2:C0:T11:L5 to vmhba2:C0:T8:L5 for device "naa.60060160a319170000e81cb579e9e011". 
Jul 13 22:18:34 esxsvr10 vmkernel: 57:09:23:40.984 cpu3:4232)WARNING: NMP: nmpDeviceAttemptFailover: Retry world failover device "naa.60060160a319170000e81cb579e9e011" - issuing command 0x41027f4bc240 
Jul 13 22:18:34 esxsvr10 vmkernel: 57:09:23:40.984 cpu6:12538)WARNING: NMP: nmpCompleteRetryForPath: Retry command 0x2a (0x41027f4bc240) to NMP device "naa.60060160a319170000e81cb579e9e011" failed on physical path "vmhba2:C0:T8:L5" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. 
Jul 13 22:18:34 esxsvr10 vmkernel: 57:09:23:40.984 cpu6:12538)WARNING: NMP: nmpCompleteRetryForPath: Retry world restored device "naa.60060160a319170000e81cb579e9e011" - no more commands to retry 
Jul 13 22:18:34 esxsvr10 vmkernel: 57:09:23:40.984 cpu6:12538)ScsiDeviceIO: 1688: Command 0x2a to device "naa.60060160a319170000e81cb579e9e011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. 
Jul 13 22:18:34 esxsvr10 vmkernel: 57:09:23:40.984 cpu6:12538)WARNING: ScsiCore: 1355: Power-on Reset occurred on naa.60060160a319170000e81cb579e9e011 
Jul 13 22:18:38 esxsvr10 vmkernel: 57:09:23:45.253 cpu1:4233)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027dbdeb40) to NMP device "mpx.vmhba32:C0:T0:L0" failed on physical path "vmhba32:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:18:38 esxsvr10 vmkernel: 57:09:23:45.253 cpu1:4233)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:18:38 esxsvr10 vmkernel: 57:09:23:45.260 cpu3:4239)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027f3b4540) to NMP device "mpx.vmhba3:C0:T0:L0" failed on physical path "vmhba3:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:18:38 esxsvr10 vmkernel: 57:09:23:45.260 cpu3:4239)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba3:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:18:46 esxsvr10 vmkernel: 57:09:23:53.384 cpu1:4233)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027f3c8240) to NMP device "mpx.vmhba32:C0:T0:L0" failed on physical path "vmhba32:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:18:46 esxsvr10 vmkernel: 57:09:23:53.384 cpu1:4233)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:18:46 esxsvr10 vmkernel: 57:09:23:53.391 cpu3:4239)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027f3c8240) to NMP device "mpx.vmhba3:C0:T0:L0" failed on physical path "vmhba3:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:18:46 esxsvr10 vmkernel: 57:09:23:53.391 cpu3:4239)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba3:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:19:55 esxsvr10 vmkernel: 57:09:25:02.345 cpu2:4191)vmw_psp_mru: psp_mruSelectPathToActivateInt: Changing active path from vmhba1:C0:T10:L1 to vmhba1:C0:T8:L1 for device "naa.60060160a3191700f280c83121e9e011". 
Jul 13 22:20:00 esxsvr10 vmkernel: 57:09:25:07.616 cpu2:4233)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027f4ead40) to NMP device "mpx.vmhba32:C0:T0:L0" failed on physical path "vmhba32:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:20:00 esxsvr10 vmkernel: 57:09:25:07.616 cpu2:4233)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:20:00 esxsvr10 vmkernel: 57:09:25:07.623 cpu3:4239)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027f3cc440) to NMP device "mpx.vmhba3:C0:T0:L0" failed on physical path "vmhba3:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:20:00 esxsvr10 vmkernel: 57:09:25:07.623 cpu3:4239)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba3:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:20:09 esxsvr10 vmkernel: 57:09:25:15.764 cpu2:4233)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027f47a040) to NMP device "mpx.vmhba32:C0:T0:L0" failed on physical path "vmhba32:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:20:09 esxsvr10 vmkernel: 57:09:25:15.764 cpu2:4233)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:20:09 esxsvr10 vmkernel: 57:09:25:15.771 cpu5:4239)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027db02d40) to NMP device "mpx.vmhba3:C0:T0:L0" failed on physical path "vmhba3:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:20:09 esxsvr10 vmkernel: 57:09:25:15.771 cpu5:4239)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba3:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:20:39 esxsvr10 vmkernel: 57:09:25:46.107 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027dbe0540) to NMP device "naa.60060160a3191700527e52783ee9e011" failed on physical path "vmhba2:C0:T8:L4" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0x3. 
Jul 13 22:20:39 esxsvr10 vmkernel: 57:09:25:46.107 cpu6:4102)WARNING: NMP: nmp_DeviceRetryCommand: Device "naa.60060160a3191700527e52783ee9e011": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device. 
Jul 13 22:20:42 esxsvr10 vmkernel: 57:09:25:49.107 cpu2:4273)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027dbc0940) to NMP device "naa.60060160a319170000e81cb579e9e011" failed on physical path "vmhba2:C0:T8:L5" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0x3. 
Jul 13 22:20:42 esxsvr10 vmkernel: 57:09:25:49.107 cpu2:4273)WARNING: NMP: nmp_DeviceRetryCommand: Device "naa.60060160a319170000e81cb579e9e011": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device. 
Jul 13 22:20:43 esxsvr10 vmkernel: 57:09:25:49.649 cpu5:4192)vmw_psp_mru: psp_mruSelectPathToActivateInt: Changing active path from vmhba2:C0:T8:L4 to vmhba1:C0:T10:L4 for device "naa.60060160a3191700527e52783ee9e011". 
Jul 13 22:20:43 esxsvr10 vmkernel: 57:09:25:49.649 cpu2:4232)WARNING: NMP: nmpDeviceAttemptFailover: Retry world failover device "naa.60060160a3191700527e52783ee9e011" - issuing command 0x41027dbe0540 
Jul 13 22:20:43 esxsvr10 vmkernel: 57:09:25:49.667 cpu3:4099)WARNING: NMP: nmpCompleteRetryForPath: Retry command 0x2a (0x41027dbe0540) to NMP device "naa.60060160a3191700527e52783ee9e011" failed on physical path "vmhba1:C0:T10:L4" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. 
Jul 13 22:20:43 esxsvr10 vmkernel: 57:09:25:49.667 cpu3:4099)WARNING: NMP: nmpCompleteRetryForPath: Retry world restored device "naa.60060160a3191700527e52783ee9e011" - no more commands to retry 
Jul 13 22:20:43 esxsvr10 vmkernel: 57:09:25:49.667 cpu3:4099)ScsiDeviceIO: 1688: Command 0x2a to device "naa.60060160a3191700527e52783ee9e011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. 
Jul 13 22:20:43 esxsvr10 vmkernel: 57:09:25:49.667 cpu3:4099)WARNING: ScsiCore: 1355: Power-on Reset occurred on naa.60060160a3191700527e52783ee9e011 
Jul 13 22:20:43 esxsvr10 vmkernel: 57:09:25:50.110 cpu1:4189)vmw_psp_mru: psp_mruSelectPathToActivateInt: Changing active path from vmhba2:C0:T8:L5 to vmhba1:C0:T10:L5 for device "naa.60060160a319170000e81cb579e9e011". 
Jul 13 22:20:43 esxsvr10 vmkernel: 57:09:25:50.111 cpu3:4232)WARNING: NMP: nmpDeviceAttemptFailover: Retry world failover device "naa.60060160a319170000e81cb579e9e011" - issuing command 0x41027dbc0940 
Jul 13 22:20:43 esxsvr10 vmkernel: 57:09:25:50.111 cpu3:4099)WARNING: NMP: nmpCompleteRetryForPath: Retry command 0x2a (0x41027dbc0940) to NMP device "naa.60060160a319170000e81cb579e9e011" failed on physical path "vmhba1:C0:T10:L5" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. 
Jul 13 22:20:43 esxsvr10 vmkernel: 57:09:25:50.111 cpu3:4099)WARNING: NMP: nmpCompleteRetryForPath: Retry world restored device "naa.60060160a319170000e81cb579e9e011" - no more commands to retry 
Jul 13 22:20:43 esxsvr10 vmkernel: 57:09:25:50.111 cpu3:4099)ScsiDeviceIO: 1688: Command 0x2a to device "naa.60060160a319170000e81cb579e9e011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. 
Jul 13 22:20:43 esxsvr10 vmkernel: 57:09:25:50.111 cpu3:4099)WARNING: ScsiCore: 1355: Power-on Reset occurred on naa.60060160a319170000e81cb579e9e011 
Jul 13 22:20:45 esxsvr10 vmkernel: 57:09:25:52.642 cpu3:4233)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027dbd6940) to NMP device "mpx.vmhba32:C0:T0:L0" failed on physical path "vmhba32:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:20:45 esxsvr10 vmkernel: 57:09:25:52.642 cpu3:4233)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:20:46 esxsvr10 vmkernel: 57:09:25:52.648 cpu4:4239)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027dbd6940) to NMP device "mpx.vmhba3:C0:T0:L0" failed on physical path "vmhba3:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:20:46 esxsvr10 vmkernel: 57:09:25:52.648 cpu4:4239)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba3:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:20:54 esxsvr10 vmkernel: 57:09:26:00.857 cpu3:4233)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027dc02e40) to NMP device "mpx.vmhba32:C0:T0:L0" failed on physical path "vmhba32:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:20:54 esxsvr10 vmkernel: 57:09:26:00.857 cpu3:4233)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:20:54 esxsvr10 vmkernel: 57:09:26:00.863 cpu4:4239)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027dc02e40) to NMP device "mpx.vmhba3:C0:T0:L0" failed on physical path "vmhba3:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:20:54 esxsvr10 vmkernel: 57:09:26:00.863 cpu4:4239)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba3:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:23:33 esxsvr10 vmkernel: 57:09:28:40.167 cpu3:4099)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f45ea40) to NMP device "naa.60060160a319170000e81cb579e9e011" failed on physical path "vmhba1:C0:T10:L5" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0x3. 
Jul 13 22:23:33 esxsvr10 vmkernel: 57:09:28:40.167 cpu3:4099)WARNING: NMP: nmp_DeviceRetryCommand: Device "naa.60060160a319170000e81cb579e9e011": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device. 
Jul 13 22:23:34 esxsvr10 vmkernel: 57:09:28:40.977 cpu3:4099)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f442140) to NMP device "naa.60060160a3191700527e52783ee9e011" failed on physical path "vmhba1:C0:T10:L4" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. 
Jul 13 22:23:34 esxsvr10 vmkernel: 57:09:28:40.977 cpu3:4099)ScsiDeviceIO: 1688: Command 0x2a to device "naa.60060160a3191700527e52783ee9e011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. 
Jul 13 22:23:34 esxsvr10 vmkernel: 57:09:28:40.977 cpu3:4099)WARNING: ScsiCore: 1355: Power-on Reset occurred on naa.60060160a3191700527e52783ee9e011 
Jul 13 22:23:34 esxsvr10 vmkernel: 57:09:28:41.170 cpu7:4196)vmw_psp_mru: psp_mruSelectPathToActivateInt: Changing active path from vmhba1:C0:T10:L5 to vmhba1:C0:T8:L5 for device "naa.60060160a319170000e81cb579e9e011". 
Jul 13 22:23:34 esxsvr10 vmkernel: 57:09:28:41.170 cpu2:4232)WARNING: NMP: nmpDeviceAttemptFailover: Retry world failover device "naa.60060160a319170000e81cb579e9e011" - issuing command 0x41027f45ea40 
Jul 13 22:23:34 esxsvr10 vmkernel: 57:09:28:41.170 cpu3:16633)WARNING: NMP: nmpCompleteRetryForPath: Retry command 0x2a (0x41027f45ea40) to NMP device "naa.60060160a319170000e81cb579e9e011" failed on physical path "vmhba1:C0:T8:L5" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. 
Jul 13 22:23:34 esxsvr10 vmkernel: 57:09:28:41.170 cpu3:16633)WARNING: NMP: nmpCompleteRetryForPath: Retry world restored device "naa.60060160a319170000e81cb579e9e011" - no more commands to retry 
Jul 13 22:23:34 esxsvr10 vmkernel: 57:09:28:41.170 cpu3:16633)ScsiDeviceIO: 1688: Command 0x2a to device "naa.60060160a319170000e81cb579e9e011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. 
Jul 13 22:23:34 esxsvr10 vmkernel: 57:09:28:41.170 cpu3:16633)WARNING: ScsiCore: 1355: Power-on Reset occurred on naa.60060160a319170000e81cb579e9e011 
Jul 13 22:23:39 esxsvr10 vmkernel: 57:09:28:46.198 cpu2:4233)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027d904b40) to NMP device "mpx.vmhba32:C0:T0:L0" failed on physical path "vmhba32:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:23:39 esxsvr10 vmkernel: 57:09:28:46.198 cpu2:4233)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:23:39 esxsvr10 vmkernel: 57:09:28:46.204 cpu5:4239)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027f45c340) to NMP device "mpx.vmhba3:C0:T0:L0" failed on physical path "vmhba3:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:23:39 esxsvr10 vmkernel: 57:09:28:46.204 cpu5:4239)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba3:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:23:47 esxsvr10 vmkernel: 57:09:28:54.587 cpu3:4233)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027db1cd40) to NMP device "mpx.vmhba32:C0:T0:L0" failed on physical path "vmhba32:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:23:47 esxsvr10 vmkernel: 57:09:28:54.587 cpu3:4233)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:23:47 esxsvr10 vmkernel: 57:09:28:54.594 cpu5:4239)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027dcbe040) to NMP device "mpx.vmhba3:C0:T0:L0" failed on physical path "vmhba3:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:23:47 esxsvr10 vmkernel: 57:09:28:54.594 cpu5:4239)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba3:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:23:56 esxsvr10 vmkernel: 57:09:29:03.018 cpu2:4233)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027f460140) to NMP device "mpx.vmhba32:C0:T0:L0" failed on physical path "vmhba32:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:23:56 esxsvr10 vmkernel: 57:09:29:03.018 cpu2:4233)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:23:56 esxsvr10 vmkernel: 57:09:29:03.024 cpu4:4239)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41027f422340) to NMP device "mpx.vmhba3:C0:T0:L0" failed on physical path "vmhba3:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 13 22:23:56 esxsvr10 vmkernel: 57:09:29:03.024 cpu4:4239)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba3:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 
Jul 14 09:11:06 esxsvr10 vmkernel: 1)ScsiPath: 3839: Plugin 'NMP' claimed path 'vmhba1:C0:T1:L0' 
Jul 14 09:11:06 esxsvr10 vmkernel: 0:00:01:10.333 cpu4:4111)ScsiPath: 3839: Plugin 'NMP' claimed path 'vmhba1:C0:T0:L14' 
Jul 14 09:11:06 esxsvr10 vmkernel: 0:00:01:10.333 cpu4:4111)ScsiPath: 3839: Plugin 'NMP' claimed path 'vmhba1:C0:T1:L1' 
Jul 14 09:11:06 esxsvr10 vmkernel: 0:00:01:10.333 cpu4:4111)ScsiPath: 3839: Plugin 'NMP' claimed path 'vmhba1:C0:T0:L15' 
Jul 14 09:11:06 esxsvr10 vmkernel: 0:00:01:10.333 cpu4:4111)ScsiPath: 3839: Plugin 'NMP' claimed path 'vmhba1:C0:T1:L2' 
Jul 14 09:11:06 esxsvr10 vmkernel: 0:00:01:10.333 cpu4:4111)ScsiPath: 3839: Plugin 'NMP' claimed path 'vmhba1:C0:T0:L16' 
Jul 14 09:11:06 esxsvr10 vmkernel: 0:00:01:10.334 cpu4:4111)ScsiPath: 3839: Plugin 'NMP' claimed path 'vmhba1:C0:T1:L3' 
Jul 14 09:11:06 esxsvr10 vmkernel: 0:00:01:10.334 cpu4:4111)ScsiPath: 3839: Plugin 'NMP' claimed path 'vmhba1:C0:T0:L17' 
Jul 14 09:11:06 esxsvr10 vmkernel: 0:00:01:10.334 cpu4:4111)ScsiPath: 3839: Plugin 'NMP' claimed path 'vmhba1:C0:T1:L4' 
Jul 14 09:11:06 esxsvr10 vmkernel: 0:00:01:10.334 cpu4:4111)ScsiPath: 3839: Plugin 'NMP' claimed path 'vmhba1:C0:T0:L18'

Open in new window


Thanks
Joseph BrownAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
from the logs it looks like some sort of failed path, are you using iSCSI connection to a SAN ?

or local datastore issue ?

iSCSI polling missing or remove datastores, can cause some versions of ESXi to lock-up.

What is the SAN?

What is the build version of ESXi ?
0
Joseph BrownAuthor Commented:
It's esx 4.1 and the san is an EMC2
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Have you removed any iSCSI LUNs ?
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

Joseph BrownAuthor Commented:
None, we are in the process of upgrading our SAN to an Aberdeen, but we haven't started the migration process yet.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Have you checked for any missing paths or datastores?
0
Joseph BrownAuthor Commented:
Thanks for your help, after digging into the logs I figured out that all of the hosts are having these same errors and all the errors are from our 3rd SAN. We had issues with the controller on that SAN and had it replaced when this issue first started, but I don't think replacing the controller fixed it. Anyway, the other question I have is once I get everything moved over to the new SAN I will need to remove the datastores from VCenter and the hosts, this article seems to go into pretty well Vmware Article, the only thing I'm not sure about is if the EMC san is target based or Lun based, I'm leaning towards target based, just want to make sure. This is what I'm getting returned.
# esxcfg-scsidevs --vmfs | grep -i cx700x3-lun0
naa.60060160a31917005ea1954bd6e3e011:1                           /dev/sdav1           4e81f347-d2c51c98-9ce5-0019b9b598dd  0  CX700x3-Lun0
# esxcfg-mpath -L | grep -i naa.60060160a31917005ea1954bd6e3e011
vmhba2:C0:T8:L0  state:active  naa.60060160a31917005ea1954bd6e3e011 vmhba2 0 8 0 NMP active san fc.200000e08b04085e:210000e08b04085e fc.50060160b06039ca:50060162306039ca
vmhba2:C0:T11:L0 state:standby naa.60060160a31917005ea1954bd6e3e011 vmhba2 0 11 0 NMP standby san fc.200000e08b04085e:210000e08b04085e fc.50060160b06039ca:5006016b306039ca
vmhba1:C0:T9:L0  state:active  naa.60060160a31917005ea1954bd6e3e011 vmhba1 0 9 0 NMP active san fc.200000e08b041c60:210000e08b041c60 fc.50060160b06039ca:50060161306039ca
vmhba1:C0:T11:L0 state:standby naa.60060160a31917005ea1954bd6e3e011 vmhba1 0 11 0 NMP standby san fc.200000e08b041c60:210000e08b041c60 fc.50060160b06039ca:50060169306039ca
vmhba2:C0:T10:L0 state:standby naa.60060160a31917005ea1954bd6e3e011 vmhba2 0 10 0 NMP standby san fc.200000e08b04085e:210000e08b04085e fc.50060160b06039ca:5006016a306039ca
vmhba1:C0:T8:L0  state:active  naa.60060160a31917005ea1954bd6e3e011 vmhba1 0 8 0 NMP active san fc.200000e08b041c60:210000e08b041c60 fc.50060160b06039ca:50060160306039ca
vmhba1:C0:T10:L0 state:standby naa.60060160a31917005ea1954bd6e3e011 vmhba1 0 10 0 NMP standby san fc.200000e08b041c60:210000e08b041c60 fc.50060160b06039ca:50060168306039ca
vmhba2:C0:T9:L0  state:active  naa.60060160a31917005ea1954bd6e3e011 vmhba2 0 9 0 NMP active san fc.200000e08b04085e:210000e08b04085e fc.50060160b06039ca:50060163306039ca

Open in new window

0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Right click the datastore delete/unmount.
0
Joseph BrownAuthor Commented:
That's it? No need to delete the paths from the hosts?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Once, you have unmounted, then you can remove the LUNs on the SAN.

Re-scan HBA
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Joseph BrownAuthor Commented:
Thanks!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.