• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1481
  • Last Modified:

vmware 5.1 host stop working

I have a vmware host stopped working in the last weekend. All vms running on this host died and we had to reboot it.

Checked the vmkwarning.log file and found the following records:

2015-01-11T15:12:33.381Z cpu48:16459)WARNING: HBX: 2829: Reclaiming timed out [HB state abcdef02 offset 3858432 gen 107 stampUS 1025031605909 uuid 54a2ae4d-0d306f84-b658-d89d672992f0 jrnl <FB 1617360> drv 14.
2015-01-11T15:12:42.343Z cpu47:19603)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T15:12:45.043Z cpu19:16403)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60080e5000296ee80000055953e0ecb6" state in doubt; requested fast path state update...
2015-01-11T15:12:50.943Z cpu45:912496)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T15:12:53.345Z cpu51:19794)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T15:13:02.342Z cpu39:19602)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T15:13:05.042Z cpu17:16401)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60080e5000296ee80000055953e0ecb6" state in doubt; requested fast path state update...
2015-01-11T15:13:10.942Z cpu47:19687)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T15:13:13.344Z cpu57:19629)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T15:13:22.541Z cpu36:19501)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60080e5000296ee80000055953e0ecb6" state in doubt; requested fast path state update...
2015-01-11T15:13:25.241Z cpu21:16405)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T15:13:31.141Z cpu43:19690)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e44" state in doubt; requested fast path state update...
2015-01-11T15:13:33.541Z cpu61:19790)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e34" state in doubt; requested fast path state update...
2015-01-11T15:13:42.540Z cpu43:19684)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60080e5000296ee80000055953e0ecb6" state in doubt; requested fast path state update...
2015-01-11T15:13:45.240Z cpu22:16406)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T15:13:51.139Z cpu40:19602)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e44" state in doubt; requested fast path state update...
2015-01-11T15:13:53.539Z cpu63:19629)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e34" state in doubt; requested fast path state update...
2015-01-11T15:14:02.739Z cpu45:19603)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e36" state in doubt; requested fast path state update...
2015-01-11T15:14:05.239Z cpu17:19327)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T15:14:11.340Z cpu35:19688)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d77" state in doubt; requested fast path state update...
2015-01-11T15:14:13.738Z cpu53:22347)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e4e" state in doubt; requested fast path state update...
2015-01-11T15:14:22.738Z cpu37:19689)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e36" state in doubt; requested fast path state update...
2015-01-11T15:14:25.238Z cpu17:19769)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T15:14:31.340Z cpu39:22374)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d77" state in doubt; requested fast path state update...
2015-01-11T15:14:33.738Z cpu57:19631)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e4e" state in doubt; requested fast path state update...
2015-01-11T15:14:42.937Z cpu45:19600)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d32" state in doubt; requested fast path state update...
2015-01-11T15:14:45.237Z cpu23:19327)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T15:14:51.537Z cpu39:19598)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566869" state in doubt; requested fast path state update...
2015-01-11T15:14:53.937Z cpu59:19508)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686f" state in doubt; requested fast path state update...
2015-01-11T15:15:02.936Z cpu44:19601)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d32" state in doubt; requested fast path state update...
2015-01-11T15:15:05.236Z cpu28:19765)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T15:15:11.536Z cpu43:19684)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566869" state in doubt; requested fast path state update...
2015-01-11T15:15:13.936Z cpu49:19629)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686f" state in doubt; requested fast path state update...
2015-01-11T15:15:15.087Z cpu34:910982)WARNING: ScsiPath: 3836: Failed to issue command 0x12 (cmdSN 0x0) on path vmhba3:C0:T1:L200: Timeout
2015-01-11T15:15:23.135Z cpu35:19684)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686d" state in doubt; requested fast path state update...
2015-01-11T15:15:25.235Z cpu16:19493)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T15:15:31.735Z cpu38:19685)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d79" state in doubt; requested fast path state update...
2015-01-11T15:15:34.135Z cpu63:19628)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e42" state in doubt; requested fast path state update...
2015-01-11T15:15:43.134Z cpu40:22374)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686d" state in doubt; requested fast path state update...
2015-01-11T15:15:45.235Z cpu19:19816)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T15:15:51.734Z cpu36:19686)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d79" state in doubt; requested fast path state update...
2015-01-11T15:15:54.134Z cpu51:19790)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e42" state in doubt; requested fast path state update...
2015-01-11T15:15:55.088Z cpu32:910982)WARNING: ScsiPath: 3836: Failed to issue command 0x12 (cmdSN 0x0) on path vmhba3:C0:T1:L201: Timeout
2015-01-11T15:16:03.336Z cpu43:19687)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T15:16:05.434Z cpu17:16401)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T15:16:11.933Z cpu46:19502)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T15:16:14.433Z cpu48:16432)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T15:16:23.334Z cpu41:19688)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T15:16:31.932Z cpu39:19601)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T15:16:43.532Z cpu38:19601)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60080e5000296ee80000055953e0ecb6" state in doubt; requested fast path state update...
2015-01-11T15:16:51.932Z cpu41:19684)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e44" state in doubt; requested fast path state update...
2015-01-11T15:16:54.432Z cpu51:19792)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e34" state in doubt; requested fast path state update...
2015-01-11T15:17:03.532Z cpu41:19600)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60080e5000296ee80000055953e0ecb6" state in doubt; requested fast path state update...
2015-01-11T15:17:05.632Z cpu23:16407)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T15:17:11.931Z cpu46:19685)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e44" state in doubt; requested fast path state update...
2015-01-11T15:17:14.432Z cpu56:19791)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e34" state in doubt; requested fast path state update...

............


2015-01-11T16:19:04.280Z cpu47:19688)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d79" state in doubt; requested fast path state update...
2015-01-11T16:19:08.180Z cpu34:19686)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e42" state in doubt; requested fast path state update...
2015-01-11T16:19:19.081Z cpu17:16401)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e36" state in doubt; requested fast path state update...
2015-01-11T16:19:19.679Z cpu33:19688)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686d" state in doubt; requested fast path state update...
2015-01-11T16:19:24.279Z cpu39:19682)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d79" state in doubt; requested fast path state update...
2015-01-11T16:19:28.178Z cpu37:22374)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e42" state in doubt; requested fast path state update...
2015-01-11T16:19:28.451Z cpu32:16459)WARNING: HBX: 2829: Reclaiming timed out [HB state abcdef02 offset 3858432 gen 107 stampUS 1045284869044 uuid 54a2ae4d-0d306f84-b658-d89d672992f0 jrnl <FB 1617360> drv 14.
2015-01-11T16:19:35.828Z cpu32:910982)WARNING: ScsiPath: 3836: Failed to issue command 0x12 (cmdSN 0x0) on path vmhba3:C0:T2:L58: Timeout
2015-01-11T16:19:39.080Z cpu21:16405)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e36" state in doubt; requested fast path state update...
2015-01-11T16:19:39.878Z cpu44:19684)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T16:19:44.277Z cpu39:19688)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T16:19:48.377Z cpu33:19603)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T16:19:59.078Z cpu24:19765)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e36" state in doubt; requested fast path state update...
2015-01-11T16:19:59.876Z cpu41:22361)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T16:20:04.276Z cpu39:914578)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T16:20:08.375Z cpu33:19684)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T16:20:19.076Z cpu26:16410)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e46" state in doubt; requested fast path state update...
2015-01-11T16:20:19.874Z cpu47:19687)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60080e5000296ee80000055953e0ecb6" state in doubt; requested fast path state update...
2015-01-11T16:20:24.474Z cpu43:22361)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e44" state in doubt; requested fast path state update...
2015-01-11T16:20:28.574Z cpu45:19598)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e34" state in doubt; requested fast path state update...
2015-01-11T16:20:39.075Z cpu16:16400)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e46" state in doubt; requested fast path state update...
2015-01-11T16:20:39.873Z cpu41:19602)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60080e5000296ee80000055953e0ecb6" state in doubt; requested fast path state update...
2015-01-11T16:20:44.472Z cpu43:19601)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e44" state in doubt; requested fast path state update...
2015-01-11T16:20:48.572Z cpu43:19604)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e34" state in doubt; requested fast path state update...
2015-01-11T16:20:59.271Z cpu16:19816)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e46" state in doubt; requested fast path state update...
2015-01-11T16:21:00.071Z cpu33:18500)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e36" state in doubt; requested fast path state update...
2015-01-11T16:21:04.671Z cpu43:16427)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d77" state in doubt; requested fast path state update...
2015-01-11T16:21:08.871Z cpu33:19601)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e4e" state in doubt; requested fast path state update...
2015-01-11T16:21:19.270Z cpu23:19768)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e46" state in doubt; requested fast path state update...
2015-01-11T16:21:20.070Z cpu45:19500)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e36" state in doubt; requested fast path state update...
2015-01-11T16:21:24.669Z cpu45:19686)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d77" state in doubt; requested fast path state update...
2015-01-11T16:21:28.869Z cpu37:16421)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e4e" state in doubt; requested fast path state update...
2015-01-11T16:21:40.268Z cpu35:19688)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d32" state in doubt; requested fast path state update...
2015-01-11T16:21:44.668Z cpu45:19604)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566869" state in doubt; requested fast path state update...
2015-01-11T16:21:48.867Z cpu39:22371)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686f" state in doubt; requested fast path state update...
2015-01-11T16:22:00.266Z cpu45:19689)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d32" state in doubt; requested fast path state update...
2015-01-11T16:22:04.666Z cpu32:16416)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566869" state in doubt; requested fast path state update...
2015-01-11T16:22:08.866Z cpu39:19685)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686f" state in doubt; requested fast path state update...
2015-01-11T16:22:17.215Z cpu33:910982)WARNING: ScsiPath: 3836: Failed to issue command 0x12 (cmdSN 0x0) on path vmhba3:C0:T2:L62: Timeout
2015-01-11T16:22:19.665Z cpu16:16400)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d77" state in doubt; requested fast path state update...
2015-01-11T16:22:20.265Z cpu35:22374)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686d" state in doubt; requested fast path state update...
2015-01-11T16:22:24.865Z cpu35:22359)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d79" state in doubt; requested fast path state update...
2015-01-11T16:22:29.066Z cpu45:19600)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e42" state in doubt; requested fast path state update...
2015-01-11T16:22:39.663Z cpu16:16400)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d77" state in doubt; requested fast path state update...
2015-01-11T16:22:40.263Z cpu35:19601)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686d" state in doubt; requested fast path state update...
2015-01-11T16:22:41.152Z cpu23:928320)WARNING: UserLinux: 1331: unsupported: (void)
2015-01-11T16:22:44.863Z cpu44:19687)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d79" state in doubt; requested fast path state update...
2015-01-11T16:22:49.065Z cpu46:19601)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e42" state in doubt; requested fast path state update...
2015-01-11T16:22:59.862Z cpu21:16405)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686d" state in doubt; requested fast path state update...
2015-01-11T16:23:00.462Z cpu37:19687)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T16:23:05.161Z cpu47:19688)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T16:23:09.261Z cpu39:19347)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T16:23:19.860Z cpu21:16405)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686d" state in doubt; requested fast path state update...
2015-01-11T16:23:20.460Z cpu33:19600)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T16:23:25.160Z cpu39:19603)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T16:23:29.259Z cpu42:16426)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T16:23:40.059Z cpu18:19816)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686d" state in doubt; requested fast path state update...


Please help for finding the root reason for this failure.

thanks.
0
Jason Yu
Asked:
Jason Yu
  • 7
  • 6
6 Solutions
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Can you explain what you mean by died ?

e.g. could not ping VMs, could not ping Host, could not RDP to VMs ?

What is the server, make and model?

Shared Storage ?

CPU and Memory ?

Number of VMs ?
0
 
Jason YuAuthor Commented:
Please see the following information from my coworker:

Found XXX VM host is in a state of disconnect. I connected it but some VM servers are not reachable or not responding.  Although some VM servers are working but not stable. For example I did a dir on \\Carmel\scanjobs and it got stuck half way showing the list.
 
Need your permission to shutdown VMs, reboot XXX, and boot up VMs.


I tried to connect to the vmware host by ssh and reboot there, it stucked in the middle of reboot. In the end, I had to use ilo management interface hard reboot the whole server. It took me several hours to bring back this server last sunday. Called vmware support and they said our storage has high latency.

their commands used are as follows:

esxcfg-nics -l
less vmkernel.log
esxcfg-nics –l
less vmkernel.log
ethtool -i vmnic0
vmkchedev -l |grep -i vmnic
vmkchdev -l |grep -i vmnic
Vmware -vl
less vmkernel.log |grep -i rx


I am not quite satisfied with this explaination, want to get some help here.
vmware-host-down.png
0
 
Jason YuAuthor Commented:
e.g. could not ping VMs, could not ping Host, could not RDP to VMs ?  

When the failure happened, I could ping the host, couldn't RDP to vms.
---------------------------
What is the server, make and model?
It's a pretty new HP ProLiant DL 560  G8 server.
-------------------
Shared Storage ?
Yes, shared storage on the netapp SAN

--------------------------
CPU and Memory ?
32 CPUs X 2.693 GHz
Number of VMs ?
22
0
Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
did you just reboot the ESXi host, did you manage to get the logs off before a reboot ?

a few questions

are you using the HP OEM versions of ESXi 5.5 ?

latest version of ESXi ?

What technology are you using to access the NetApp SAN, iSCSI or NFS?

Are you also using the recommending settings as applied by NetApp VSC ?

Multipath iSCSI and/or NFS ?

Jumbo Frames?

Can you repeat the issue ?

Do you know the amount of memory in use?

Total Memory in Host ?
0
 
Jason YuAuthor Commented:
we are using ESXi 5.1.0 1065491 from vmware company.  We didn't use the HP OEM one.

It's not the latest version of ESXi, do I need upgrade it?

We use iscsi to access the NetApp.

Not sure what VSC is, I just joined this company.

Yes, multipathing on iscsi.

Jumbo Frames: yes

The issue was the server stopped working all of sudden, we even couldn't reboot is nicely. We had to hard reboot it.

The total memory for the host is 1.5 TB.

Only 265 G is being used now.
0
 
Jason YuAuthor Commented:
Here is the copy screen of the server config.


from the "Image profile" it shows "HP-ESXi-5.1.0-standard-iso, does this mean this installation was from a HP version of wmware?

I am pretty new in this company, please advise. thank you.
vmware-host-failure.png
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Has this just occurred once?

You have iSCSI multipath enabled?

Do you have logging enabled so the logs are recorded off the host?

Do you have performance charts before the crash?

It does look like storage related which if iSCSI can lock the host up.

The host would then just start polling for ever and cause delays to everything on the host.

Is this just one massive host? No cluster
0
 
Jason YuAuthor Commented:
It looks like this host is not inside a cluster, I am attaching a copy screen, please verify it for me.


How to check if logging is enabled or not?

We don't have performance charts before the crash. I have the vmkwarning log file uploaded for you.
vmkwarning.log
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
The best log is the vmkernel.log the issue is rebooting the host this log can be lost or overwritten.

This is just one single ESXi server?

Are all firmwares on server updated and also update ESXi to 5.1 U3

Has this just happenedvonce?
0
 
Jason YuAuthor Commented:
yes, I am planning to upgrade the ESXi to 5.1 U3, do you have some action plan to follow/

thank you.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
is this a single server, with no other hosts to migrate VMs to ?
0
 
Jason YuAuthor Commented:
Currently, this vmware host is not in any cluster yet. We plan to add it into a cluster and vmotion all the vms from this host to the new added host, then do an upgrade. After the upgrade is done, we will move back those vms.

Is this procedure OK?

I had done similar project before. If you can provide documentation for upgrading vmware host, please advise.

thanks.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
The reason I ask, is because yo update you've got to Enter Maintenance Mode, which means TURN OFF all the VMs.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

  • 7
  • 6
Tackle projects and never again get stuck behind a technical roadblock.
Join Now