Solved

vmware 5.1 host stop working

Posted on 2015-01-13
13
877 Views
Last Modified: 2015-01-26
I have a vmware host stopped working in the last weekend. All vms running on this host died and we had to reboot it.

Checked the vmkwarning.log file and found the following records:

2015-01-11T15:12:33.381Z cpu48:16459)WARNING: HBX: 2829: Reclaiming timed out [HB state abcdef02 offset 3858432 gen 107 stampUS 1025031605909 uuid 54a2ae4d-0d306f84-b658-d89d672992f0 jrnl <FB 1617360> drv 14.
2015-01-11T15:12:42.343Z cpu47:19603)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T15:12:45.043Z cpu19:16403)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60080e5000296ee80000055953e0ecb6" state in doubt; requested fast path state update...
2015-01-11T15:12:50.943Z cpu45:912496)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T15:12:53.345Z cpu51:19794)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T15:13:02.342Z cpu39:19602)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T15:13:05.042Z cpu17:16401)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60080e5000296ee80000055953e0ecb6" state in doubt; requested fast path state update...
2015-01-11T15:13:10.942Z cpu47:19687)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T15:13:13.344Z cpu57:19629)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T15:13:22.541Z cpu36:19501)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60080e5000296ee80000055953e0ecb6" state in doubt; requested fast path state update...
2015-01-11T15:13:25.241Z cpu21:16405)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T15:13:31.141Z cpu43:19690)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e44" state in doubt; requested fast path state update...
2015-01-11T15:13:33.541Z cpu61:19790)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e34" state in doubt; requested fast path state update...
2015-01-11T15:13:42.540Z cpu43:19684)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60080e5000296ee80000055953e0ecb6" state in doubt; requested fast path state update...
2015-01-11T15:13:45.240Z cpu22:16406)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T15:13:51.139Z cpu40:19602)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e44" state in doubt; requested fast path state update...
2015-01-11T15:13:53.539Z cpu63:19629)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e34" state in doubt; requested fast path state update...
2015-01-11T15:14:02.739Z cpu45:19603)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e36" state in doubt; requested fast path state update...
2015-01-11T15:14:05.239Z cpu17:19327)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T15:14:11.340Z cpu35:19688)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d77" state in doubt; requested fast path state update...
2015-01-11T15:14:13.738Z cpu53:22347)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e4e" state in doubt; requested fast path state update...
2015-01-11T15:14:22.738Z cpu37:19689)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e36" state in doubt; requested fast path state update...
2015-01-11T15:14:25.238Z cpu17:19769)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T15:14:31.340Z cpu39:22374)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d77" state in doubt; requested fast path state update...
2015-01-11T15:14:33.738Z cpu57:19631)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e4e" state in doubt; requested fast path state update...
2015-01-11T15:14:42.937Z cpu45:19600)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d32" state in doubt; requested fast path state update...
2015-01-11T15:14:45.237Z cpu23:19327)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T15:14:51.537Z cpu39:19598)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566869" state in doubt; requested fast path state update...
2015-01-11T15:14:53.937Z cpu59:19508)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686f" state in doubt; requested fast path state update...
2015-01-11T15:15:02.936Z cpu44:19601)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d32" state in doubt; requested fast path state update...
2015-01-11T15:15:05.236Z cpu28:19765)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T15:15:11.536Z cpu43:19684)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566869" state in doubt; requested fast path state update...
2015-01-11T15:15:13.936Z cpu49:19629)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686f" state in doubt; requested fast path state update...
2015-01-11T15:15:15.087Z cpu34:910982)WARNING: ScsiPath: 3836: Failed to issue command 0x12 (cmdSN 0x0) on path vmhba3:C0:T1:L200: Timeout
2015-01-11T15:15:23.135Z cpu35:19684)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686d" state in doubt; requested fast path state update...
2015-01-11T15:15:25.235Z cpu16:19493)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T15:15:31.735Z cpu38:19685)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d79" state in doubt; requested fast path state update...
2015-01-11T15:15:34.135Z cpu63:19628)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e42" state in doubt; requested fast path state update...
2015-01-11T15:15:43.134Z cpu40:22374)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686d" state in doubt; requested fast path state update...
2015-01-11T15:15:45.235Z cpu19:19816)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T15:15:51.734Z cpu36:19686)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d79" state in doubt; requested fast path state update...
2015-01-11T15:15:54.134Z cpu51:19790)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e42" state in doubt; requested fast path state update...
2015-01-11T15:15:55.088Z cpu32:910982)WARNING: ScsiPath: 3836: Failed to issue command 0x12 (cmdSN 0x0) on path vmhba3:C0:T1:L201: Timeout
2015-01-11T15:16:03.336Z cpu43:19687)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T15:16:05.434Z cpu17:16401)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T15:16:11.933Z cpu46:19502)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T15:16:14.433Z cpu48:16432)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T15:16:23.334Z cpu41:19688)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T15:16:31.932Z cpu39:19601)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T15:16:43.532Z cpu38:19601)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60080e5000296ee80000055953e0ecb6" state in doubt; requested fast path state update...
2015-01-11T15:16:51.932Z cpu41:19684)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e44" state in doubt; requested fast path state update...
2015-01-11T15:16:54.432Z cpu51:19792)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e34" state in doubt; requested fast path state update...
2015-01-11T15:17:03.532Z cpu41:19600)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60080e5000296ee80000055953e0ecb6" state in doubt; requested fast path state update...
2015-01-11T15:17:05.632Z cpu23:16407)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T15:17:11.931Z cpu46:19685)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e44" state in doubt; requested fast path state update...
2015-01-11T15:17:14.432Z cpu56:19791)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e34" state in doubt; requested fast path state update...

............


2015-01-11T16:19:04.280Z cpu47:19688)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d79" state in doubt; requested fast path state update...
2015-01-11T16:19:08.180Z cpu34:19686)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e42" state in doubt; requested fast path state update...
2015-01-11T16:19:19.081Z cpu17:16401)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e36" state in doubt; requested fast path state update...
2015-01-11T16:19:19.679Z cpu33:19688)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686d" state in doubt; requested fast path state update...
2015-01-11T16:19:24.279Z cpu39:19682)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d79" state in doubt; requested fast path state update...
2015-01-11T16:19:28.178Z cpu37:22374)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e42" state in doubt; requested fast path state update...
2015-01-11T16:19:28.451Z cpu32:16459)WARNING: HBX: 2829: Reclaiming timed out [HB state abcdef02 offset 3858432 gen 107 stampUS 1045284869044 uuid 54a2ae4d-0d306f84-b658-d89d672992f0 jrnl <FB 1617360> drv 14.
2015-01-11T16:19:35.828Z cpu32:910982)WARNING: ScsiPath: 3836: Failed to issue command 0x12 (cmdSN 0x0) on path vmhba3:C0:T2:L58: Timeout
2015-01-11T16:19:39.080Z cpu21:16405)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e36" state in doubt; requested fast path state update...
2015-01-11T16:19:39.878Z cpu44:19684)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T16:19:44.277Z cpu39:19688)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T16:19:48.377Z cpu33:19603)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T16:19:59.078Z cpu24:19765)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e36" state in doubt; requested fast path state update...
2015-01-11T16:19:59.876Z cpu41:22361)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T16:20:04.276Z cpu39:914578)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T16:20:08.375Z cpu33:19684)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T16:20:19.076Z cpu26:16410)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e46" state in doubt; requested fast path state update...
2015-01-11T16:20:19.874Z cpu47:19687)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60080e5000296ee80000055953e0ecb6" state in doubt; requested fast path state update...
2015-01-11T16:20:24.474Z cpu43:22361)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e44" state in doubt; requested fast path state update...
2015-01-11T16:20:28.574Z cpu45:19598)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e34" state in doubt; requested fast path state update...
2015-01-11T16:20:39.075Z cpu16:16400)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e46" state in doubt; requested fast path state update...
2015-01-11T16:20:39.873Z cpu41:19602)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60080e5000296ee80000055953e0ecb6" state in doubt; requested fast path state update...
2015-01-11T16:20:44.472Z cpu43:19601)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e44" state in doubt; requested fast path state update...
2015-01-11T16:20:48.572Z cpu43:19604)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e34" state in doubt; requested fast path state update...
2015-01-11T16:20:59.271Z cpu16:19816)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e46" state in doubt; requested fast path state update...
2015-01-11T16:21:00.071Z cpu33:18500)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e36" state in doubt; requested fast path state update...
2015-01-11T16:21:04.671Z cpu43:16427)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d77" state in doubt; requested fast path state update...
2015-01-11T16:21:08.871Z cpu33:19601)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e4e" state in doubt; requested fast path state update...
2015-01-11T16:21:19.270Z cpu23:19768)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e46" state in doubt; requested fast path state update...
2015-01-11T16:21:20.070Z cpu45:19500)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e36" state in doubt; requested fast path state update...
2015-01-11T16:21:24.669Z cpu45:19686)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d77" state in doubt; requested fast path state update...
2015-01-11T16:21:28.869Z cpu37:16421)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e4e" state in doubt; requested fast path state update...
2015-01-11T16:21:40.268Z cpu35:19688)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d32" state in doubt; requested fast path state update...
2015-01-11T16:21:44.668Z cpu45:19604)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566869" state in doubt; requested fast path state update...
2015-01-11T16:21:48.867Z cpu39:22371)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686f" state in doubt; requested fast path state update...
2015-01-11T16:22:00.266Z cpu45:19689)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d32" state in doubt; requested fast path state update...
2015-01-11T16:22:04.666Z cpu32:16416)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566869" state in doubt; requested fast path state update...
2015-01-11T16:22:08.866Z cpu39:19685)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686f" state in doubt; requested fast path state update...
2015-01-11T16:22:17.215Z cpu33:910982)WARNING: ScsiPath: 3836: Failed to issue command 0x12 (cmdSN 0x0) on path vmhba3:C0:T2:L62: Timeout
2015-01-11T16:22:19.665Z cpu16:16400)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d77" state in doubt; requested fast path state update...
2015-01-11T16:22:20.265Z cpu35:22374)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686d" state in doubt; requested fast path state update...
2015-01-11T16:22:24.865Z cpu35:22359)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d79" state in doubt; requested fast path state update...
2015-01-11T16:22:29.066Z cpu45:19600)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e42" state in doubt; requested fast path state update...
2015-01-11T16:22:39.663Z cpu16:16400)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d77" state in doubt; requested fast path state update...
2015-01-11T16:22:40.263Z cpu35:19601)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686d" state in doubt; requested fast path state update...
2015-01-11T16:22:41.152Z cpu23:928320)WARNING: UserLinux: 1331: unsupported: (void)
2015-01-11T16:22:44.863Z cpu44:19687)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484d79" state in doubt; requested fast path state update...
2015-01-11T16:22:49.065Z cpu46:19601)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e42" state in doubt; requested fast path state update...
2015-01-11T16:22:59.862Z cpu21:16405)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686d" state in doubt; requested fast path state update...
2015-01-11T16:23:00.462Z cpu37:19687)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T16:23:05.161Z cpu47:19688)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T16:23:09.261Z cpu39:19347)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T16:23:19.860Z cpu21:16405)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686d" state in doubt; requested fast path state update...
2015-01-11T16:23:20.460Z cpu33:19600)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566873" state in doubt; requested fast path state update...
2015-01-11T16:23:25.160Z cpu39:19603)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d437143566871" state in doubt; requested fast path state update...
2015-01-11T16:23:29.259Z cpu42:16426)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4c3124437574484e54" state in doubt; requested fast path state update...
2015-01-11T16:23:40.059Z cpu18:19816)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800044322d4b5a5d43714356686d" state in doubt; requested fast path state update...


Please help for finding the root reason for this failure.

thanks.
0
Comment
Question by:Jason Yu
  • 7
  • 6
13 Comments
 
LVL 118

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 40547547
Can you explain what you mean by died ?

e.g. could not ping VMs, could not ping Host, could not RDP to VMs ?

What is the server, make and model?

Shared Storage ?

CPU and Memory ?

Number of VMs ?
0
 

Author Comment

by:Jason Yu
ID: 40547718
Please see the following information from my coworker:

Found XXX VM host is in a state of disconnect. I connected it but some VM servers are not reachable or not responding.  Although some VM servers are working but not stable. For example I did a dir on \\Carmel\scanjobs and it got stuck half way showing the list.
 
Need your permission to shutdown VMs, reboot XXX, and boot up VMs.


I tried to connect to the vmware host by ssh and reboot there, it stucked in the middle of reboot. In the end, I had to use ilo management interface hard reboot the whole server. It took me several hours to bring back this server last sunday. Called vmware support and they said our storage has high latency.

their commands used are as follows:

esxcfg-nics -l
less vmkernel.log
esxcfg-nics –l
less vmkernel.log
ethtool -i vmnic0
vmkchedev -l |grep -i vmnic
vmkchdev -l |grep -i vmnic
Vmware -vl
less vmkernel.log |grep -i rx


I am not quite satisfied with this explaination, want to get some help here.
vmware-host-down.png
0
 

Author Comment

by:Jason Yu
ID: 40547806
e.g. could not ping VMs, could not ping Host, could not RDP to VMs ?  

When the failure happened, I could ping the host, couldn't RDP to vms.
---------------------------
What is the server, make and model?
It's a pretty new HP ProLiant DL 560  G8 server.
-------------------
Shared Storage ?
Yes, shared storage on the netapp SAN

--------------------------
CPU and Memory ?
32 CPUs X 2.693 GHz
Number of VMs ?
22
0
 
LVL 118

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 40547833
did you just reboot the ESXi host, did you manage to get the logs off before a reboot ?

a few questions

are you using the HP OEM versions of ESXi 5.5 ?

latest version of ESXi ?

What technology are you using to access the NetApp SAN, iSCSI or NFS?

Are you also using the recommending settings as applied by NetApp VSC ?

Multipath iSCSI and/or NFS ?

Jumbo Frames?

Can you repeat the issue ?

Do you know the amount of memory in use?

Total Memory in Host ?
0
 

Author Comment

by:Jason Yu
ID: 40547999
we are using ESXi 5.1.0 1065491 from vmware company.  We didn't use the HP OEM one.

It's not the latest version of ESXi, do I need upgrade it?

We use iscsi to access the NetApp.

Not sure what VSC is, I just joined this company.

Yes, multipathing on iscsi.

Jumbo Frames: yes

The issue was the server stopped working all of sudden, we even couldn't reboot is nicely. We had to hard reboot it.

The total memory for the host is 1.5 TB.

Only 265 G is being used now.
0
 

Author Comment

by:Jason Yu
ID: 40548001
Here is the copy screen of the server config.


from the "Image profile" it shows "HP-ESXi-5.1.0-standard-iso, does this mean this installation was from a HP version of wmware?

I am pretty new in this company, please advise. thank you.
vmware-host-failure.png
0
Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

 
LVL 118

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 40548021
Has this just occurred once?

You have iSCSI multipath enabled?

Do you have logging enabled so the logs are recorded off the host?

Do you have performance charts before the crash?

It does look like storage related which if iSCSI can lock the host up.

The host would then just start polling for ever and cause delays to everything on the host.

Is this just one massive host? No cluster
0
 

Author Comment

by:Jason Yu
ID: 40548028
It looks like this host is not inside a cluster, I am attaching a copy screen, please verify it for me.


How to check if logging is enabled or not?

We don't have performance charts before the crash. I have the vmkwarning log file uploaded for you.
vmkwarning.log
0
 
LVL 118

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 40548036
The best log is the vmkernel.log the issue is rebooting the host this log can be lost or overwritten.

This is just one single ESXi server?

Are all firmwares on server updated and also update ESXi to 5.1 U3

Has this just happenedvonce?
0
 

Author Comment

by:Jason Yu
ID: 40548287
yes, I am planning to upgrade the ESXi to 5.1 U3, do you have some action plan to follow/

thank you.
0
 
LVL 118

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 40548422
is this a single server, with no other hosts to migrate VMs to ?
0
 

Author Comment

by:Jason Yu
ID: 40549551
Currently, this vmware host is not in any cluster yet. We plan to add it into a cluster and vmotion all the vms from this host to the new added host, then do an upgrade. After the upgrade is done, we will move back those vms.

Is this procedure OK?

I had done similar project before. If you can provide documentation for upgrading vmware host, please advise.

thanks.
0
 
LVL 118

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 40549578
The reason I ask, is because yo update you've got to Enter Maintenance Mode, which means TURN OFF all the VMs.
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

Last article we focus in how to VMware: How to create and use VMs TAGs – Part 1 so before follow this article and perform the next tasks, you should read the first article how to create the TAG before using them in Veeam Backup Jobs.
This article will show you how to create an ISO CD-ROM/DVD-ROM image (*.iso), and MD5 checksum signature, for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5). It's a good idea to compare checksums, because many installations fail because of a corr…
This Micro Tutorial walks you through using a remote console to access a server and install ESXi 5.1. This example is showing remote access and installation using a Dell server. The hypervisor is the very first component of your virtual infrastructu…
How to install and configure Citrix XenApp 6.5 - Part 1. In this video tutorial we have explained step by step installation of Citrix XenApp 6.5 Server on Windows Server 2008 R2 is explained in this video. We have explained the difference between…

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now