vmware 5.1 host stop working
I have a vmware host stopped working in the last weekend. All vms running on this host died and we had to reboot it.
Checked the vmkwarning.log file and found the following records:
2015-01-11T15:12:33.381Z cpu48:16459)WARNING: HBX: 2829: Reclaiming timed out [HB state abcdef02 offset 3858432 gen 107 stampUS 1025031605909 uuid 54a2ae4d-0d306f84-b658-d89
2015-01-11T15:12:42.343Z cpu47:19603)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:12:45.043Z cpu19:16403)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:12:50.943Z cpu45:912496)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:12:53.345Z cpu51:19794)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:02.342Z cpu39:19602)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:05.042Z cpu17:16401)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:10.942Z cpu47:19687)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:13.344Z cpu57:19629)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:22.541Z cpu36:19501)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:25.241Z cpu21:16405)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:31.141Z cpu43:19690)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:33.541Z cpu61:19790)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:42.540Z cpu43:19684)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:45.240Z cpu22:16406)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:51.139Z cpu40:19602)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:53.539Z cpu63:19629)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:02.739Z cpu45:19603)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:05.239Z cpu17:19327)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:11.340Z cpu35:19688)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:13.738Z cpu53:22347)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:22.738Z cpu37:19689)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:25.238Z cpu17:19769)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:31.340Z cpu39:22374)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:33.738Z cpu57:19631)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:42.937Z cpu45:19600)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:45.237Z cpu23:19327)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:51.537Z cpu39:19598)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:53.937Z cpu59:19508)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:02.936Z cpu44:19601)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:05.236Z cpu28:19765)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:11.536Z cpu43:19684)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:13.936Z cpu49:19629)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:15.087Z cpu34:910982)WARNING: ScsiPath: 3836: Failed to issue command 0x12 (cmdSN 0x0) on path vmhba3:C0:T1:L200: Timeout
2015-01-11T15:15:23.135Z cpu35:19684)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:25.235Z cpu16:19493)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:31.735Z cpu38:19685)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:34.135Z cpu63:19628)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:43.134Z cpu40:22374)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:45.235Z cpu19:19816)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:51.734Z cpu36:19686)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:54.134Z cpu51:19790)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:55.088Z cpu32:910982)WARNING: ScsiPath: 3836: Failed to issue command 0x12 (cmdSN 0x0) on path vmhba3:C0:T1:L201: Timeout
2015-01-11T15:16:03.336Z cpu43:19687)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:16:05.434Z cpu17:16401)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:16:11.933Z cpu46:19502)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:16:14.433Z cpu48:16432)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:16:23.334Z cpu41:19688)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:16:31.932Z cpu39:19601)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:16:43.532Z cpu38:19601)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:16:51.932Z cpu41:19684)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:16:54.432Z cpu51:19792)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:17:03.532Z cpu41:19600)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:17:05.632Z cpu23:16407)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:17:11.931Z cpu46:19685)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:17:14.432Z cpu56:19791)WARNING: NMP: nmp_DeviceRequestFastDevic
............
2015-01-11T16:19:04.280Z cpu47:19688)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:08.180Z cpu34:19686)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:19.081Z cpu17:16401)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:19.679Z cpu33:19688)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:24.279Z cpu39:19682)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:28.178Z cpu37:22374)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:28.451Z cpu32:16459)WARNING: HBX: 2829: Reclaiming timed out [HB state abcdef02 offset 3858432 gen 107 stampUS 1045284869044 uuid 54a2ae4d-0d306f84-b658-d89
2015-01-11T16:19:35.828Z cpu32:910982)WARNING: ScsiPath: 3836: Failed to issue command 0x12 (cmdSN 0x0) on path vmhba3:C0:T2:L58: Timeout
2015-01-11T16:19:39.080Z cpu21:16405)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:39.878Z cpu44:19684)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:44.277Z cpu39:19688)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:48.377Z cpu33:19603)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:59.078Z cpu24:19765)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:59.876Z cpu41:22361)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:04.276Z cpu39:914578)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:08.375Z cpu33:19684)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:19.076Z cpu26:16410)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:19.874Z cpu47:19687)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:24.474Z cpu43:22361)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:28.574Z cpu45:19598)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:39.075Z cpu16:16400)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:39.873Z cpu41:19602)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:44.472Z cpu43:19601)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:48.572Z cpu43:19604)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:59.271Z cpu16:19816)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:00.071Z cpu33:18500)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:04.671Z cpu43:16427)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:08.871Z cpu33:19601)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:19.270Z cpu23:19768)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:20.070Z cpu45:19500)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:24.669Z cpu45:19686)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:28.869Z cpu37:16421)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:40.268Z cpu35:19688)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:44.668Z cpu45:19604)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:48.867Z cpu39:22371)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:00.266Z cpu45:19689)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:04.666Z cpu32:16416)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:08.866Z cpu39:19685)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:17.215Z cpu33:910982)WARNING: ScsiPath: 3836: Failed to issue command 0x12 (cmdSN 0x0) on path vmhba3:C0:T2:L62: Timeout
2015-01-11T16:22:19.665Z cpu16:16400)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:20.265Z cpu35:22374)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:24.865Z cpu35:22359)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:29.066Z cpu45:19600)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:39.663Z cpu16:16400)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:40.263Z cpu35:19601)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:41.152Z cpu23:928320)WARNING: UserLinux: 1331: unsupported: (void)
2015-01-11T16:22:44.863Z cpu44:19687)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:49.065Z cpu46:19601)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:59.862Z cpu21:16405)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:23:00.462Z cpu37:19687)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:23:05.161Z cpu47:19688)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:23:09.261Z cpu39:19347)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:23:19.860Z cpu21:16405)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:23:20.460Z cpu33:19600)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:23:25.160Z cpu39:19603)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:23:29.259Z cpu42:16426)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:23:40.059Z cpu18:19816)WARNING: NMP: nmp_DeviceRequestFastDevic
Please help for finding the root reason for this failure.
thanks.
Checked the vmkwarning.log file and found the following records:
2015-01-11T15:12:33.381Z cpu48:16459)WARNING: HBX: 2829: Reclaiming timed out [HB state abcdef02 offset 3858432 gen 107 stampUS 1025031605909 uuid 54a2ae4d-0d306f84-b658-d89
2015-01-11T15:12:42.343Z cpu47:19603)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:12:45.043Z cpu19:16403)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:12:50.943Z cpu45:912496)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:12:53.345Z cpu51:19794)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:02.342Z cpu39:19602)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:05.042Z cpu17:16401)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:10.942Z cpu47:19687)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:13.344Z cpu57:19629)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:22.541Z cpu36:19501)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:25.241Z cpu21:16405)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:31.141Z cpu43:19690)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:33.541Z cpu61:19790)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:42.540Z cpu43:19684)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:45.240Z cpu22:16406)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:51.139Z cpu40:19602)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:13:53.539Z cpu63:19629)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:02.739Z cpu45:19603)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:05.239Z cpu17:19327)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:11.340Z cpu35:19688)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:13.738Z cpu53:22347)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:22.738Z cpu37:19689)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:25.238Z cpu17:19769)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:31.340Z cpu39:22374)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:33.738Z cpu57:19631)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:42.937Z cpu45:19600)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:45.237Z cpu23:19327)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:51.537Z cpu39:19598)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:14:53.937Z cpu59:19508)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:02.936Z cpu44:19601)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:05.236Z cpu28:19765)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:11.536Z cpu43:19684)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:13.936Z cpu49:19629)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:15.087Z cpu34:910982)WARNING: ScsiPath: 3836: Failed to issue command 0x12 (cmdSN 0x0) on path vmhba3:C0:T1:L200: Timeout
2015-01-11T15:15:23.135Z cpu35:19684)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:25.235Z cpu16:19493)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:31.735Z cpu38:19685)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:34.135Z cpu63:19628)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:43.134Z cpu40:22374)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:45.235Z cpu19:19816)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:51.734Z cpu36:19686)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:54.134Z cpu51:19790)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:15:55.088Z cpu32:910982)WARNING: ScsiPath: 3836: Failed to issue command 0x12 (cmdSN 0x0) on path vmhba3:C0:T1:L201: Timeout
2015-01-11T15:16:03.336Z cpu43:19687)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:16:05.434Z cpu17:16401)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:16:11.933Z cpu46:19502)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:16:14.433Z cpu48:16432)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:16:23.334Z cpu41:19688)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:16:31.932Z cpu39:19601)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:16:43.532Z cpu38:19601)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:16:51.932Z cpu41:19684)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:16:54.432Z cpu51:19792)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:17:03.532Z cpu41:19600)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:17:05.632Z cpu23:16407)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:17:11.931Z cpu46:19685)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T15:17:14.432Z cpu56:19791)WARNING: NMP: nmp_DeviceRequestFastDevic
............
2015-01-11T16:19:04.280Z cpu47:19688)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:08.180Z cpu34:19686)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:19.081Z cpu17:16401)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:19.679Z cpu33:19688)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:24.279Z cpu39:19682)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:28.178Z cpu37:22374)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:28.451Z cpu32:16459)WARNING: HBX: 2829: Reclaiming timed out [HB state abcdef02 offset 3858432 gen 107 stampUS 1045284869044 uuid 54a2ae4d-0d306f84-b658-d89
2015-01-11T16:19:35.828Z cpu32:910982)WARNING: ScsiPath: 3836: Failed to issue command 0x12 (cmdSN 0x0) on path vmhba3:C0:T2:L58: Timeout
2015-01-11T16:19:39.080Z cpu21:16405)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:39.878Z cpu44:19684)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:44.277Z cpu39:19688)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:48.377Z cpu33:19603)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:59.078Z cpu24:19765)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:19:59.876Z cpu41:22361)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:04.276Z cpu39:914578)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:08.375Z cpu33:19684)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:19.076Z cpu26:16410)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:19.874Z cpu47:19687)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:24.474Z cpu43:22361)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:28.574Z cpu45:19598)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:39.075Z cpu16:16400)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:39.873Z cpu41:19602)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:44.472Z cpu43:19601)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:48.572Z cpu43:19604)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:20:59.271Z cpu16:19816)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:00.071Z cpu33:18500)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:04.671Z cpu43:16427)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:08.871Z cpu33:19601)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:19.270Z cpu23:19768)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:20.070Z cpu45:19500)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:24.669Z cpu45:19686)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:28.869Z cpu37:16421)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:40.268Z cpu35:19688)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:44.668Z cpu45:19604)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:21:48.867Z cpu39:22371)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:00.266Z cpu45:19689)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:04.666Z cpu32:16416)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:08.866Z cpu39:19685)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:17.215Z cpu33:910982)WARNING: ScsiPath: 3836: Failed to issue command 0x12 (cmdSN 0x0) on path vmhba3:C0:T2:L62: Timeout
2015-01-11T16:22:19.665Z cpu16:16400)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:20.265Z cpu35:22374)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:24.865Z cpu35:22359)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:29.066Z cpu45:19600)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:39.663Z cpu16:16400)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:40.263Z cpu35:19601)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:41.152Z cpu23:928320)WARNING: UserLinux: 1331: unsupported: (void)
2015-01-11T16:22:44.863Z cpu44:19687)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:49.065Z cpu46:19601)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:22:59.862Z cpu21:16405)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:23:00.462Z cpu37:19687)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:23:05.161Z cpu47:19688)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:23:09.261Z cpu39:19347)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:23:19.860Z cpu21:16405)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:23:20.460Z cpu33:19600)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:23:25.160Z cpu39:19603)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:23:29.259Z cpu42:16426)WARNING: NMP: nmp_DeviceRequestFastDevic
2015-01-11T16:23:40.059Z cpu18:19816)WARNING: NMP: nmp_DeviceRequestFastDevic
Please help for finding the root reason for this failure.
thanks.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
e.g. could not ping VMs, could not ping Host, could not RDP to VMs ?
When the failure happened, I could ping the host, couldn't RDP to vms.
--------------------------
What is the server, make and model?
It's a pretty new HP ProLiant DL 560 G8 server.
-------------------
Shared Storage ?
Yes, shared storage on the netapp SAN
--------------------------
CPU and Memory ?
32 CPUs X 2.693 GHz
Number of VMs ?
22
When the failure happened, I could ping the host, couldn't RDP to vms.
--------------------------
What is the server, make and model?
It's a pretty new HP ProLiant DL 560 G8 server.
-------------------
Shared Storage ?
Yes, shared storage on the netapp SAN
--------------------------
CPU and Memory ?
32 CPUs X 2.693 GHz
Number of VMs ?
22
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
we are using ESXi 5.1.0 1065491 from vmware company. We didn't use the HP OEM one.
It's not the latest version of ESXi, do I need upgrade it?
We use iscsi to access the NetApp.
Not sure what VSC is, I just joined this company.
Yes, multipathing on iscsi.
Jumbo Frames: yes
The issue was the server stopped working all of sudden, we even couldn't reboot is nicely. We had to hard reboot it.
The total memory for the host is 1.5 TB.
Only 265 G is being used now.
It's not the latest version of ESXi, do I need upgrade it?
We use iscsi to access the NetApp.
Not sure what VSC is, I just joined this company.
Yes, multipathing on iscsi.
Jumbo Frames: yes
The issue was the server stopped working all of sudden, we even couldn't reboot is nicely. We had to hard reboot it.
The total memory for the host is 1.5 TB.
Only 265 G is being used now.
ASKER
Here is the copy screen of the server config.
from the "Image profile" it shows "HP-ESXi-5.1.0-standard-is
I am pretty new in this company, please advise. thank you.
vmware-host-failure.png
from the "Image profile" it shows "HP-ESXi-5.1.0-standard-is
I am pretty new in this company, please advise. thank you.
vmware-host-failure.png
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
It looks like this host is not inside a cluster, I am attaching a copy screen, please verify it for me.
How to check if logging is enabled or not?
We don't have performance charts before the crash. I have the vmkwarning log file uploaded for you.
vmkwarning.log
How to check if logging is enabled or not?
We don't have performance charts before the crash. I have the vmkwarning log file uploaded for you.
vmkwarning.log
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
yes, I am planning to upgrade the ESXi to 5.1 U3, do you have some action plan to follow/
thank you.
thank you.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Currently, this vmware host is not in any cluster yet. We plan to add it into a cluster and vmotion all the vms from this host to the new added host, then do an upgrade. After the upgrade is done, we will move back those vms.
Is this procedure OK?
I had done similar project before. If you can provide documentation for upgrading vmware host, please advise.
thanks.
Is this procedure OK?
I had done similar project before. If you can provide documentation for upgrading vmware host, please advise.
thanks.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Found XXX VM host is in a state of disconnect. I connected it but some VM servers are not reachable or not responding. Although some VM servers are working but not stable. For example I did a dir on \\Carmel\scanjobs and it got stuck half way showing the list.
Need your permission to shutdown VMs, reboot XXX, and boot up VMs.
I tried to connect to the vmware host by ssh and reboot there, it stucked in the middle of reboot. In the end, I had to use ilo management interface hard reboot the whole server. It took me several hours to bring back this server last sunday. Called vmware support and they said our storage has high latency.
their commands used are as follows:
esxcfg-nics -l
less vmkernel.log
esxcfg-nics –l
less vmkernel.log
ethtool -i vmnic0
vmkchedev -l |grep -i vmnic
vmkchdev -l |grep -i vmnic
Vmware -vl
less vmkernel.log |grep -i rx
I am not quite satisfied with this explaination, want to get some help here.
vmware-host-down.png