Link to home
Start Free TrialLog in
Avatar of Akibia
Akibia

asked on

Help with troubleshooting a vSphere 4.1 Host

I have an ESX 4.1 Host that can not run more than 1 guest without becoming unstable. Prior to a software update on it's backend storage, it ran 31 guests without issue. I can start and manage 1 guest, though it does feel suffer from poor performance. With any additional guest, it gets slower and becomes unusable with frequent disconnects from vCenter and SSH (Putty).

I was hoping to not have to rebuild everything, does anyone have troubleshooting suggestions?

Thanks in advance!!!

(See below for system specifics)

Console output after 2 guests started:
Error interacting with configuration file /etc/vmware/esx.conf: Failed attempting to lock file. Another process has locked the file for more than 20 seconds. The process holding the lock is esx-init (11706). The operation will complete if it is run again after the lock is released.
Problem creating CIMqlogicProviderEnabled option.
Error interacting with configuration file /etc/vmware/esx.conf: Failed attempting to lock file. Another process has locked the file for more than 20 seconds. The process holding the lock is esx-init (11706). The operation will complete if it is run again after the lock is released.
Option with the name CIMemulexProviderEnabled already exists.


This is the outut of Top with one guest running:
top - 17:44:59 up 2 days, 17:15,  1 user,  load average: 0.21, 0.22, 1.16
Tasks: 106 total,   1 running, 105 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  7.4%sy,  0.0%ni, 91.6%id,  0.0%wa,  0.0%hi,  0.7%si,  0.0%st
Mem:    633264k total,   585612k used,    47652k free,   115260k buffers
Swap:  1293192k total,      340k used,  1292852k free,   271312k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5996 root      17   0 87984 6784 4200 S  6.3  1.1 137:41.83 dsm_sa_datamgr3
 5584 root      15   0 14988 1928 1452 S  0.7  0.3  21:36.29 vmware-usbarbit
11433 root      15   0 34516 1996 1452 R  0.7  0.3   0:06.62 top
    1 root      15   0 10344  640  544 S  0.0  0.1   0:18.31 init
    2 root      34  19     0    0    0 S  0.0  0.0   0:53.33 ksoftirqd/0
    3 root      10  -5     0    0    0 S  0.0  0.0   0:20.08 events/0
    4 root      10  -5     0    0    0 S  0.0  0.0   0:00.07 khelper
    5 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 kthread
    8 root      10  -5     0    0    0 S  0.0  0.0   1:15.43 kblockd/0
    9 root      18  -5     0    0    0 S  0.0  0.0   0:00.00 kseriod
  151 root      15   0     0    0    0 S  0.0  0.0   0:00.10 pdflush
  152 root      15   0     0    0    0 S  0.0  0.0  10:32.78 pdflush
  153 root      10  -5     0    0    0 S  0.0  0.0   0:04.78 kswapd0
  154 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0
  275 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 kpsmoused
  332 root      15   0     0    0    0 S  0.0  0.0   0:00.01 vmkmsgd-2
  333 root      15   0     0    0    0 S  0.0  0.0   3:55.25 vmkmsgd
  334 root      15   0     0    0    0 S  0.0  0.0   0:00.06 vmnixhbd
  341 root      15   0     0    0    0 S  0.0  0.0   7:03.25 vmkmsgd-7
  345 root      25   0     0    0    0 S  0.0  0.0   0:00.00 vmkdevd
  346 root      15   0     0    0    0 S  0.0  0.0   0:00.15 vmkregd
  347 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_0
  420 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_1
  430 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_2
  508 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_3
  510 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_4
  512 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_5
  514 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_6
  517 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_7
  734 root      10  -5     0    0    0 S  0.0  0.0   0:00.94 vmxnet_console
  844 root      10  -5     0    0    0 S  0.0  0.0  86:12.87 kjournald
  929 root      18  -4 34424 1088  392 S  0.0  0.2   0:00.39 udevd
 3618 root      10  -5     0    0    0 S  0.0  0.0   3:02.03 kjournald
 3620 root      10  -5     0    0    0 S  0.0  0.0   0:54.79 kjournald
 3622 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 kjournald
 3624 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 kjournald
 4321 root      20   0  9912 3596 2904 S  0.0  0.6   2:19.06 lsassd
 4340 root      25   0  7856  872  604 S  0.0  0.1   0:00.00 lwiod

This is top with 2 guests running:
top - 17:49:25 up 2 days, 17:19,  1 user,  load average: 3.00, 1.40, 1.38
Tasks: 107 total,   5 running, 102 sleeping,   0 stopped,   0 zombie
Cpu(s): 63.1%us, 34.9%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  1.9%si,  0.0%st
Mem:    633264k total,   588020k used,    45244k free,   115860k buffers
Swap:  1293192k total,      340k used,  1292852k free,   272008k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7198 root      15   0  138m  93m  48m S 63.4 15.2   1055:42 hostd-worker
 8552 root      15   0 16776 8468 2352 S 27.8  1.3 261:36.80 sfcb-vmware_raw
 7347 root      15   0 79884  34m  19m S  5.6  5.6 124:14.06 vpxa
11433 root      16   0 34516 1996 1452 R  3.3  0.3   0:11.75 top
13270 root       6 -10 15472 2040 1312 S  1.3  0.3   0:04.44 vmkload_app64
 7827 root      16   0 12388 3520  980 S  0.3  0.6  14:14.20 sfcb-vmware_int
    1 root      15   0 10344  640  544 S  0.0  0.1   0:18.31 init
    2 root      34  19     0    0    0 R  0.0  0.0   0:53.43 ksoftirqd/0
    3 root      10  -5     0    0    0 S  0.0  0.0   0:20.11 events/0
    4 root      10  -5     0    0    0 S  0.0  0.0   0:00.07 khelper
    5 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 kthread
    8 root      10  -5     0    0    0 S  0.0  0.0   1:15.44 kblockd/0
    9 root      18  -5     0    0    0 S  0.0  0.0   0:00.00 kseriod
  151 root      15   0     0    0    0 S  0.0  0.0   0:00.10 pdflush
  152 root      15   0     0    0    0 S  0.0  0.0  10:33.11 pdflush
  153 root      10  -5     0    0    0 S  0.0  0.0   0:04.78 kswapd0
  154 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0
  275 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 kpsmoused
  332 root      15   0     0    0    0 S  0.0  0.0   0:00.01 vmkmsgd-2
  333 root      15   0     0    0    0 S  0.0  0.0   3:55.46 vmkmsgd
  334 root      15   0     0    0    0 S  0.0  0.0   0:00.06 vmnixhbd
  341 root      15   0     0    0    0 S  0.0  0.0   7:03.57 vmkmsgd-7
  345 root      25   0     0    0    0 S  0.0  0.0   0:00.00 vmkdevd
  346 root      15   0     0    0    0 S  0.0  0.0   0:00.15 vmkregd
  347 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_0
  420 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_1
  430 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_2
  508 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_3
  510 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_4
  512 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_5
  514 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_6
  517 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_7
  734 root      10  -5     0    0    0 S  0.0  0.0   0:00.94 vmxnet_console
  844 root      10  -5     0    0    0 S  0.0  0.0  86:15.35 kjournald
  929 root      18  -4 34424 1088  392 S  0.0  0.2   0:00.39 udevd
 3618 root      10  -5     0    0    0 S  0.0  0.0   3:02.53 kjournald
 3620 root      10  -5     0    0    0 S  0.0  0.0   0:54.79 kjournald
 3622 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 kjournald


Host:
Dell PowerEdge R900
2 x 7350 @ 2.93GHz
81920 MB RAM
QLogic 4062c iscsi HBA
ESX 4.1.0, 348481

Storage:
Compellent (Single Controller & 1 ATA Enclosure with 16 1TB Drives)

To prep for the SAN update, I shutdown all guest vm's and put the Host into maintenance mode. Then shutdown. Processed the Compellent upgrade version 4.5.3 to 5.4.3.

Then turned ESX back on, took out of maintenance mode and checked/confirmed all datastores connected. Server was taken out of maintenance mode and vm's were started.

The 1st guest came online fine. All others appeared to be stuck at 95% in the Power-on task in our vCenter Server. Eventually... we're talking hours... the majority of them timed out and could not powereed on. I again put ESX in maintenance mode and then rebooted. Post looks healthy with no failures.
Avatar of Danny McDaniel
Danny McDaniel
Flag of United States of America image

Instead of top, you will want to use esxtop .  www.vmware.com/pdf/esx2_using_esxtop.pdf will get you started on the basic commands and measurements.  Post the errors if you need help researching them. You should also look at the compatibility guide for the storage to make sure that there aren't any known issues or problems with it and ESX 4.1.

In the service console, go to /var/log and look at the vmkwarning file(s).  They will have all of the warnings and errors from the vmkernel logs in the same directory.

I'm going to guess that you are having storage issues which are causing the mgmt processes to hang and that's why you see the host disconnect.  There is probably really high latency on storage commands so that the VM's are suffering as bad as they are.
VMware Communities: Interpreting esxtop Statistics - http://communities.vmware.com/docs/DOC-9279 should be of help, too.
Avatar of Akibia
Akibia

ASKER

Thanks for the info... This is a lot to digest. I will see what I can figure out and let you know. I'm posting info from vmkwarning and vmkernel logs as extra info. Perhaps if you can point me in the right dirction I'll be able to learn faster.

Also, all hardware appears to be on the HCL. There is a firmware update available for the qlogic HBA, but I'm not sure how to update it as of yet. I'll have to research that as well.

Thanks!

vmkwarning:
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.168 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000d. This will clear any SCSI-2 reservations on the device.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.198 cpu3:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000e. This will clear any SCSI-2 reservations on the device.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.220 cpu5:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000f. This will clear any SCSI-2 reservations on the device.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.242 cpu7:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000010. This will clear any SCSI-2 reservations on the device.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.265 cpu5:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000011. This will clear any SCSI-2 reservations on the device.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.289 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000012. This will clear any SCSI-2 reservations on the device.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.311 cpu7:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000013. This will clear any SCSI-2 reservations on the device.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.331 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000014. This will clear any SCSI-2 reservations on the device.
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.620 cpu7:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L33
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.622 cpu4:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L35
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.622 cpu2:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L34
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.623 cpu1:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L36
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.623 cpu3:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L37
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.623 cpu5:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L38
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.624 cpu4:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L35
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.624 cpu7:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L33
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.624 cpu1:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L40
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.624 cpu4:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L39
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.627 cpu4:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L38
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.627 cpu3:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L34
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.627 cpu2:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L37
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.627 cpu1:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L36
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.627 cpu7:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L33
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.627 cpu4:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L35
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.629 cpu1:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L40
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.629 cpu4:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L39
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.629 cpu4:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L38
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.630 cpu3:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L34
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.631 cpu3:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L37
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.632 cpu3:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L36
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.632 cpu2:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L39
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.633 cpu4:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L35
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.633 cpu6:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L33
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.634 cpu1:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L40
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.634 cpu4:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L38
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.634 cpu2:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L34
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.635 cpu3:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L37
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.635 cpu2:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L36
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.636 cpu3:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L39
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.638 cpu1:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L40
Mar 30 17:44:15 dr-esx1 vmkernel: 2:17:14:03.308 cpu7:4304)WARNING: MemSched: 12625: Non-overhead memory reservation for vmx user-world (worldID=4304) is greater than desired minimum amount of 57344 KB (min=57344 KB, reservedOverhead=0 KB, totalReserved=68844 KB)
Mar 30 17:44:18 dr-esx1 vmkernel: 2:17:14:06.027 cpu7:4304)WARNING: MemSched: vm 4304: 5199: Cannot reduce reservation by 5094 pages (total reservation: 57828 pages, consumed reservation: 53443 pages)
Mar 30 17:46:24 dr-esx1 vmkernel: 2:17:16:11.687 cpu1:4311)WARNING: MemSched: 12625: Non-overhead memory reservation for vmx user-world (worldID=4311) is greater than desired minimum amount of 57344 KB (min=57344 KB, reservedOverhead=0 KB, totalReserved=69220 KB)
Mar 30 17:46:31 dr-esx1 vmkernel: 2:17:16:18.917 cpu7:4311)WARNING: MemSched: vm 4311: 5199: Cannot reduce reservation by 5310 pages (total reservation: 58924 pages, consumed reservation: 53828 pages)
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.331 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000d. This will clear any SCSI-2 reservations on the device.
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.351 cpu7:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000e. This will clear any SCSI-2 reservations on the device.
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.372 cpu7:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000f. This will clear any SCSI-2 reservations on the device.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.394 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000010. This will clear any SCSI-2 reservations on the device.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.394 cpu6:4183)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa.6000d31000163c000000000000000008" state in doubt; requested fast path state update...
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.419 cpu2:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000011. This will clear any SCSI-2 reservations on the device.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.445 cpu1:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000012. This will clear any SCSI-2 reservations on the device.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.468 cpu5:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000013. This will clear any SCSI-2 reservations on the device.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.492 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000014. This will clear any SCSI-2 reservations on the device.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.538 cpu6:4102)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c00000000000000000d
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.541 cpu6:4102)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c00000000000000000e
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.546 cpu6:4102)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c00000000000000000f
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.552 cpu6:4112)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000010
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.559 cpu6:4102)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000011
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.565 cpu6:4102)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000012
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.572 cpu6:4102)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000013
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.601 cpu6:4102)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000014
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.036 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000d. This will clear any SCSI-2 reservations on the device.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.059 cpu7:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000e. This will clear any SCSI-2 reservations on the device.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.080 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000f. This will clear any SCSI-2 reservations on the device.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.104 cpu7:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000010. This will clear any SCSI-2 reservations on the device.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.126 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000011. This will clear any SCSI-2 reservations on the device.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.150 cpu7:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000012. This will clear any SCSI-2 reservations on the device.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.173 cpu7:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000013. This will clear any SCSI-2 reservations on the device.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.197 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000014. This will clear any SCSI-2 reservations on the device.
Mar 30 17:48:37 dr-esx1 vmkernel: 2:17:18:25.259 cpu3:4307)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c00000000000000000d
Mar 30 17:48:37 dr-esx1 vmkernel: 2:17:18:25.371 cpu3:4307)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c00000000000000000e
Mar 30 17:48:37 dr-esx1 vmkernel: 2:17:18:25.492 cpu3:4307)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c00000000000000000f
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:25.605 cpu3:4307)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000010
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:25.727 cpu3:4307)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000011
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:25.855 cpu3:4307)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000012
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:25.958 cpu3:4307)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000013
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:26.052 cpu3:4307)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000014
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.629 cpu4:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L33
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.630 cpu6:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L34
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.632 cpu4:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L35
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.632 cpu1:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L36
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.632 cpu7:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L38
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.633 cpu6:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L37
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.638 cpu4:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L33
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.641 cpu5:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L35
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.643 cpu0:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L36
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.643 cpu4:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L33
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.643 cpu5:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L40
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.644 cpu4:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L39
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.645 cpu5:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L35
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.646 cpu0:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L37
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.646 cpu5:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L40
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.651 cpu0:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L37
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.656 cpu5:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L40
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.656 cpu0:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L36
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.657 cpu5:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L34
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.664 cpu5:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L34
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.666 cpu3:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L39
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.668 cpu2:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L38
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.676 cpu5:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L38
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.677 cpu3:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L39
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.818 cpu3:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000d. This will clear any SCSI-2 reservations on the device.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.840 cpu2:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000e. This will clear any SCSI-2 reservations on the device.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.863 cpu3:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000f. This will clear any SCSI-2 reservations on the device.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.885 cpu2:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000010. This will clear any SCSI-2 reservations on the device.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.906 cpu3:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000011. This will clear any SCSI-2 reservations on the device.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.929 cpu2:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000012. This will clear any SCSI-2 reservations on the device.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.950 cpu3:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000013. This will clear any SCSI-2 reservations on the device.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.971 cpu2:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000014. This will clear any SCSI-2 reservations on the device.
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.618 cpu1:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L33
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.618 cpu7:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L34
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.619 cpu7:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L35
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.619 cpu1:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L36
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.621 cpu1:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L38
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.623 cpu7:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L37
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.624 cpu6:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L34
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.624 cpu7:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L35
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.625 cpu1:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L33
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.625 cpu6:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L37
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.625 cpu2:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L36
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.626 cpu7:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L39
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.627 cpu1:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L38
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.627 cpu6:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L34
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.627 cpu7:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L40
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.628 cpu7:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L35
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.628 cpu6:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L39
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.628 cpu3:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L33
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.630 cpu6:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L37
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.630 cpu3:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L36
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.630 cpu1:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L38
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.630 cpu6:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L40
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.631 cpu7:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L34
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.631 cpu2:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L35
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.631 cpu6:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L39
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.635 cpu2:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L33
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.635 cpu1:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L37
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.635 cpu7:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L39
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.635 cpu4:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L36
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.635 cpu5:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L38
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.636 cpu2:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L40
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.640 cpu2:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L40
Mar 30 22:49:50 dr-esx1 vmkernel: 2:22:19:37.578 cpu7:4331)WARNING: MemSched: 12625: Non-overhead memory reservation for vmx user-world (worldID=4331) is greater than desired minimum amount of 57344 KB (min=57344 KB, reservedOverhead=0 KB, totalReserved=68764 KB)
Mar 30 22:49:54 dr-esx1 vmkernel: 2:22:19:40.824 cpu7:4331)WARNING: MemSched: vm 4331: 5199: Cannot reduce reservation by 5094 pages (total reservation: 57828 pages, consumed reservation: 53442 pages)
Mar 30 22:54:54 dr-esx1 vmkernel: 2:22:24:41.680 cpu7:4338)WARNING: MemSched: 12625: Non-overhead memory reservation for vmx user-world (worldID=4338) is greater than desired minimum amount of 57344 KB (min=57344 KB, reservedOverhead=0 KB, totalReserved=68708 KB)
Mar 30 22:54:59 dr-esx1 vmkernel: 2:22:24:46.434 cpu6:4338)WARNING: MemSched: vm 4338: 5199: Cannot reduce reservation by 5309 pages (total reservation: 43152 pages, consumed reservation: 37969 pages)

vmkernel:
Mar 28 01:36:04 dr-esx1 vmkernel: 0:01:06:01.718 cpu3:4278)<6>megasas_service_aen[9]: aen received
Mar 28 01:36:04 dr-esx1 vmkernel: 0:01:06:01.718 cpu6:4216)<6>megasas_hotplug_work[9]: event code 0x002c
Mar 28 01:36:08 dr-esx1 vmkernel: 0:01:06:06.009 cpu0:4216)<6>megasas_hotplug_work[9]: aen registered
Mar 30 17:27:37 dr-esx1 vmkernel: 2:16:57:25.530 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f400340) to NMP device "naa.6000d31000163c00000000000000000d" failed on physical path "vmhba3:C0:T1:L33" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:37 dr-esx1 vmkernel: 2:16:57:25.530 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:37 dr-esx1 vmkernel: 2:16:57:25.535 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f3aac40) to NMP device "naa.6000d31000163c00000000000000000e" failed on physical path "vmhba3:C0:T1:L34" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:37 dr-esx1 vmkernel: 2:16:57:25.535 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000e" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:37 dr-esx1 vmkernel: 2:16:57:25.539 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f3aac40) to NMP device "naa.6000d31000163c00000000000000000f" failed on physical path "vmhba3:C0:T1:L35" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:37 dr-esx1 vmkernel: 2:16:57:25.539 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:37 dr-esx1 vmkernel: 2:16:57:25.540 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f3c2140) to NMP device "naa.6000d31000163c000000000000000010" failed on physical path "vmhba3:C0:T1:L36" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:37 dr-esx1 vmkernel: 2:16:57:25.540 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000010" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:37 dr-esx1 vmkernel: 2:16:57:25.543 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f3c2140) to NMP device "naa.6000d31000163c000000000000000011" failed on physical path "vmhba3:C0:T1:L37" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:37 dr-esx1 vmkernel: 2:16:57:25.543 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:37 dr-esx1 vmkernel: 2:16:57:25.543 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f3aac40) to NMP device "naa.6000d31000163c000000000000000012" failed on physical path "vmhba3:C0:T1:L38" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:37 dr-esx1 vmkernel: 2:16:57:25.543 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000012" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:37 dr-esx1 vmkernel: 2:16:57:25.543 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f4a0b40) to NMP device "naa.6000d31000163c000000000000000014" failed on physical path "vmhba3:C0:T0:L40" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:37 dr-esx1 vmkernel: 2:16:57:25.543 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000014" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.544 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f400340) to NMP device "naa.6000d31000163c000000000000000013" failed on physical path "vmhba3:C0:T1:L39" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.544 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000013" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.731 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f400340) to NMP device "naa.6000d31000163c00000000000000000d" failed on physical path "vmhba3:C0:T1:L33" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.731 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.732 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f498540) to NMP device "naa.6000d31000163c00000000000000000e" failed on physical path "vmhba3:C0:T1:L34" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.732 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000e" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.745 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f498540) to NMP device "naa.6000d31000163c000000000000000011" failed on physical path "vmhba3:C0:T1:L37" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.745 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.745 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f372840) to NMP device "naa.6000d31000163c00000000000000000f" failed on physical path "vmhba3:C0:T1:L35" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.745 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.746 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f400340) to NMP device "naa.6000d31000163c000000000000000012" failed on physical path "vmhba3:C0:T1:L38" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.746 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000012" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.749 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f400340) to NMP device "naa.6000d31000163c000000000000000014" failed on physical path "vmhba3:C0:T0:L40" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.749 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000014" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.749 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f396040) to NMP device "naa.6000d31000163c000000000000000010" failed on physical path "vmhba3:C0:T1:L36" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.749 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000010" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.750 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f396040) to NMP device "naa.6000d31000163c000000000000000013" failed on physical path "vmhba3:C0:T1:L39" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.750 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000013" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.936 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f414040) to NMP device "naa.6000d31000163c00000000000000000e" failed on physical path "vmhba3:C0:T1:L34" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.936 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000e" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.939 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f414040) to NMP device "naa.6000d31000163c00000000000000000d" failed on physical path "vmhba3:C0:T1:L33" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.939 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.950 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f414040) to NMP device "naa.6000d31000163c000000000000000011" failed on physical path "vmhba3:C0:T1:L37" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.950 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.951 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f396040) to NMP device "naa.6000d31000163c00000000000000000f" failed on physical path "vmhba3:C0:T1:L35" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.951 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.955 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f396040) to NMP device "naa.6000d31000163c000000000000000012" failed on physical path "vmhba3:C0:T1:L38" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.955 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000012" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.956 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f396040) to NMP device "naa.6000d31000163c000000000000000014" failed on physical path "vmhba3:C0:T0:L40" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.957 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000014" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.957 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f414040) to NMP device "naa.6000d31000163c000000000000000010" failed on physical path "vmhba3:C0:T1:L36" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.957 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000010" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.957 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f400340) to NMP device "naa.6000d31000163c000000000000000013" failed on physical path "vmhba3:C0:T1:L39" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:25.957 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000013" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.141 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f398840) to NMP device "naa.6000d31000163c00000000000000000e" failed on physical path "vmhba3:C0:T1:L34" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.141 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000e" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.157 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f398840) to NMP device "naa.6000d31000163c00000000000000000d" failed on physical path "vmhba3:C0:T1:L33" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.157 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.168 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f398840) to NMP device "naa.6000d31000163c000000000000000011" failed on physical path "vmhba3:C0:T1:L37" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.168 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.169 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f400340) to NMP device "naa.6000d31000163c00000000000000000f" failed on physical path "vmhba3:C0:T1:L35" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.169 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.176 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f400340) to NMP device "naa.6000d31000163c000000000000000012" failed on physical path "vmhba3:C0:T1:L38" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.176 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000012" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.182 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f400340) to NMP device "naa.6000d31000163c000000000000000014" failed on physical path "vmhba3:C0:T0:L40" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.182 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000014" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.182 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f398840) to NMP device "naa.6000d31000163c000000000000000010" failed on physical path "vmhba3:C0:T1:L36" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.182 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000010" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.183 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f398840) to NMP device "naa.6000d31000163c000000000000000013" failed on physical path "vmhba3:C0:T1:L39" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.183 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000013" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.344 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f4c0240) to NMP device "naa.6000d31000163c00000000000000000e" failed on physical path "vmhba3:C0:T1:L34" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.344 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000e" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.386 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f4c0240) to NMP device "naa.6000d31000163c00000000000000000d" failed on physical path "vmhba3:C0:T1:L33" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.386 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.387 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f398840) to NMP device "naa.6000d31000163c000000000000000011" failed on physical path "vmhba3:C0:T1:L37" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.387 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.387 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f400340) to NMP device "naa.6000d31000163c00000000000000000f" failed on physical path "vmhba3:C0:T1:L35" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.387 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.394 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f398840) to NMP device "naa.6000d31000163c000000000000000014" failed on physical path "vmhba3:C0:T0:L40" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.394 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000014" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.394 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f400340) to NMP device "naa.6000d31000163c000000000000000012" failed on physical path "vmhba3:C0:T1:L38" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.394 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000012" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.396 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f400340) to NMP device "naa.6000d31000163c000000000000000010" failed on physical path "vmhba3:C0:T1:L36" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.396 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000010" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.403 cpu5:4281)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f400340) to NMP device "naa.6000d31000163c000000000000000013" failed on physical path "vmhba3:C0:T1:L39" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:38 dr-esx1 vmkernel: 2:16:57:26.403 cpu5:4281)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000013" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.158 cpu2:4278)VSCSI: 2245: handle 8193(vscsi0:0):Reset request on FSS handle 32794 (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.158 cpu2:4278)VSCSI: 2245: handle 8194(vscsi0:1):Reset request on FSS handle 98331 (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.165 cpu2:4278)VSCSI: 2245: handle 8195(vscsi0:2):Reset request on FSS handle 262172 (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.165 cpu6:4183)VSCSI: 2519: handle 8193(vscsi0:0):Reset [Retries: 0/0]
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.166 cpu2:4278)VSCSI: 2245: handle 8196(vscsi0:3):Reset request on FSS handle 294941 (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.166 cpu2:4278)VSCSI: 2245: handle 8197(vscsi0:4):Reset request on FSS handle 294942 (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.166 cpu2:4278)VSCSI: 2245: handle 8198(vscsi0:5):Reset request on FSS handle 294943 (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.166 cpu2:4278)VSCSI: 2245: handle 8199(vscsi0:6):Reset request on FSS handle 294944 (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.166 cpu2:4278)VSCSI: 2245: handle 8200(vscsi0:8):Reset request on FSS handle 294945 (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.167 cpu6:4183)VSCSI: 2319: handle 8193(vscsi0:0):Completing reset (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.167 cpu2:4278)VSCSI: 2245: handle 8201(vscsi0:9):Reset request on FSS handle 294946 (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.167 cpu6:4183)VSCSI: 2519: handle 8194(vscsi0:1):Reset [Retries: 0/0]
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.168 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000d. This will clear any SCSI-2 reservations on the device.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.168 cpu6:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:33: DEVICE RESET ISSUED.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.196 cpu3:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:33): DEVICE RESET SUCCEEDED.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.197 cpu3:4183)VSCSI: 2319: handle 8194(vscsi0:1):Completing reset (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.198 cpu3:4183)VSCSI: 2519: handle 8195(vscsi0:2):Reset [Retries: 0/0]
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.198 cpu3:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000e. This will clear any SCSI-2 reservations on the device.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.198 cpu3:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:34: DEVICE RESET ISSUED.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.219 cpu5:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:34): DEVICE RESET SUCCEEDED.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.219 cpu5:4183)VSCSI: 2319: handle 8195(vscsi0:2):Completing reset (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.220 cpu5:4183)VSCSI: 2519: handle 8196(vscsi0:3):Reset [Retries: 0/0]
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.220 cpu5:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000f. This will clear any SCSI-2 reservations on the device.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.220 cpu5:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:35: DEVICE RESET ISSUED.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.242 cpu7:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:35): DEVICE RESET SUCCEEDED.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.242 cpu7:4183)VSCSI: 2319: handle 8196(vscsi0:3):Completing reset (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.242 cpu7:4183)VSCSI: 2519: handle 8197(vscsi0:4):Reset [Retries: 0/0]
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.242 cpu7:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000010. This will clear any SCSI-2 reservations on the device.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.243 cpu7:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:36: DEVICE RESET ISSUED.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.265 cpu5:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:36): DEVICE RESET SUCCEEDED.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.265 cpu5:4183)VSCSI: 2319: handle 8197(vscsi0:4):Completing reset (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.265 cpu5:4183)VSCSI: 2519: handle 8198(vscsi0:5):Reset [Retries: 0/0]
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.265 cpu5:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000011. This will clear any SCSI-2 reservations on the device.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.265 cpu5:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:37: DEVICE RESET ISSUED.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.289 cpu6:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:37): DEVICE RESET SUCCEEDED.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.289 cpu6:4183)VSCSI: 2319: handle 8198(vscsi0:5):Completing reset (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.289 cpu6:4183)VSCSI: 2519: handle 8199(vscsi0:6):Reset [Retries: 0/0]
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.289 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000012. This will clear any SCSI-2 reservations on the device.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.290 cpu6:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:38: DEVICE RESET ISSUED.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.311 cpu7:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:38): DEVICE RESET SUCCEEDED.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.311 cpu7:4183)VSCSI: 2319: handle 8199(vscsi0:6):Completing reset (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.311 cpu7:4183)VSCSI: 2519: handle 8200(vscsi0:8):Reset [Retries: 0/0]
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.311 cpu7:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000013. This will clear any SCSI-2 reservations on the device.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.311 cpu7:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:39: DEVICE RESET ISSUED.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.331 cpu6:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:39): DEVICE RESET SUCCEEDED.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.331 cpu6:4183)VSCSI: 2319: handle 8200(vscsi0:8):Completing reset (0 outstanding commands)
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.331 cpu6:4183)VSCSI: 2519: handle 8201(vscsi0:9):Reset [Retries: 0/0]
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.331 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000014. This will clear any SCSI-2 reservations on the device.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.331 cpu6:4183)<6>qla4xxx 0000:0f:01.1: scsi7:0:1:40: DEVICE RESET ISSUED.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.353 cpu7:4183)<6>qla4xxx 0000:0f:01.1: scsi(7:0:1:40): DEVICE RESET SUCCEEDED.
Mar 30 17:27:39 dr-esx1 vmkernel: 2:16:57:27.353 cpu7:4183)VSCSI: 2319: handle 8201(vscsi0:9):Completing reset (0 outstanding commands)
Mar 30 17:27:40 dr-esx1 vmkernel: 2:16:57:28.393 cpu7:4131)NetPort: 1157: disabled port 0x1000005
Mar 30 17:27:40 dr-esx1 vmkernel: 2:16:57:28.394 cpu7:4131)Net: 1847: disconnected client from port 0x1000005
Mar 30 17:27:42 dr-esx1 vmkernel: 2:16:57:29.597 cpu7:4277)CBT: 682: Disconnecting the cbt device 30018-cbt with filehandle 196632
Mar 30 17:28:32 dr-esx1 vmkernel: 2:16:58:20.284 cpu2:4276)VSCSI: 2245: handle 8202(vscsi0:0):Reset request on FSS handle 32806 (0 outstanding commands)
Mar 30 17:28:32 dr-esx1 vmkernel: 2:16:58:20.285 cpu6:4183)VSCSI: 2519: handle 8202(vscsi0:0):Reset [Retries: 0/0]
Mar 30 17:28:32 dr-esx1 vmkernel: 2:16:58:20.287 cpu6:4183)VSCSI: 2319: handle 8202(vscsi0:0):Completing reset (0 outstanding commands)
Mar 30 17:28:33 dr-esx1 vmkernel: 2:16:58:21.143 cpu1:4133)NetPort: 1157: disabled port 0x1000006
Mar 30 17:28:33 dr-esx1 vmkernel: 2:16:58:21.143 cpu1:4133)NetPort: 1157: disabled port 0x1000007
Mar 30 17:28:33 dr-esx1 vmkernel: 2:16:58:21.144 cpu1:4133)Net: 1847: disconnected client from port 0x1000006
Mar 30 17:28:33 dr-esx1 vmkernel: 2:16:58:21.144 cpu1:4133)Net: 1847: disconnected client from port 0x1000007
Mar 30 17:28:34 dr-esx1 vmkernel: 2:16:58:22.021 cpu5:4273)CBT: 682: Disconnecting the cbt device 298024-cbt with filehandle 2719780
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.620 cpu7:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L33
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.622 cpu4:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L35
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.622 cpu2:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L34
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.623 cpu1:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L36
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.623 cpu3:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L37
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.623 cpu5:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L38
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.624 cpu4:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L35
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.624 cpu7:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L33
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.624 cpu1:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L40
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.624 cpu4:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L39
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.627 cpu4:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L38
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.627 cpu3:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L34
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.627 cpu2:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L37
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.627 cpu1:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L36
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.627 cpu7:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L33
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.627 cpu4:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L35
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.629 cpu1:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L40
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.629 cpu4:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L39
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.629 cpu4:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L38
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.630 cpu3:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L34
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.631 cpu3:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L37
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.632 cpu3:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L36
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.632 cpu2:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L39
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.633 cpu4:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L35
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.633 cpu6:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L33
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.634 cpu1:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L40
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.634 cpu4:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L38
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.634 cpu2:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L34
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.635 cpu3:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L37
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.635 cpu2:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L36
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.636 cpu3:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L39
Mar 30 17:30:15 dr-esx1 vmkernel: 2:17:00:02.638 cpu1:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L40
Mar 30 17:44:12 dr-esx1 vmkernel: 2:17:14:00.366 cpu2:4110)World: vm 4304: 1534: Starting world vmware-vmx with flags 4
Mar 30 17:44:15 dr-esx1 vmkernel: 2:17:14:03.306 cpu7:4304)World: vm 4305: 1534: Starting world vmm0:VMDRHT1 with flags 4008
Mar 30 17:44:15 dr-esx1 vmkernel: 2:17:14:03.308 cpu7:4304)Sched: vm 4305: 6047: Adding world 'vmm0:VMDRHT1', group 'host/user/pool6', cpu: shares=-3 min=0 minLimit=-1 max=-1, mem: shares=-3 min=0 minLimit=-1 max=-1
Mar 30 17:44:15 dr-esx1 vmkernel: 2:17:14:03.308 cpu7:4304)WARNING: MemSched: 12625: Non-overhead memory reservation for vmx user-world (worldID=4304) is greater than desired minimum amount of 57344 KB (min=57344 KB, reservedOverhead=0 KB, totalReserved=68844 KB)
Mar 30 17:44:15 dr-esx1 vmkernel: 2:17:14:03.308 cpu7:4304)Sched: vm 4305: 6066: renamed group 63 to vm.4304
Mar 30 17:44:15 dr-esx1 vmkernel: 2:17:14:03.308 cpu7:4304)Sched: vm 4305: 6083: group 63 is located under group 51
Mar 30 17:44:15 dr-esx1 vmkernel: 2:17:14:03.314 cpu7:4304)Swap: vm 4305: 2325: extending swap to 7954432 KB
Mar 30 17:44:15 dr-esx1 vmkernel: 2:17:14:03.406 cpu7:4304)World: vm 4307: 1534: Starting world vmm1:VMDRHT1 with flags 4008
Mar 30 17:44:18 dr-esx1 vmkernel: 2:17:14:05.607 cpu7:4304)CBT: 1029: Created device 80016-cbt for cbt driver with filehandle 524310
Mar 30 17:44:18 dr-esx1 vmkernel: 2:17:14:05.814 cpu7:4304)VSCSI: 3482: handle 8203(vscsi0:0):Using sync mode due to sparse disks
Mar 30 17:44:18 dr-esx1 vmkernel: 2:17:14:05.814 cpu7:4304)VSCSI: 3523: handle 8203(vscsi0:0):Creating Virtual Device for world 4305 (FSS handle 327704)
Mar 30 17:44:18 dr-esx1 vmkernel: 2:17:14:06.027 cpu7:4304)WARNING: MemSched: vm 4304: 5199: Cannot reduce reservation by 5094 pages (total reservation: 57828 pages, consumed reservation: 53443 pages)
Mar 30 17:44:18 dr-esx1 vmkernel: 2:17:14:06.106 cpu6:4305)VMMVMKCall: 255: Received INIT from world 4305
Mar 30 17:44:18 dr-esx1 vmkernel: 2:17:14:06.257 cpu6:4305)LSI: 1895: LSI: Initialized rings for scsi0 async=1, record=0 replay=0
Mar 30 17:44:18 dr-esx1 vmkernel: 2:17:14:06.340 cpu1:4307)VMMVMKCall: 255: Received INIT from world 4307
Mar 30 17:44:19 dr-esx1 vmkernel: 2:17:14:06.684 cpu6:4305)Net: 1421: connected VMDRHT1 eth0 to VMNetwork-prod, portID 0x1000008
Mar 30 17:44:19 dr-esx1 vmkernel: 2:17:14:06.688 cpu6:4305)NetPort: 982: enabled port 0x1000008 with mac 00:00:00:00:00:00
Mar 30 17:44:19 dr-esx1 vmkernel: 2:17:14:06.688 cpu6:4305)Net: 1421: connected VMDRHT1 eth0 to VMNetwork-prod, portID 0x1000009
Mar 30 17:44:19 dr-esx1 vmkernel: 2:17:14:06.785 cpu6:4305)NetPort: 982: enabled port 0x1000009 with mac 00:00:00:00:00:00
Mar 30 17:44:21 dr-esx1 vmkernel: 2:17:14:08.749 cpu7:4305)VSCSI: 2245: handle 8203(vscsi0:0):Reset request on FSS handle 327704 (0 outstanding commands)
Mar 30 17:44:21 dr-esx1 vmkernel: 2:17:14:08.749 cpu6:4183)VSCSI: 2519: handle 8203(vscsi0:0):Reset [Retries: 0/0]
Mar 30 17:44:21 dr-esx1 vmkernel: 2:17:14:08.749 cpu6:4183)VSCSI: 2319: handle 8203(vscsi0:0):Completing reset (0 outstanding commands)
Mar 30 17:44:52 dr-esx1 vmkernel: 2:17:14:39.996 cpu5:4305)VSCSI: 2245: handle 8203(vscsi0:0):Reset request on FSS handle 327704 (0 outstanding commands)
Mar 30 17:44:52 dr-esx1 vmkernel: 2:17:14:39.997 cpu6:4183)VSCSI: 2519: handle 8203(vscsi0:0):Reset [Retries: 0/0]
Mar 30 17:44:52 dr-esx1 vmkernel: 2:17:14:39.997 cpu6:4183)VSCSI: 2319: handle 8203(vscsi0:0):Completing reset (0 outstanding commands)
Mar 30 17:46:19 dr-esx1 vmkernel: 2:17:16:06.797 cpu1:4112)World: vm 4311: 1534: Starting world vmware-vmx with flags 4
Mar 30 17:46:24 dr-esx1 vmkernel: 2:17:16:11.683 cpu1:4311)World: vm 4312: 1534: Starting world vmm0:VMDRMB1 with flags 4008
Mar 30 17:46:24 dr-esx1 vmkernel: 2:17:16:11.687 cpu1:4311)Sched: vm 4312: 6047: Adding world 'vmm0:VMDRMB1', group 'host/user/pool6', cpu: shares=-3 min=0 minLimit=-1 max=-1, mem: shares=-3 min=0 minLimit=-1 max=-1
Mar 30 17:46:24 dr-esx1 vmkernel: 2:17:16:11.687 cpu1:4311)WARNING: MemSched: 12625: Non-overhead memory reservation for vmx user-world (worldID=4311) is greater than desired minimum amount of 57344 KB (min=57344 KB, reservedOverhead=0 KB, totalReserved=69220 KB)
Mar 30 17:46:24 dr-esx1 vmkernel: 2:17:16:11.688 cpu1:4311)Sched: vm 4312: 6066: renamed group 64 to vm.4311
Mar 30 17:46:24 dr-esx1 vmkernel: 2:17:16:11.688 cpu1:4311)Sched: vm 4312: 6083: group 64 is located under group 51
Mar 30 17:46:24 dr-esx1 vmkernel: 2:17:16:11.772 cpu1:4311)Swap: vm 4312: 2325: extending swap to 7954432 KB
Mar 30 17:46:24 dr-esx1 vmkernel: 2:17:16:11.933 cpu4:4311)World: vm 4314: 1534: Starting world vmm1:VMDRMB1 with flags 4008
Mar 30 17:46:24 dr-esx1 vmkernel: 2:17:16:11.933 cpu4:4311)World: vm 4315: 1534: Starting world vmm2:VMDRMB1 with flags 4008
Mar 30 17:46:24 dr-esx1 vmkernel: 2:17:16:11.933 cpu4:4311)World: vm 4316: 1534: Starting world vmm3:VMDRMB1 with flags 4008
Mar 30 17:46:28 dr-esx1 vmkernel: 2:17:16:16.451 cpu2:4311)CBT: 1029: Created device 9001e-cbt for cbt driver with filehandle 589854
Mar 30 17:46:30 dr-esx1 vmkernel: 2:17:16:17.729 cpu2:4311)VSCSI: 3523: handle 8204(vscsi0:0):Creating Virtual Device for world 4312 (FSS handle 360480)
Mar 30 17:46:30 dr-esx1 vmkernel: 2:17:16:17.777 cpu2:4311)VSCSI: 3523: handle 8205(vscsi0:1):Creating Virtual Device for world 4312 (FSS handle 426017)
Mar 30 17:46:30 dr-esx1 vmkernel: 2:17:16:17.808 cpu2:4311)VSCSI: 3523: handle 8206(vscsi0:2):Creating Virtual Device for world 4312 (FSS handle 589858)
Mar 30 17:46:30 dr-esx1 vmkernel: 2:17:16:17.836 cpu2:4311)VSCSI: 3523: handle 8207(vscsi0:3):Creating Virtual Device for world 4312 (FSS handle 1114147)
Mar 30 17:46:30 dr-esx1 vmkernel: 2:17:16:17.861 cpu2:4311)VSCSI: 3523: handle 8208(vscsi0:4):Creating Virtual Device for world 4312 (FSS handle 3014692)
Mar 30 17:46:30 dr-esx1 vmkernel: 2:17:16:17.877 cpu2:4311)VSCSI: 3523: handle 8209(vscsi0:5):Creating Virtual Device for world 4312 (FSS handle 753701)
Mar 30 17:46:30 dr-esx1 vmkernel: 2:17:16:17.891 cpu2:4311)VSCSI: 3523: handle 8210(vscsi0:6):Creating Virtual Device for world 4312 (FSS handle 360486)
Mar 30 17:46:30 dr-esx1 vmkernel: 2:17:16:17.917 cpu2:4311)VSCSI: 3523: handle 8211(vscsi0:8):Creating Virtual Device for world 4312 (FSS handle 360487)
Mar 30 17:46:30 dr-esx1 vmkernel: 2:17:16:17.929 cpu2:4311)VSCSI: 3523: handle 8212(vscsi0:9):Creating Virtual Device for world 4312 (FSS handle 557096)
Mar 30 17:46:31 dr-esx1 vmkernel: 2:17:16:18.917 cpu7:4311)WARNING: MemSched: vm 4311: 5199: Cannot reduce reservation by 5310 pages (total reservation: 58924 pages, consumed reservation: 53828 pages)
Mar 30 17:46:31 dr-esx1 vmkernel: 2:17:16:19.165 cpu3:4312)VMMVMKCall: 255: Received INIT from world 4312
Mar 30 17:46:31 dr-esx1 vmkernel: 2:17:16:19.371 cpu3:4312)LSI: 1895: LSI: Initialized rings for scsi0 async=1, record=0 replay=0
Mar 30 17:46:32 dr-esx1 vmkernel: 2:17:16:19.529 cpu6:4314)VMMVMKCall: 255: Received INIT from world 4314
Mar 30 17:46:32 dr-esx1 vmkernel: 2:17:16:19.557 cpu5:4315)VMMVMKCall: 255: Received INIT from world 4315
Mar 30 17:46:32 dr-esx1 vmkernel: 2:17:16:19.577 cpu6:4316)VMMVMKCall: 255: Received INIT from world 4316
Mar 30 17:46:34 dr-esx1 vmkernel: 2:17:16:21.663 cpu7:4312)Net: 1421: connected VMDRMB1 eth0 to VMNetwork-prod, portID 0x100000a
Mar 30 17:46:34 dr-esx1 vmkernel: 2:17:16:21.668 cpu7:4312)NetPort: 982: enabled port 0x100000a with mac 00:00:00:00:00:00
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.330 cpu5:4312)VSCSI: 2245: handle 8204(vscsi0:0):Reset request on FSS handle 360480 (0 outstanding commands)
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.330 cpu5:4312)VSCSI: 2245: handle 8205(vscsi0:1):Reset request on FSS handle 426017 (0 outstanding commands)
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.331 cpu6:4183)VSCSI: 2519: handle 8204(vscsi0:0):Reset [Retries: 0/0]
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.331 cpu5:4312)VSCSI: 2245: handle 8206(vscsi0:2):Reset request on FSS handle 589858 (0 outstanding commands)
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.331 cpu5:4312)VSCSI: 2245: handle 8207(vscsi0:3):Reset request on FSS handle 1114147 (0 outstanding commands)
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.331 cpu5:4312)VSCSI: 2245: handle 8208(vscsi0:4):Reset request on FSS handle 3014692 (0 outstanding commands)
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.331 cpu5:4312)VSCSI: 2245: handle 8209(vscsi0:5):Reset request on FSS handle 753701 (0 outstanding commands)
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.331 cpu6:4183)VSCSI: 2319: handle 8204(vscsi0:0):Completing reset (0 outstanding commands)
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.331 cpu5:4312)VSCSI: 2245: handle 8210(vscsi0:6):Reset request on FSS handle 360486 (0 outstanding commands)
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.331 cpu6:4183)VSCSI: 2519: handle 8205(vscsi0:1):Reset [Retries: 0/0]
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.331 cpu5:4312)VSCSI: 2245: handle 8211(vscsi0:8):Reset request on FSS handle 360487 (0 outstanding commands)
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.331 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000d. This will clear any SCSI-2 reservations on the device.
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.331 cpu5:4312)VSCSI: 2245: handle 8212(vscsi0:9):Reset request on FSS handle 557096 (0 outstanding commands)
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.331 cpu6:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:33: DEVICE RESET ISSUED.
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.351 cpu7:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:33): DEVICE RESET SUCCEEDED.
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.351 cpu7:4183)VSCSI: 2319: handle 8205(vscsi0:1):Completing reset (0 outstanding commands)
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.351 cpu7:4183)VSCSI: 2519: handle 8206(vscsi0:2):Reset [Retries: 0/0]
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.351 cpu7:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000e. This will clear any SCSI-2 reservations on the device.
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.351 cpu7:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:34: DEVICE RESET ISSUED.
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.371 cpu7:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:34): DEVICE RESET SUCCEEDED.
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.371 cpu7:4183)VSCSI: 2319: handle 8206(vscsi0:2):Completing reset (0 outstanding commands)
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.372 cpu7:4183)VSCSI: 2519: handle 8207(vscsi0:3):Reset [Retries: 0/0]
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.372 cpu7:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000f. This will clear any SCSI-2 reservations on the device.
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.372 cpu7:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:35: DEVICE RESET ISSUED.
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.394 cpu6:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:35): DEVICE RESET SUCCEEDED.
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.394 cpu6:4183)VSCSI: 2319: handle 8207(vscsi0:3):Completing reset (0 outstanding commands)
Mar 30 17:46:44 dr-esx1 vmkernel: 2:17:16:32.394 cpu6:4183)VSCSI: 2519: handle 8208(vscsi0:4):Reset [Retries: 0/0]
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.394 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000010. This will clear any SCSI-2 reservations on the device.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.394 cpu6:4183)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f47ce40) to NMP device "naa.6000d31000163c000000000000000008" failed on physical path "vmhba3:C0:T0:L15" H:0x8 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.394 cpu6:4183)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa.6000d31000163c000000000000000008" state in doubt; requested fast path state update...
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.394 cpu6:4183)ScsiDeviceIO: 1672: Command 0x28 to device "naa.6000d31000163c000000000000000008" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.395 cpu6:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:36: DEVICE RESET ISSUED.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.417 cpu2:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:36): DEVICE RESET SUCCEEDED.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.418 cpu2:4183)VSCSI: 2319: handle 8208(vscsi0:4):Completing reset (0 outstanding commands)
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.419 cpu2:4183)VSCSI: 2519: handle 8209(vscsi0:5):Reset [Retries: 0/0]
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.419 cpu2:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000011. This will clear any SCSI-2 reservations on the device.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.420 cpu2:4183)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f47ce40) to NMP device "naa.6000d31000163c000000000000000008" failed on physical path "vmhba3:C0:T0:L15" H:0x8 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.420 cpu2:4183)ScsiDeviceIO: 1672: Command 0x28 to device "naa.6000d31000163c000000000000000008" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.421 cpu2:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:37: DEVICE RESET ISSUED.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.443 cpu1:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:37): DEVICE RESET SUCCEEDED.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.444 cpu1:4183)VSCSI: 2319: handle 8209(vscsi0:5):Completing reset (0 outstanding commands)
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.445 cpu1:4183)VSCSI: 2519: handle 8210(vscsi0:6):Reset [Retries: 0/0]
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.445 cpu1:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000012. This will clear any SCSI-2 reservations on the device.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.445 cpu1:4183)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f47ce40) to NMP device "naa.6000d31000163c000000000000000008" failed on physical path "vmhba3:C0:T0:L15" H:0x8 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.445 cpu1:4183)ScsiDeviceIO: 1672: Command 0x28 to device "naa.6000d31000163c000000000000000008" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.446 cpu1:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:38: DEVICE RESET ISSUED.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.468 cpu5:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:38): DEVICE RESET SUCCEEDED.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.468 cpu5:4183)VSCSI: 2319: handle 8210(vscsi0:6):Completing reset (0 outstanding commands)
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.468 cpu5:4183)VSCSI: 2519: handle 8211(vscsi0:8):Reset [Retries: 0/0]
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.468 cpu5:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000013. This will clear any SCSI-2 reservations on the device.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.468 cpu5:4183)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f47ce40) to NMP device "naa.6000d31000163c000000000000000008" failed on physical path "vmhba3:C0:T0:L15" H:0x8 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.469 cpu5:4183)ScsiDeviceIO: 1672: Command 0x28 to device "naa.6000d31000163c000000000000000008" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.469 cpu5:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:39: DEVICE RESET ISSUED.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.492 cpu6:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:39): DEVICE RESET SUCCEEDED.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.492 cpu6:4183)VSCSI: 2319: handle 8211(vscsi0:8):Completing reset (0 outstanding commands)
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.492 cpu6:4183)VSCSI: 2519: handle 8212(vscsi0:9):Reset [Retries: 0/0]
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.492 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000014. This will clear any SCSI-2 reservations on the device.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.492 cpu6:4183)<6>qla4xxx 0000:0f:01.1: scsi7:0:1:40: DEVICE RESET ISSUED.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.516 cpu6:4183)<6>qla4xxx 0000:0f:01.1: scsi(7:0:1:40): DEVICE RESET SUCCEEDED.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.516 cpu6:4183)VSCSI: 2319: handle 8212(vscsi0:9):Completing reset (0 outstanding commands)
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.538 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x0 (0x41027f424140) to NMP device "naa.6000d31000163c00000000000000000d" failed on physical path "vmhba3:C0:T1:L33" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.538 cpu6:4102)ScsiDeviceIO: 1672: Command 0x0 to device "naa.6000d31000163c00000000000000000d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.538 cpu6:4102)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c00000000000000000d
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.541 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x0 (0x41027f400840) to NMP device "naa.6000d31000163c00000000000000000e" failed on physical path "vmhba3:C0:T1:L34" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.541 cpu6:4102)ScsiDeviceIO: 1672: Command 0x0 to device "naa.6000d31000163c00000000000000000e" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.541 cpu6:4102)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c00000000000000000e
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.546 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x0 (0x41027f3c6740) to NMP device "naa.6000d31000163c00000000000000000f" failed on physical path "vmhba3:C0:T1:L35" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.546 cpu6:4102)ScsiDeviceIO: 1672: Command 0x0 to device "naa.6000d31000163c00000000000000000f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.546 cpu6:4102)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c00000000000000000f
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.552 cpu6:4112)NMP: nmp_CompleteCommandForPath: Command 0x0 (0x41027f462e40) to NMP device "naa.6000d31000163c000000000000000010" failed on physical path "vmhba3:C0:T1:L36" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.552 cpu6:4112)ScsiDeviceIO: 1672: Command 0x0 to device "naa.6000d31000163c000000000000000010" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.552 cpu6:4112)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000010
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.559 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x0 (0x41027f4c4740) to NMP device "naa.6000d31000163c000000000000000011" failed on physical path "vmhba3:C0:T1:L37" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.559 cpu6:4102)ScsiDeviceIO: 1672: Command 0x0 to device "naa.6000d31000163c000000000000000011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.559 cpu6:4102)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000011
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.565 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x0 (0x41027f3d2e40) to NMP device "naa.6000d31000163c000000000000000012" failed on physical path "vmhba3:C0:T1:L38" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.565 cpu6:4102)ScsiDeviceIO: 1672: Command 0x0 to device "naa.6000d31000163c000000000000000012" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.565 cpu6:4102)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000012
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.572 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x0 (0x41027f468840) to NMP device "naa.6000d31000163c000000000000000013" failed on physical path "vmhba3:C0:T1:L39" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.572 cpu6:4102)ScsiDeviceIO: 1672: Command 0x0 to device "naa.6000d31000163c000000000000000013" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.572 cpu6:4102)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000013
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.601 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x0 (0x41027f396140) to NMP device "naa.6000d31000163c000000000000000014" failed on physical path "vmhba3:C0:T0:L40" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.601 cpu6:4102)ScsiDeviceIO: 1672: Command 0x0 to device "naa.6000d31000163c000000000000000014" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:46:45 dr-esx1 vmkernel: 2:17:16:32.601 cpu6:4102)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000014
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.023 cpu4:4312)VSCSI: 2245: handle 8204(vscsi0:0):Reset request on FSS handle 360480 (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.034 cpu4:4312)VSCSI: 2245: handle 8205(vscsi0:1):Reset request on FSS handle 426017 (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.034 cpu6:4183)VSCSI: 2519: handle 8204(vscsi0:0):Reset [Retries: 0/0]
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.035 cpu4:4312)VSCSI: 2245: handle 8206(vscsi0:2):Reset request on FSS handle 589858 (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.035 cpu4:4312)VSCSI: 2245: handle 8207(vscsi0:3):Reset request on FSS handle 1114147 (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.036 cpu4:4312)VSCSI: 2245: handle 8208(vscsi0:4):Reset request on FSS handle 3014692 (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.036 cpu4:4312)VSCSI: 2245: handle 8209(vscsi0:5):Reset request on FSS handle 753701 (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.036 cpu6:4183)VSCSI: 2319: handle 8204(vscsi0:0):Completing reset (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.036 cpu4:4312)VSCSI: 2245: handle 8210(vscsi0:6):Reset request on FSS handle 360486 (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.036 cpu6:4183)VSCSI: 2519: handle 8205(vscsi0:1):Reset [Retries: 0/0]
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.036 cpu4:4312)VSCSI: 2245: handle 8211(vscsi0:8):Reset request on FSS handle 360487 (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.036 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000d. This will clear any SCSI-2 reservations on the device.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.036 cpu4:4312)VSCSI: 2245: handle 8212(vscsi0:9):Reset request on FSS handle 557096 (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.037 cpu6:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:33: DEVICE RESET ISSUED.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.059 cpu7:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:33): DEVICE RESET SUCCEEDED.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.059 cpu7:4183)VSCSI: 2319: handle 8205(vscsi0:1):Completing reset (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.059 cpu7:4183)VSCSI: 2519: handle 8206(vscsi0:2):Reset [Retries: 0/0]
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.059 cpu7:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000e. This will clear any SCSI-2 reservations on the device.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.059 cpu7:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:34: DEVICE RESET ISSUED.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.079 cpu6:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:34): DEVICE RESET SUCCEEDED.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.079 cpu6:4183)VSCSI: 2319: handle 8206(vscsi0:2):Completing reset (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.079 cpu6:4183)VSCSI: 2519: handle 8207(vscsi0:3):Reset [Retries: 0/0]
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.080 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000f. This will clear any SCSI-2 reservations on the device.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.080 cpu6:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:35: DEVICE RESET ISSUED.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.103 cpu7:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:35): DEVICE RESET SUCCEEDED.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.103 cpu7:4183)VSCSI: 2319: handle 8207(vscsi0:3):Completing reset (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.104 cpu7:4183)VSCSI: 2519: handle 8208(vscsi0:4):Reset [Retries: 0/0]
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.104 cpu7:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000010. This will clear any SCSI-2 reservations on the device.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.104 cpu7:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:36: DEVICE RESET ISSUED.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.126 cpu6:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:36): DEVICE RESET SUCCEEDED.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.126 cpu6:4183)VSCSI: 2319: handle 8208(vscsi0:4):Completing reset (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.126 cpu6:4183)VSCSI: 2519: handle 8209(vscsi0:5):Reset [Retries: 0/0]
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.126 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000011. This will clear any SCSI-2 reservations on the device.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.126 cpu6:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:37: DEVICE RESET ISSUED.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.150 cpu7:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:37): DEVICE RESET SUCCEEDED.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.150 cpu7:4183)VSCSI: 2319: handle 8209(vscsi0:5):Completing reset (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.150 cpu7:4183)VSCSI: 2519: handle 8210(vscsi0:6):Reset [Retries: 0/0]
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.150 cpu7:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000012. This will clear any SCSI-2 reservations on the device.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.150 cpu7:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:38: DEVICE RESET ISSUED.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.173 cpu7:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:38): DEVICE RESET SUCCEEDED.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.173 cpu7:4183)VSCSI: 2319: handle 8210(vscsi0:6):Completing reset (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.173 cpu7:4183)VSCSI: 2519: handle 8211(vscsi0:8):Reset [Retries: 0/0]
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.173 cpu7:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000013. This will clear any SCSI-2 reservations on the device.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.174 cpu7:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:39: DEVICE RESET ISSUED.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.197 cpu6:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:39): DEVICE RESET SUCCEEDED.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.197 cpu6:4183)VSCSI: 2319: handle 8211(vscsi0:8):Completing reset (0 outstanding commands)
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.197 cpu6:4183)VSCSI: 2519: handle 8212(vscsi0:9):Reset [Retries: 0/0]
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.197 cpu6:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000014. This will clear any SCSI-2 reservations on the device.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.198 cpu6:4183)<6>qla4xxx 0000:0f:01.1: scsi7:0:1:40: DEVICE RESET ISSUED.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.220 cpu7:4183)<6>qla4xxx 0000:0f:01.1: scsi(7:0:1:40): DEVICE RESET SUCCEEDED.
Mar 30 17:48:34 dr-esx1 vmkernel: 2:17:18:22.220 cpu7:4183)VSCSI: 2319: handle 8212(vscsi0:9):Completing reset (0 outstanding commands)
Mar 30 17:48:37 dr-esx1 vmkernel: 2:17:18:25.258 cpu3:4307)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f3b2240) to NMP device "naa.6000d31000163c00000000000000000d" failed on physical path "vmhba3:C0:T1:L33" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:48:37 dr-esx1 vmkernel: 2:17:18:25.259 cpu3:4307)ScsiDeviceIO: 1672: Command 0x28 to device "naa.6000d31000163c00000000000000000d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:48:37 dr-esx1 vmkernel: 2:17:18:25.259 cpu3:4307)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c00000000000000000d
Mar 30 17:48:37 dr-esx1 vmkernel: 2:17:18:25.370 cpu3:4307)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f4ea040) to NMP device "naa.6000d31000163c00000000000000000e" failed on physical path "vmhba3:C0:T1:L34" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:48:37 dr-esx1 vmkernel: 2:17:18:25.371 cpu3:4307)ScsiDeviceIO: 1672: Command 0x28 to device "naa.6000d31000163c00000000000000000e" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:48:37 dr-esx1 vmkernel: 2:17:18:25.371 cpu3:4307)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c00000000000000000e
Mar 30 17:48:37 dr-esx1 vmkernel: 2:17:18:25.492 cpu3:4307)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f4c2e40) to NMP device "naa.6000d31000163c00000000000000000f" failed on physical path "vmhba3:C0:T1:L35" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:48:37 dr-esx1 vmkernel: 2:17:18:25.492 cpu3:4307)ScsiDeviceIO: 1672: Command 0x28 to device "naa.6000d31000163c00000000000000000f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:48:37 dr-esx1 vmkernel: 2:17:18:25.492 cpu3:4307)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c00000000000000000f
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:25.604 cpu3:4307)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f3a2840) to NMP device "naa.6000d31000163c000000000000000010" failed on physical path "vmhba3:C0:T1:L36" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:25.604 cpu3:4307)ScsiDeviceIO: 1672: Command 0x28 to device "naa.6000d31000163c000000000000000010" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:25.605 cpu3:4307)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000010
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:25.727 cpu3:4307)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f482240) to NMP device "naa.6000d31000163c000000000000000011" failed on physical path "vmhba3:C0:T1:L37" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:25.727 cpu3:4307)ScsiDeviceIO: 1672: Command 0x28 to device "naa.6000d31000163c000000000000000011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:25.727 cpu3:4307)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000011
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:25.854 cpu3:4307)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f4da740) to NMP device "naa.6000d31000163c000000000000000012" failed on physical path "vmhba3:C0:T1:L38" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:25.854 cpu3:4307)ScsiDeviceIO: 1672: Command 0x28 to device "naa.6000d31000163c000000000000000012" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:25.855 cpu3:4307)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000012
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:25.956 cpu3:4307)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f3d0040) to NMP device "naa.6000d31000163c000000000000000013" failed on physical path "vmhba3:C0:T1:L39" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:25.957 cpu3:4307)ScsiDeviceIO: 1672: Command 0x28 to device "naa.6000d31000163c000000000000000013" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:25.958 cpu3:4307)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000013
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:26.052 cpu3:4307)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f3de340) to NMP device "naa.6000d31000163c000000000000000014" failed on physical path "vmhba3:C0:T0:L40" H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:26.052 cpu3:4307)ScsiDeviceIO: 1672: Command 0x28 to device "naa.6000d31000163c000000000000000014" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Mar 30 17:48:38 dr-esx1 vmkernel: 2:17:18:26.052 cpu3:4307)WARNING: ScsiCore: 1354: Power-on Reset occurred on naa.6000d31000163c000000000000000014
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.629 cpu4:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L33
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.630 cpu6:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L34
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.632 cpu4:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L35
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.632 cpu1:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L36
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.632 cpu7:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L38
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.633 cpu6:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L37
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.638 cpu4:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L33
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.641 cpu5:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L35
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.643 cpu0:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L36
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.643 cpu4:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L33
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.643 cpu5:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L40
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.644 cpu4:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L39
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.645 cpu5:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L35
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.646 cpu0:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L37
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.646 cpu5:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L40
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.651 cpu0:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L37
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.656 cpu5:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L40
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.656 cpu0:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L36
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.657 cpu5:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L34
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.664 cpu5:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L34
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.666 cpu3:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L39
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.668 cpu2:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L38
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.676 cpu5:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L38
Mar 30 17:50:15 dr-esx1 vmkernel: 2:17:20:02.677 cpu3:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L39
Mar 30 18:07:24 dr-esx1 vmkernel: 2:17:37:12.278 cpu3:4305)VSCSI: 2245: handle 8203(vscsi0:0):Reset request on FSS handle 327704 (0 outstanding commands)
Mar 30 18:07:24 dr-esx1 vmkernel: 2:17:37:12.284 cpu3:4183)VSCSI: 2519: handle 8203(vscsi0:0):Reset [Retries: 0/0]
Mar 30 18:07:24 dr-esx1 vmkernel: 2:17:37:12.285 cpu3:4183)VSCSI: 2319: handle 8203(vscsi0:0):Completing reset (0 outstanding commands)
Mar 30 18:07:26 dr-esx1 vmkernel: 2:17:37:13.889 cpu6:4131)NetPort: 1157: disabled port 0x1000008
Mar 30 18:07:26 dr-esx1 vmkernel: 2:17:37:13.889 cpu6:4131)NetPort: 1157: disabled port 0x1000009
Mar 30 18:07:26 dr-esx1 vmkernel: 2:17:37:13.891 cpu6:4131)Net: 1847: disconnected client from port 0x1000008
Mar 30 18:07:26 dr-esx1 vmkernel: 2:17:37:13.892 cpu6:4131)Net: 1847: disconnected client from port 0x1000009
Mar 30 18:07:27 dr-esx1 vmkernel: 2:17:37:14.635 cpu1:4304)CBT: 682: Disconnecting the cbt device 80016-cbt with filehandle 524310
Mar 30 18:10:41 dr-esx1 vmkernel: 2:17:40:29.386 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f48c740) to NMP device "naa.6000d31000163c00000000000000000d" failed on physical path "vmhba3:C0:T1:L33" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:41 dr-esx1 vmkernel: 2:17:40:29.387 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:41 dr-esx1 vmkernel: 2:17:40:29.388 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f4a8b40) to NMP device "naa.6000d31000163c00000000000000000e" failed on physical path "vmhba3:C0:T1:L34" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:41 dr-esx1 vmkernel: 2:17:40:29.388 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000e" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:41 dr-esx1 vmkernel: 2:17:40:29.388 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f42cd40) to NMP device "naa.6000d31000163c00000000000000000f" failed on physical path "vmhba3:C0:T1:L35" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:41 dr-esx1 vmkernel: 2:17:40:29.388 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:41 dr-esx1 vmkernel: 2:17:40:29.390 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f3e6540) to NMP device "naa.6000d31000163c000000000000000010" failed on physical path "vmhba3:C0:T1:L36" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:41 dr-esx1 vmkernel: 2:17:40:29.390 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000010" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:41 dr-esx1 vmkernel: 2:17:40:29.391 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f3cac40) to NMP device "naa.6000d31000163c000000000000000011" failed on physical path "vmhba3:C0:T1:L37" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:41 dr-esx1 vmkernel: 2:17:40:29.391 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:41 dr-esx1 vmkernel: 2:17:40:29.392 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f3dc840) to NMP device "naa.6000d31000163c000000000000000012" failed on physical path "vmhba3:C0:T1:L38" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:41 dr-esx1 vmkernel: 2:17:40:29.392 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000012" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:41 dr-esx1 vmkernel: 2:17:40:29.393 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f398840) to NMP device "naa.6000d31000163c000000000000000014" failed on physical path "vmhba3:C0:T0:L40" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:41 dr-esx1 vmkernel: 2:17:40:29.393 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000014" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:41 dr-esx1 vmkernel: 2:17:40:29.397 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f394040) to NMP device "naa.6000d31000163c000000000000000013" failed on physical path "vmhba3:C0:T1:L39" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:41 dr-esx1 vmkernel: 2:17:40:29.397 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000013" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.589 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f47c940) to NMP device "naa.6000d31000163c00000000000000000d" failed on physical path "vmhba3:C0:T1:L33" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.590 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.592 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f484040) to NMP device "naa.6000d31000163c00000000000000000e" failed on physical path "vmhba3:C0:T1:L34" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.592 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000e" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.593 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f4bcb40) to NMP device "naa.6000d31000163c00000000000000000f" failed on physical path "vmhba3:C0:T1:L35" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.593 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.599 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f460040) to NMP device "naa.6000d31000163c000000000000000012" failed on physical path "vmhba3:C0:T1:L38" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.599 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000012" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.600 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f43e540) to NMP device "naa.6000d31000163c000000000000000013" failed on physical path "vmhba3:C0:T1:L39" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.600 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000013" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.601 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f3c6740) to NMP device "naa.6000d31000163c000000000000000010" failed on physical path "vmhba3:C0:T1:L36" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.601 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000010" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.601 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f432540) to NMP device "naa.6000d31000163c000000000000000014" failed on physical path "vmhba3:C0:T0:L40" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.601 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000014" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.603 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f460940) to NMP device "naa.6000d31000163c000000000000000011" failed on physical path "vmhba3:C0:T1:L37" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.603 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.801 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f4dce40) to NMP device "naa.6000d31000163c00000000000000000d" failed on physical path "vmhba3:C0:T1:L33" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.801 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.802 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f3c0740) to NMP device "naa.6000d31000163c000000000000000011" failed on physical path "vmhba3:C0:T1:L37" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.802 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.803 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f428d40) to NMP device "naa.6000d31000163c00000000000000000e" failed on physical path "vmhba3:C0:T1:L34" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.803 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000e" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.804 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f3d2640) to NMP device "naa.6000d31000163c000000000000000012" failed on physical path "vmhba3:C0:T1:L38" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.804 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000012" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.805 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f440040) to NMP device "naa.6000d31000163c00000000000000000f" failed on physical path "vmhba3:C0:T1:L35" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.805 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.807 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f40c740) to NMP device "naa.6000d31000163c000000000000000013" failed on physical path "vmhba3:C0:T1:L39" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.807 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000013" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.807 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f3c4a40) to NMP device "naa.6000d31000163c000000000000000010" failed on physical path "vmhba3:C0:T1:L36" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.808 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000010" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.809 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f400640) to NMP device "naa.6000d31000163c000000000000000014" failed on physical path "vmhba3:C0:T0:L40" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:29.809 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000014" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.003 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f412e40) to NMP device "naa.6000d31000163c000000000000000013" failed on physical path "vmhba3:C0:T1:L39" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.004 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000013" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.005 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f362e40) to NMP device "naa.6000d31000163c000000000000000010" failed on physical path "vmhba3:C0:T1:L36" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.005 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000010" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.007 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f4dc240) to NMP device "naa.6000d31000163c00000000000000000d" failed on physical path "vmhba3:C0:T1:L33" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.007 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.008 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f434a40) to NMP device "naa.6000d31000163c000000000000000014" failed on physical path "vmhba3:C0:T0:L40" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.009 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000014" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.010 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f3f4240) to NMP device "naa.6000d31000163c000000000000000011" failed on physical path "vmhba3:C0:T1:L37" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.010 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.012 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f488440) to NMP device "naa.6000d31000163c00000000000000000e" failed on physical path "vmhba3:C0:T1:L34" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.012 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000e" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.025 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f45a240) to NMP device "naa.6000d31000163c000000000000000012" failed on physical path "vmhba3:C0:T1:L38" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.025 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000012" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.029 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f4b6e40) to NMP device "naa.6000d31000163c00000000000000000f" failed on physical path "vmhba3:C0:T1:L35" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.029 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.225 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f3c6640) to NMP device "naa.6000d31000163c00000000000000000f" failed on physical path "vmhba3:C0:T1:L35" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.225 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.226 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f448840) to NMP device "naa.6000d31000163c000000000000000013" failed on physical path "vmhba3:C0:T1:L39" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.226 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000013" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.228 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f40ab40) to NMP device "naa.6000d31000163c000000000000000010" failed on physical path "vmhba3:C0:T1:L36" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.228 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000010" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.229 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f440c40) to NMP device "naa.6000d31000163c00000000000000000d" failed on physical path "vmhba3:C0:T1:L33" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.230 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.231 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f4d0840) to NMP device "naa.6000d31000163c000000000000000014" failed on physical path "vmhba3:C0:T0:L40" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.231 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000014" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.233 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f358d40) to NMP device "naa.6000d31000163c000000000000000011" failed on physical path "vmhba3:C0:T1:L37" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.233 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000011" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.234 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f3b2b40) to NMP device "naa.6000d31000163c00000000000000000e" failed on physical path "vmhba3:C0:T1:L34" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.234 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c00000000000000000e" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.241 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x35 (0x41027f418c40) to NMP device "naa.6000d31000163c000000000000000012" failed on physical path "vmhba3:C0:T1:L38" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:42 dr-esx1 vmkernel: 2:17:40:30.241 cpu6:4102)ScsiDeviceIO: 1672: Command 0x35 to device "naa.6000d31000163c000000000000000012" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.814 cpu2:4312)VSCSI: 2245: handle 8204(vscsi0:0):Reset request on FSS handle 360480 (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.815 cpu2:4312)VSCSI: 2245: handle 8205(vscsi0:1):Reset request on FSS handle 426017 (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.815 cpu3:4183)VSCSI: 2519: handle 8204(vscsi0:0):Reset [Retries: 0/0]
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.815 cpu2:4312)VSCSI: 2245: handle 8206(vscsi0:2):Reset request on FSS handle 589858 (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.816 cpu2:4312)VSCSI: 2245: handle 8207(vscsi0:3):Reset request on FSS handle 1114147 (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.816 cpu2:4312)VSCSI: 2245: handle 8208(vscsi0:4):Reset request on FSS handle 3014692 (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.817 cpu2:4312)VSCSI: 2245: handle 8209(vscsi0:5):Reset request on FSS handle 753701 (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.817 cpu2:4312)VSCSI: 2245: handle 8210(vscsi0:6):Reset request on FSS handle 360486 (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.817 cpu3:4183)VSCSI: 2319: handle 8204(vscsi0:0):Completing reset (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.818 cpu2:4312)VSCSI: 2245: handle 8211(vscsi0:8):Reset request on FSS handle 360487 (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.818 cpu3:4183)VSCSI: 2519: handle 8205(vscsi0:1):Reset [Retries: 0/0]
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.818 cpu2:4312)VSCSI: 2245: handle 8212(vscsi0:9):Reset request on FSS handle 557096 (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.818 cpu3:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000d. This will clear any SCSI-2 reservations on the device.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.818 cpu3:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:33: DEVICE RESET ISSUED.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.840 cpu2:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:33): DEVICE RESET SUCCEEDED.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.840 cpu2:4183)VSCSI: 2319: handle 8205(vscsi0:1):Completing reset (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.840 cpu2:4183)VSCSI: 2519: handle 8206(vscsi0:2):Reset [Retries: 0/0]
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.840 cpu2:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000e. This will clear any SCSI-2 reservations on the device.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.840 cpu2:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:34: DEVICE RESET ISSUED.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.863 cpu3:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:34): DEVICE RESET SUCCEEDED.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.863 cpu3:4183)VSCSI: 2319: handle 8206(vscsi0:2):Completing reset (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.863 cpu3:4183)VSCSI: 2519: handle 8207(vscsi0:3):Reset [Retries: 0/0]
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.863 cpu3:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c00000000000000000f. This will clear any SCSI-2 reservations on the device.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.863 cpu3:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:35: DEVICE RESET ISSUED.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.885 cpu2:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:35): DEVICE RESET SUCCEEDED.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.885 cpu2:4183)VSCSI: 2319: handle 8207(vscsi0:3):Completing reset (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.885 cpu2:4183)VSCSI: 2519: handle 8208(vscsi0:4):Reset [Retries: 0/0]
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.885 cpu2:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000010. This will clear any SCSI-2 reservations on the device.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.885 cpu2:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:36: DEVICE RESET ISSUED.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.905 cpu3:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:36): DEVICE RESET SUCCEEDED.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.905 cpu3:4183)VSCSI: 2319: handle 8208(vscsi0:4):Completing reset (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.905 cpu3:4183)VSCSI: 2519: handle 8209(vscsi0:5):Reset [Retries: 0/0]
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.906 cpu3:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000011. This will clear any SCSI-2 reservations on the device.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.906 cpu3:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:37: DEVICE RESET ISSUED.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.928 cpu2:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:37): DEVICE RESET SUCCEEDED.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.928 cpu2:4183)VSCSI: 2319: handle 8209(vscsi0:5):Completing reset (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.928 cpu2:4183)VSCSI: 2519: handle 8210(vscsi0:6):Reset [Retries: 0/0]
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.929 cpu2:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000012. This will clear any SCSI-2 reservations on the device.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.929 cpu2:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:38: DEVICE RESET ISSUED.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.950 cpu3:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:38): DEVICE RESET SUCCEEDED.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.950 cpu3:4183)VSCSI: 2319: handle 8210(vscsi0:6):Completing reset (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.950 cpu3:4183)VSCSI: 2519: handle 8211(vscsi0:8):Reset [Retries: 0/0]
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.950 cpu3:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000013. This will clear any SCSI-2 reservations on the device.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.950 cpu3:4183)<6>qla4xxx 0000:0f:01.3: scsi8:0:0:39: DEVICE RESET ISSUED.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.971 cpu2:4183)<6>qla4xxx 0000:0f:01.3: scsi(8:0:0:39): DEVICE RESET SUCCEEDED.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.971 cpu2:4183)VSCSI: 2319: handle 8211(vscsi0:8):Completing reset (0 outstanding commands)
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.971 cpu2:4183)VSCSI: 2519: handle 8212(vscsi0:9):Reset [Retries: 0/0]
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.971 cpu2:4183)WARNING: NMP: nmpDeviceTaskMgmt: Attempt to issue lun reset on device naa.6000d31000163c000000000000000014. This will clear any SCSI-2 reservations on the device.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.971 cpu2:4183)<6>qla4xxx 0000:0f:01.1: scsi7:0:1:40: DEVICE RESET ISSUED.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.992 cpu6:4183)<6>qla4xxx 0000:0f:01.1: scsi(7:0:1:40): DEVICE RESET SUCCEEDED.
Mar 30 18:10:43 dr-esx1 vmkernel: 2:17:40:30.992 cpu6:4183)VSCSI: 2319: handle 8212(vscsi0:9):Completing reset (0 outstanding commands)
Mar 30 18:10:44 dr-esx1 vmkernel: 2:17:40:32.171 cpu2:4132)NetPort: 1157: disabled port 0x100000a
Mar 30 18:10:44 dr-esx1 vmkernel: 2:17:40:32.173 cpu2:4132)Net: 1847: disconnected client from port 0x100000a
Mar 30 18:10:45 dr-esx1 vmkernel: 2:17:40:32.602 cpu7:4311)CBT: 682: Disconnecting the cbt device 9001e-cbt with filehandle 589854
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.618 cpu1:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L33
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.618 cpu7:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L34
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.619 cpu7:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L35
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.619 cpu1:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L36
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.621 cpu1:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L38
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.623 cpu7:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L37
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.624 cpu6:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L34
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.624 cpu7:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L35
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.625 cpu1:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L33
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.625 cpu6:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L37
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.625 cpu2:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L36
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.626 cpu7:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L39
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.627 cpu1:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L38
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.627 cpu6:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L34
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.627 cpu7:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L40
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.628 cpu7:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L35
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.628 cpu6:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T1:L39
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.628 cpu3:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L33
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.630 cpu6:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L37
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.630 cpu3:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L36
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.630 cpu1:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L38
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.630 cpu6:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L40
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.631 cpu7:4191)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L34
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.631 cpu2:4190)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L35
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.631 cpu6:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba2:C0:T0:L39
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.635 cpu2:4193)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L33
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.635 cpu1:4192)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L37
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.635 cpu7:4189)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L39
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.635 cpu4:4195)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L36
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.635 cpu5:4196)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L38
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.636 cpu2:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T1:L40
Mar 30 18:15:15 dr-esx1 vmkernel: 2:17:45:02.640 cpu2:4194)WARNING: ScsiCore: 1354: Power-on Reset occurred on vmhba3:C0:T0:L40
Mar 30 22:49:49 dr-esx1 vmkernel: 2:22:19:35.864 cpu3:4111)World: vm 4331: 1534: Starting world vmware-vmx with flags 4
Mar 30 22:49:50 dr-esx1 vmkernel: 2:22:19:37.574 cpu7:4331)World: vm 4332: 1534: Starting world vmm0:VMDRHT1 with flags 4008
Mar 30 22:49:50 dr-esx1 vmkernel: 2:22:19:37.578 cpu7:4331)Sched: vm 4332: 6047: Adding world 'vmm0:VMDRHT1', group 'host/user/pool6', cpu: shares=-3 min=0 minLimit=-1 max=-1, mem: shares=-3 min=0 minLimit=-1 max=-1
Mar 30 22:49:50 dr-esx1 vmkernel: 2:22:19:37.578 cpu7:4331)WARNING: MemSched: 12625: Non-overhead memory reservation for vmx user-world (worldID=4331) is greater than desired minimum amount of 57344 KB (min=57344 KB, reservedOverhead=0 KB, totalReserved=68764 KB)
Mar 30 22:49:50 dr-esx1 vmkernel: 2:22:19:37.578 cpu7:4331)Sched: vm 4332: 6066: renamed group 65 to vm.4331
Mar 30 22:49:50 dr-esx1 vmkernel: 2:22:19:37.578 cpu7:4331)Sched: vm 4332: 6083: group 65 is located under group 51
Mar 30 22:49:50 dr-esx1 vmkernel: 2:22:19:37.586 cpu7:4331)Swap: vm 4332: 2325: extending swap to 7954432 KB
Mar 30 22:49:50 dr-esx1 vmkernel: 2:22:19:37.628 cpu7:4331)World: vm 4334: 1534: Starting world vmm1:VMDRHT1 with flags 4008
Mar 30 22:49:53 dr-esx1 vmkernel: 2:22:19:40.431 cpu7:4331)CBT: 1029: Created device 78021-cbt for cbt driver with filehandle 491553
Mar 30 22:49:53 dr-esx1 vmkernel: 2:22:19:40.633 cpu7:4331)VSCSI: 3482: handle 8213(vscsi0:0):Using sync mode due to sparse disks
Mar 30 22:49:53 dr-esx1 vmkernel: 2:22:19:40.633 cpu7:4331)VSCSI: 3523: handle 8213(vscsi0:0):Creating Virtual Device for world 4332 (FSS handle 1179683)
Mar 30 22:49:54 dr-esx1 vmkernel: 2:22:19:40.824 cpu7:4331)WARNING: MemSched: vm 4331: 5199: Cannot reduce reservation by 5094 pages (total reservation: 57828 pages, consumed reservation: 53442 pages)
Mar 30 22:49:54 dr-esx1 vmkernel: 2:22:19:40.884 cpu1:4332)VMMVMKCall: 255: Received INIT from world 4332
Mar 30 22:49:54 dr-esx1 vmkernel: 2:22:19:41.060 cpu1:4332)LSI: 1895: LSI: Initialized rings for scsi0 async=1, record=0 replay=0
Mar 30 22:49:54 dr-esx1 vmkernel: 2:22:19:41.152 cpu5:4334)VMMVMKCall: 255: Received INIT from world 4334
Mar 30 22:49:54 dr-esx1 vmkernel: 2:22:19:41.472 cpu1:4332)Net: 1421: connected VMDRHT1 eth0 to VMNetwork-prod, portID 0x100000b
Mar 30 22:49:54 dr-esx1 vmkernel: 2:22:19:41.479 cpu1:4332)NetPort: 982: enabled port 0x100000b with mac 00:00:00:00:00:00
Mar 30 22:49:54 dr-esx1 vmkernel: 2:22:19:41.480 cpu1:4332)Net: 1421: connected VMDRHT1 eth0 to VMNetwork-prod, portID 0x100000c
Mar 30 22:49:54 dr-esx1 vmkernel: 2:22:19:41.587 cpu1:4332)NetPort: 982: enabled port 0x100000c with mac 00:00:00:00:00:00
Mar 30 22:49:56 dr-esx1 vmkernel: 2:22:19:43.740 cpu2:4332)VSCSI: 2245: handle 8213(vscsi0:0):Reset request on FSS handle 1179683 (0 outstanding commands)
Mar 30 22:49:56 dr-esx1 vmkernel: 2:22:19:43.740 cpu6:4183)VSCSI: 2519: handle 8213(vscsi0:0):Reset [Retries: 0/0]
Mar 30 22:49:56 dr-esx1 vmkernel: 2:22:19:43.740 cpu6:4183)VSCSI: 2319: handle 8213(vscsi0:0):Completing reset (0 outstanding commands)
Mar 30 22:50:29 dr-esx1 vmkernel: 2:22:20:16.003 cpu2:4332)VSCSI: 2245: handle 8213(vscsi0:0):Reset request on FSS handle 1179683 (0 outstanding commands)
Mar 30 22:50:29 dr-esx1 vmkernel: 2:22:20:16.004 cpu6:4183)VSCSI: 2519: handle 8213(vscsi0:0):Reset [Retries: 0/0]
Mar 30 22:50:29 dr-esx1 vmkernel: 2:22:20:16.004 cpu6:4183)VSCSI: 2319: handle 8213(vscsi0:0):Completing reset (0 outstanding commands)
Mar 30 22:54:49 dr-esx1 vmkernel: 2:22:24:35.900 cpu5:4112)World: vm 4338: 1534: Starting world vmware-vmx with flags 4
Mar 30 22:54:52 dr-esx1 vmkernel: 2:22:24:38.795 cpu1:4338)FS3: 3001: [Req mode: 1] Checking liveness of [type 10c00001 offset 20365312 v 867, hb offset 3977216
Mar 30 22:54:52 dr-esx1 vmkernel: gen 9, mode 1, owner 4d8778db-e4cc766c-ee64-0015176fbfba mtime 2136] on vol 'iscsi-datastore3'
Mar 30 22:54:52 dr-esx1 vmkernel: 2:22:24:38.830 cpu0:4338)FS3: 1209: Clearing stale owner for lock [type 10c00001 offset 20365312 v 867, hb offset 3977216
Mar 30 22:54:52 dr-esx1 vmkernel: gen 9, mode 1, owner 4d8778db-e4cc766c-ee64-0015176fbfba mtime 2136] on volume 'iscsi-datastore3'
Mar 30 22:54:54 dr-esx1 vmkernel: 2:22:24:41.677 cpu7:4338)World: vm 4339: 1534: Starting world vmm0:VMDRISA with flags 4008
Mar 30 22:54:54 dr-esx1 vmkernel: 2:22:24:41.680 cpu7:4338)Sched: vm 4339: 6047: Adding world 'vmm0:VMDRISA', group 'host/user/pool6', cpu: shares=-3 min=0 minLimit=-1 max=-1, mem: shares=-3 min=0 minLimit=-1 max=-1
Mar 30 22:54:54 dr-esx1 vmkernel: 2:22:24:41.680 cpu7:4338)WARNING: MemSched: 12625: Non-overhead memory reservation for vmx user-world (worldID=4338) is greater than desired minimum amount of 57344 KB (min=57344 KB, reservedOverhead=0 KB, totalReserved=68708 KB)
Mar 30 22:54:54 dr-esx1 vmkernel: 2:22:24:41.680 cpu7:4338)Sched: vm 4339: 6066: renamed group 66 to vm.4338
Mar 30 22:54:54 dr-esx1 vmkernel: 2:22:24:41.680 cpu7:4338)Sched: vm 4339: 6083: group 66 is located under group 51
Mar 30 22:54:54 dr-esx1 vmkernel: 2:22:24:41.725 cpu7:4338)Swap: vm 4339: 2325: extending swap to 4194304 KB
Mar 30 22:54:55 dr-esx1 vmkernel: 2:22:24:41.868 cpu7:4338)World: vm 4341: 1534: Starting world vmm1:VMDRISA with flags 4008
Mar 30 22:54:55 dr-esx1 vmkernel: 2:22:24:41.870 cpu7:4338)World: vm 4342: 1534: Starting world vmm2:VMDRISA with flags 4008
Mar 30 22:54:55 dr-esx1 vmkernel: 2:22:24:41.871 cpu7:4338)World: vm 4343: 1534: Starting world vmm3:VMDRISA with flags 4008
Mar 30 22:54:58 dr-esx1 vmkernel: 2:22:24:45.012 cpu6:4338)CBT: 1029: Created device 72802a-cbt for cbt driver with filehandle 7503914
Mar 30 22:54:58 dr-esx1 vmkernel: 2:22:24:45.540 cpu6:4338)VSCSI: 3523: handle 8214(vscsi0:0):Creating Virtual Device for world 4339 (FSS handle 11567147)
Mar 30 22:54:59 dr-esx1 vmkernel: 2:22:24:46.434 cpu6:4338)WARNING: MemSched: vm 4338: 5199: Cannot reduce reservation by 5309 pages (total reservation: 43152 pages, consumed reservation: 37969 pages)
Mar 30 22:55:00 dr-esx1 vmkernel: 2:22:24:46.806 cpu1:4339)VMMVMKCall: 255: Received INIT from world 4339
Mar 30 22:55:00 dr-esx1 vmkernel: 2:22:24:47.040 cpu1:4339)LSI: 1895: LSI: Initialized rings for scsi0 async=1, record=0 replay=0
Mar 30 22:55:00 dr-esx1 vmkernel: 2:22:24:47.079 cpu1:4345)Net: 1421: connected VMDRISA eth0 to VMNetwork-dmz, portID 0x4000005
Mar 30 22:55:00 dr-esx1 vmkernel: 2:22:24:47.081 cpu1:4345)Net: 1421: connected VMDRISA eth0 to VMNetwork-prod, portID 0x100000d
Mar 30 22:55:00 dr-esx1 vmkernel: 2:22:24:47.186 cpu3:4341)VMMVMKCall: 255: Received INIT from world 4341
Mar 30 22:55:00 dr-esx1 vmkernel: 2:22:24:47.210 cpu7:4342)VMMVMKCall: 255: Received INIT from world 4342
Mar 30 22:55:00 dr-esx1 vmkernel: 2:22:24:47.228 cpu6:4343)VMMVMKCall: 255: Received INIT from world 4343
Mar 30 22:55:12 dr-esx1 vmkernel: 2:22:24:59.664 cpu1:4339)VSCSI: 2245: handle 8214(vscsi0:0):Reset request on FSS handle 11567147 (0 outstanding commands)
Mar 30 22:55:12 dr-esx1 vmkernel: 2:22:24:59.666 cpu2:4183)VSCSI: 2519: handle 8214(vscsi0:0):Reset [Retries: 0/0]
Mar 30 22:55:12 dr-esx1 vmkernel: 2:22:24:59.668 cpu2:4183)VSCSI: 2319: handle 8214(vscsi0:0):Completing reset (0 outstanding commands)
Mar 30 22:56:46 dr-esx1 vmkernel: 2:22:26:33.051 cpu7:4339)VSCSI: 2245: handle 8214(vscsi0:0):Reset request on FSS handle 11567147 (0 outstanding commands)
Mar 30 22:56:46 dr-esx1 vmkernel: 2:22:26:33.059 cpu3:4183)VSCSI: 2519: handle 8214(vscsi0:0):Reset [Retries: 0/0]
Mar 30 22:56:46 dr-esx1 vmkernel: 2:22:26:33.060 cpu3:4183)VSCSI: 2319: handle 8214(vscsi0:0):Completing reset (0 outstanding commands)
Mar 30 22:57:29 dr-esx1 vmkernel: 2:22:27:16.136 cpu1:4348)NetPort: 982: enabled port 0x4000005 with mac 00:50:56:b0:46:42
Mar 30 22:57:32 dr-esx1 vmkernel: 2:22:27:19.353 cpu1:4348)NetPort: 982: enabled port 0x100000d with mac 00:50:56:b0:6f:70
Mar 30 23:35:26 dr-esx1 vmkernel: 2:23:05:13.379 cpu4:4346)NetPort: 1157: disabled port 0x100000d
Mar 30 23:35:28 dr-esx1 vmkernel: 2:23:05:15.036 cpu4:4348)NetPort: 982: enabled port 0x100000d with mac 00:50:56:b0:6f:70
Mar 30 23:35:59 dr-esx1 vmkernel: 2:23:05:46.197 cpu2:4345)NetPort: 1157: disabled port 0x4000005
Mar 30 23:36:02 dr-esx1 vmkernel: 2:23:05:49.062 cpu4:4348)NetPort: 982: enabled port 0x4000005 with mac 00:50:56:b0:46:42
Mar 30 23:41:41 dr-esx1 vmkernel: 2:23:11:28.449 cpu7:4346)NetPort: 1157: disabled port 0x4000005
Mar 30 23:41:42 dr-esx1 vmkernel: 2:23:11:28.616 cpu6:4346)NetPort: 1157: disabled port 0x100000d
Mar 30 23:41:43 dr-esx1 vmkernel: 2:23:11:30.396 cpu6:4132)Net: 1847: disconnected client from port 0x4000005
Mar 30 23:41:43 dr-esx1 vmkernel: 2:23:11:30.396 cpu6:4132)Net: 1847: disconnected client from port 0x100000d
Mar 30 23:41:44 dr-esx1 vmkernel: 2:23:11:30.773 cpu1:4338)CBT: 682: Disconnecting the cbt device 72802a-cbt with filehandle 7503914
Avatar of Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Rollback the firmware on the storage, if you can.
Avatar of Akibia

ASKER

Rolling back the SAN firmware is not an option. I spoke with Compellent and they are confident that the upgrade should not have caused this. I can only assume that it has to do with the HBA's at this point. I'm going to look at updating the firmware of the qlogic card now.

I examined post again and found this warning:
Restoring SCSI Paths [Date/Time 'StorageInfo' warning] Unable to name LUN mpx.vmhba32:C0:T0:L0. Cannot set display name on this device. Unable to guarantee name will not change across reboots or media change.

That is for the ISCSI software initiator that is not in use. It was enabled before I put the HBA's in about 1.5 years ago. Even with the warning, the process got an OK and there are no Failures in the post at all.
oops, forgot to hit send last night...

Do you know what the mode and pathing policy are set to?  Looks like it should be using VMW_SATP_DEFAULT_AA      and VMW_PSP_FIXED .

Do you have both ports on the qlogic hooked up on the HBA?
Avatar of Akibia

ASKER

All pathing policies were set to: VMW_SATP_DEFAULT_AA and (FIXED)VMware. I changed them to VMW_SATP_DEFAULT_AA and VMW_PSP_FIXED and it did not change any performance. I then changed it to VMW_SATP_DEFAULT_AA and Round Robin (VMware). That had better results. I am now able to start and run guests configured with esx datastore connections. The guests configured with RDM's though are unusable. Once started they have horrible performance and kill the load average on esxtop and top.
have you looked at esxtop, yet?  go to d(isk) measurements and look at latencies (hit f to change the fields).  the kavg should be close to or less than 1.0, davg is best when it's under 20 and gavg is the sum of the kavg and davg.
Avatar of Akibia

ASKER

I guess I wasn't clear... sorry for that. In esxtop the stats are horrible. Below is captured with only one guest running it has Physical RDM's configured (which appears to be the issue).

login as: root
Using keyboard-interactive authentication.
Password:
Last login: Wed Apr  6 10:40:28 2011
[root@dr-esx1 ~]# esxtop
10:57:03am up 29 min, 137 worlds; CPU load average: 0.63, 0.65, 0.59

DEVICE                               DQLEN WQLEN ACTV QUED %USD  LOAD DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd DAVG/rd KAVG/rd GA
mpx.vmhba1:C0:T0:L0                      1     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
mpx.vmhba32:C0:T0:L0                     1     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c000000000000000004    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c000000000000000005    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c000000000000000006    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c000000000000000007    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c000000000000000008    32     -    0    0    0  0.00     5.18     0.50     5.68     0.15   15.11    0.50
naa.6000d31000163c000000000000000009    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c00000000000000000a    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c00000000000000000c    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c00000000000000000d    32     -    0    0    0  0.00   103.13     1.50   104.63     0.01  103.13    1.50  1
naa.6000d31000163c00000000000000000e    32     -    0    0    0  0.00    57.53     0.03    57.55     0.01   57.53    0.03
naa.6000d31000163c00000000000000000f    32     -    0    0    0  0.00    92.55     0.03    92.58     0.01   92.55    0.03
naa.6000d31000163c000000000000000010    32     -    0    0    0  0.00    13.16     0.51    13.68     0.01   13.17    0.51
naa.6000d31000163c000000000000000011    32     -    0    0    0  0.00    46.70     0.49    47.19     0.12   46.70    0.49
naa.6000d31000163c000000000000000012    32     -    0    0    0  0.00    30.07     0.97    31.03     0.28   30.07    0.97
naa.6000d31000163c000000000000000013    32     -    0    0    0  0.00    15.59     0.73    16.32     0.12   15.59    0.73
naa.6000d31000163c000000000000000014    32     -    0    0    0  0.00    43.61     0.96    44.58     0.17   43.61    0.96
naa.6000d31000163c000000000000000015    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c000000000000000018    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c000000000000000019    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c00000000000000001a    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c00000000000000001c    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c00000000000000001d    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c00000000000000001e    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c00000000000000001f    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c000000000000000020    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c000000000000000021    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c000000000000000022    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c000000000000000023    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c000000000000000025    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6000d31000163c000000000000000026    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
naa.6001e4f0363052000f9bc991050ed998   128     -    0    0    0  0.00     1.08     0.19     1.27     0.05    0.00    0.00
naa.6001e4f0363052000f9bc9910514b986    32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
t10.DP______BACKPLANE000000             32     -    0    0    0  0.00     0.00     0.00     0.00     0.00    0.00    0.00
ASKER CERTIFIED SOLUTION
Avatar of Akibia
Akibia

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Akibia

ASKER

No solution found... time dictated a complete rebuild.
"we could find a resolution."

what was the resolution for the Experts?
There is none, I have the same problem. I been working this problem off and on for the last few weeks, I have a HP BL460 G7 and Compellent ISCSI storage as soon as the HOST is put under a load of over 50% i get the messages everytime.  This has to be some type of  a bug !!!