We help IT Professionals succeed at work.

ESXi v5.5 communication issues with my iSCSI SAN - Dell md3200

gopher_49
gopher_49 asked
on
329 Views
Last Modified: 2018-05-24
My ESXi v5.5 Dell Update 3 server quit talking to my Dell md3200i iSCSI SAN.  This caused all VMs to stop responding...  The iSCSI SAN was deployed with best practice in regards to dedicated VLANs and vSwitches for each iSCSI port.  I'm using the  VMWare method for redundant pathing... Where do I start to troubleshoot what happened?  I downloaded the vmkernel logs.. I had errors pertaining to performance and not being able to talk to my iSCSI SAN.  Now I just see logs like below... When I see these things are working just fine.. But..  A day ago I had other logs pertaining to it not being cable to communicate with the SAN.  Should I maybe update the firmware on my Dell md3200i?

2017-03-06T20:52:37.456Z cpu6:34103)ScsiDeviceIO: 2363: Cmd(0x412e807f3b00) 0x4d, CmdSN 0x22 from world 34103 to dev "naa.6001c230d9b57b00164feeed1031d490" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2017-03-06T20:52:37.456Z cpu6:34103)ScsiDeviceIO: 2363: Cmd(0x412e807f3b00) 0x1a, CmdSN 0x23 from world 34103 to dev "naa.6001c230d9b57b00164feeed1031d490" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2017-03-06T20:59:27.835Z cpu6:32795)NMP: nmp_ThrottleLogForDevice:2349: Cmd 0x1a (0x412e80819800, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2017-03-07T14:19:28.194Z cpu5:32794)WARNING: ScsiDeviceIO: 1223: Device naa.6842b2b00012d53300008bd957dbb540 performance has deteriorated. I/O latency increased from average value of 24108 microseconds to 602199 microseconds.
2017-03-07T14:19:29.808Z cpu1:36047)ScsiDeviceIO: 1203: Device naa.6842b2b00012d53300008bd957dbb540 performance has improved. I/O latency reduced from 602199 microseconds to 119635 microseconds.
2017-03-07T14:19:33.209Z cpu6:36250)ScsiDeviceIO: 1203: Device naa.6842b2b00012d53300008bd957dbb540 performance has improved. I/O latency reduced from 119635 microseconds to 47817 microseconds.
Comment
Watch Question

Grant BerezanSenior Network Engineer
CERTIFIED EXPERT

Commented:
For intermittent communication issues with a SAN via iSCSI, I would be more inclined to look at the switches involved. Do you have paired switches , splitting your communication paths between vSwitches? Perhaps what you are seeing is a imminent failure in one switch, and when the traffic hits a certain load, it breaks. The latency messages are common, as the more traffic your SAN gets, the higher the I/O. Also, have you updated all your VMWare hosts to the latest patch version for 5.5?

Author

Commented:
My ESXi hosts are running Dell update 3.  I think that's the most current .  My switches seem to be fine and other host are not having issues .  Each iSCSI port is in its own VLAN and dedicated vswitch.  I think the md3200 firmware is old.  Also.  It's running raid 6 and one drive is failed
Senior Network Engineer
CERTIFIED EXPERT
Commented:
This problem has been solved!
(Unlock this solution with a 7-day Free Trial)
UNLOCK SOLUTION

Author

Commented:
I'll post the vmkernel logs from the day I had issues.  I think they have what you're looking for.  Does the vmkernel log file store entries after a reboot?  Are those secure to post on here?

Author

Commented:
OrcaKnight,

I looked at the managed switch ports that the ESXi host and SAN are plugged into.  I don't see any errors on any of the switch ports.  Which ESXi log should I review to see if the NIC is having issues?

Author

Commented:
Not sure to exactly what was wrong but there was a communication issue between the NIC and the iSCSI network.

Thanks.

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions