We help IT Professionals succeed at work.
Get Started

Severe disk latency on NFS stores from EMC VNXe

Brian_B
Brian_B asked
on
161 Views
Last Modified: 2015-10-13
We have 4 1TB NFS stores carved out on a VNXe3200 . these stores are being used by two VMware servers.  There's also LUNs setup on the EMC being used by a separate physical Windows Server.

2 of the 4 NFS stores are suffering from sever latency.  Every few seconds, could be a couple hundred MS but we've seen 20k+ MS.  Best we can tell the to other two NFS stores are fine.  and the LUNs are fine.

Three different EMC techs have looks at the EMC SAN and say it's fine.  they pulled another set of logs and such and are analyzing.  Only connection is that they are both using SPA - but they're not seeing issues with SPB nor are the LUNs showing latency.

NFS no iSCSI so most issues with framing and such don't apply.  besides everything is setup the same way and only 2 of the stores are effected.  had one VMware tech look - in VMware didn't see anything.  have case open with VMware.  so far no one can tell me anything of interest.  Some of the more common issues like the IOPs bug in VMware bug we checked for and eliminated - all VMs set for unlimited IOPs.  

There's 5 VMs using these two stores and they are NOT at all taxing the CPU, RAM or storage.  all very secondary servers.  my most busy server is on one of the other stores and it's fine.  

any ideas?  since myself, my team, 2 EMC engineers, and 2 techs from the installed that put in the SAN have looked at this - I think we can forgo some of the "easy" items unless it's really easily overlooked.  The VMs used to be connect to a different SAN using iSCSI but that was late last year.  this issue seems to be fairly new - like weeks but seems to be getting worse.
Comment
Watch Question
IT Systems Director, DevOps Engineer.
CERTIFIED EXPERT
Top Expert 2014
Commented:
This problem has been solved!
Unlock 2 Answers and 8 Comments.
See Answers
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE