Solved

high disk queue lengths and latency on iSCSI volume used for file sharing - Netgear ReadyNAS 2120

Posted on 2014-03-25
7
2,598 Views
Last Modified: 2016-11-23
We have a Dell Equallogic P4000 with 16 x 600GB 15k SAS drives in a
RAID 50 configuration.  Separate LUNs for VMs, SQL DBs, and file data.  All LUNs are accessed via iSCSI on a separate dedicated network than LAN data.  Everything has been working well.  I started running low on space so on wanted to offload the file data to a different storage device.  I chose a Netgear ReadyNAS 2120.  I'm beginning to regret my choice.  There are four 2 TB 7200 RPM SATA drives in a RAID 10 config.  I have a single LUN configured for iSCSI.  I moved all the file share data off the Equallogic volume to the Netgear volume.  The same file server managing all the shared folders is in place.  It has a dedicated network interface for LAN traffic and one dedicated for iSCSI.  Ever since I did this, our clients are saying performance is horrendous.  Sometimes it takes 30 seconds to open a file, then the next file opens right away.  Seems to be intermittent.  But one thing is certain, it's been really bad since moving data to the Netgear.  Did I just make a bad decision with the Netgear?  Before I migrated, I tested by copying large files back and forth and comparing that with copying to and from the equallogic.  File copy speeds were identical.  I know the drive RPM is half, and going from SAS to SATA, but I wouldn't expect it to be this bad for simple file sharing.  I have opened a support case with Netgear - they are still analyzing the logs.  The NIC settings are the same as they were when connected to the iSCSI volume on the Equallogic.  Should I try changing settings like MTU, large send offload, TCP offload, etc?  Or should I just chalk it up to a low quality NAS device?  Please let me know if you have any ideas for improving performance.
0
Comment
Question by:ITLighthouse
7 Comments
 
LVL 30

Assisted Solution

by:pgm554
pgm554 earned 167 total points
ID: 39953818
Almost sounds like cache settings.

Have a look:

http://kb.netgear.com/app/answers/detail/a_id/12242/~/how-to-optimize-%28-x86-%29readynas-performance

With that being said,SAS is a superior drive interface ,hands down.

The SAS interface can use Tagged Command Queuing which SATA does not do well.

http://en.wikipedia.org/wiki/Tagged_Command_Queuing

There are SAS 7200 drives for just a few bucks more than the SATA's.

If you can afford them,replace with SAS.

You could turn on jumbo frames if you need a speed burst too.
0
 
LVL 76

Assisted Solution

by:arnold
arnold earned 333 total points
ID: 39954024
Does it make a difference whether you open an Office document or a PDF or non-office document.  Office documents create a lock file on the NAS while other apps do not.

Is the switch on for the iscsi managed such that you can see whether it has resource contention on the network?

Not clear why you are using a SHARE in an iscsi setup versus functioning as a CIFS/SAMBA share?

Your setup is
server iscsi to storage
share files to workstations
?
0
 
LVL 1

Author Comment

by:ITLighthouse
ID: 39954170
Thank you for the responses.  
PGM554:  Thanks- I have stumbled across that netgear document already.  But a lot of those settings are not available in the OS on this particular device.

Arnold:  I like having a Windows server manage the shared folders.  It gives me the most control over domain access permissions, performance monitoring, and backups.  I realize it's another layer to worry about when troubleshooting, but also gives me more flexibility and options for troubleshooting.

So, my setup is FILESERVER connects to iSCSI volume.  one NIC is for iSCSI traffic and the other is for LAN traffic.  FILESERVER sees the volume as a local disk.  Windows manages the shared folders.  Terminal servers access files through \\FILESERVER\Share.  There's a whole range of file types.  Everything from Word, Excel, PDF, to medical software accessing scanned charts attached to patient records.  It's very hard to reproduce.  I've tried opening several types of files in several folders.  The majority of them open instantaneously, but then you get the one that takes 30 seconds to open for some reason.  Then try the same file again and it opens immediately.  This was all working fine when the iSCSI volume was on the Equallogic.  I'm thinking the Netgear just isn't able to handle the load.  But I want to be sure before ditching that as an option.  Netgear support is talking about trying a beta firmware - I'm not keen on that.  From the searching I've done since having this problem, many people complain about performance issues with Netgear in general.
0
6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

 
LVL 76

Assisted Solution

by:arnold
arnold earned 333 total points
ID: 39954870
Not sure what options you have besides the windows systems, using SNMP you can try polling the local server interface using Cacti.net then you can see whether the issue occurs during peak times when the iscsi dedicated network is approaching high utilization.

have a look at sqlio http://www.microsoft.com/en-us/download/details.aspx?id=20163
This is a tool that evaluates the iscsi/storage for suitability for sql.
You can look at the performance.

I am unfamiliar with the mode by which the RAID is managed on the Netgear i.e. whether this is a software raid (linux OS with raid) and when ........

You could try on the server running a dir command on the top level of the Netgear LUN resource to see whether that will continually maintain ......
0
 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 39955802
What does your new network setup look like. How many and what speed are your network connections?
0
 
LVL 1

Accepted Solution

by:
ITLighthouse earned 0 total points
ID: 39995515
I finally figured out the problem.  We ended up purchasing a Netgear ReadyNAS 4220 to replace the 2120 thinking that it was a more robust device and would provide better performance.  After installing it, I did a test copy and to my dismay, experienced the exact same lag times as the 2120.  After much testing, going over settings, trying different RAID levels and netgear firmware, and beating my head against the wall,  I finally stumbled across a setting that resolved the issue.  In Disk Management, properties of the connected iSCSI disk, under the policies tab, I checked the box to turn off Windows write-cache buffer flushing.  That immediately fixed the disk write lag times.  No reboot required.  Both the 4220 and 2120 does not allow me to disable write caching, and is enabled by default.  But it does let me disable buffer flushing.  Previous Netgear devices as well as the Equallogic have write caching and buffer flushing disabled by default.  Now large file copies are steady and don't hang for 30 seconds during the copy.  Previously, disk response times would spike to over 3000 and sometimes as high as 40,000.  Now it does't go above 400ms and usually averages around 60ms.  

Although I ended up figuring it out by myself, I did appreciate your input!
0
 
LVL 1

Author Closing Comment

by:ITLighthouse
ID: 40003493
figured it out myself
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Before I go to far, let's explain HA (High Availability) and why you should consider it.  High availability is the mechanism used to provide redundancy to any service at the same site and appears as a single service to the users of that service.  As…
I previously wrote an article addressing the use of UBCD4WIN and SARDU. All are great, but I have always been an advocate of SARDU. Recently it was suggested that I go back and take a look at Easy2Boot in comparison.
In this video, we discuss why the need for additional vertical screen space has become more important in recent years, namely, due to the transition in the marketplace of 4x3 computer screens to 16x9 and 16x10 screens (so-called widescreen format). …
With the advent of Windows 10, Microsoft is pushing a Get Windows 10 icon into the notification area (system tray) of qualifying computers. There are many reasons for wanting to remove this icon. This two-part Experts Exchange video Micro Tutorial s…

759 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now