• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 3048
  • Last Modified:

high disk queue lengths and latency on iSCSI volume used for file sharing - Netgear ReadyNAS 2120

We have a Dell Equallogic P4000 with 16 x 600GB 15k SAS drives in a
RAID 50 configuration.  Separate LUNs for VMs, SQL DBs, and file data.  All LUNs are accessed via iSCSI on a separate dedicated network than LAN data.  Everything has been working well.  I started running low on space so on wanted to offload the file data to a different storage device.  I chose a Netgear ReadyNAS 2120.  I'm beginning to regret my choice.  There are four 2 TB 7200 RPM SATA drives in a RAID 10 config.  I have a single LUN configured for iSCSI.  I moved all the file share data off the Equallogic volume to the Netgear volume.  The same file server managing all the shared folders is in place.  It has a dedicated network interface for LAN traffic and one dedicated for iSCSI.  Ever since I did this, our clients are saying performance is horrendous.  Sometimes it takes 30 seconds to open a file, then the next file opens right away.  Seems to be intermittent.  But one thing is certain, it's been really bad since moving data to the Netgear.  Did I just make a bad decision with the Netgear?  Before I migrated, I tested by copying large files back and forth and comparing that with copying to and from the equallogic.  File copy speeds were identical.  I know the drive RPM is half, and going from SAS to SATA, but I wouldn't expect it to be this bad for simple file sharing.  I have opened a support case with Netgear - they are still analyzing the logs.  The NIC settings are the same as they were when connected to the iSCSI volume on the Equallogic.  Should I try changing settings like MTU, large send offload, TCP offload, etc?  Or should I just chalk it up to a low quality NAS device?  Please let me know if you have any ideas for improving performance.
0
ITLighthouse
Asked:
ITLighthouse
4 Solutions
 
pgm554Commented:
Almost sounds like cache settings.

Have a look:

http://kb.netgear.com/app/answers/detail/a_id/12242/~/how-to-optimize-%28-x86-%29readynas-performance

With that being said,SAS is a superior drive interface ,hands down.

The SAS interface can use Tagged Command Queuing which SATA does not do well.

http://en.wikipedia.org/wiki/Tagged_Command_Queuing

There are SAS 7200 drives for just a few bucks more than the SATA's.

If you can afford them,replace with SAS.

You could turn on jumbo frames if you need a speed burst too.
0
 
arnoldCommented:
Does it make a difference whether you open an Office document or a PDF or non-office document.  Office documents create a lock file on the NAS while other apps do not.

Is the switch on for the iscsi managed such that you can see whether it has resource contention on the network?

Not clear why you are using a SHARE in an iscsi setup versus functioning as a CIFS/SAMBA share?

Your setup is
server iscsi to storage
share files to workstations
?
0
 
ITLighthouseAuthor Commented:
Thank you for the responses.  
PGM554:  Thanks- I have stumbled across that netgear document already.  But a lot of those settings are not available in the OS on this particular device.

Arnold:  I like having a Windows server manage the shared folders.  It gives me the most control over domain access permissions, performance monitoring, and backups.  I realize it's another layer to worry about when troubleshooting, but also gives me more flexibility and options for troubleshooting.

So, my setup is FILESERVER connects to iSCSI volume.  one NIC is for iSCSI traffic and the other is for LAN traffic.  FILESERVER sees the volume as a local disk.  Windows manages the shared folders.  Terminal servers access files through \\FILESERVER\Share.  There's a whole range of file types.  Everything from Word, Excel, PDF, to medical software accessing scanned charts attached to patient records.  It's very hard to reproduce.  I've tried opening several types of files in several folders.  The majority of them open instantaneously, but then you get the one that takes 30 seconds to open for some reason.  Then try the same file again and it opens immediately.  This was all working fine when the iSCSI volume was on the Equallogic.  I'm thinking the Netgear just isn't able to handle the load.  But I want to be sure before ditching that as an option.  Netgear support is talking about trying a beta firmware - I'm not keen on that.  From the searching I've done since having this problem, many people complain about performance issues with Netgear in general.
0
Get quick recovery of individual SharePoint items

Free tool – Veeam Explorer for Microsoft SharePoint, enables fast, easy restores of SharePoint sites, documents, libraries and lists — all with no agents to manage and no additional licenses to buy.

 
arnoldCommented:
Not sure what options you have besides the windows systems, using SNMP you can try polling the local server interface using Cacti.net then you can see whether the issue occurs during peak times when the iscsi dedicated network is approaching high utilization.

have a look at sqlio http://www.microsoft.com/en-us/download/details.aspx?id=20163
This is a tool that evaluates the iscsi/storage for suitability for sql.
You can look at the performance.

I am unfamiliar with the mode by which the RAID is managed on the Netgear i.e. whether this is a software raid (linux OS with raid) and when ........

You could try on the server running a dir command on the top level of the Netgear LUN resource to see whether that will continually maintain ......
0
 
Gerald ConnollyCommented:
What does your new network setup look like. How many and what speed are your network connections?
0
 
ITLighthouseAuthor Commented:
I finally figured out the problem.  We ended up purchasing a Netgear ReadyNAS 4220 to replace the 2120 thinking that it was a more robust device and would provide better performance.  After installing it, I did a test copy and to my dismay, experienced the exact same lag times as the 2120.  After much testing, going over settings, trying different RAID levels and netgear firmware, and beating my head against the wall,  I finally stumbled across a setting that resolved the issue.  In Disk Management, properties of the connected iSCSI disk, under the policies tab, I checked the box to turn off Windows write-cache buffer flushing.  That immediately fixed the disk write lag times.  No reboot required.  Both the 4220 and 2120 does not allow me to disable write caching, and is enabled by default.  But it does let me disable buffer flushing.  Previous Netgear devices as well as the Equallogic have write caching and buffer flushing disabled by default.  Now large file copies are steady and don't hang for 30 seconds during the copy.  Previously, disk response times would spike to over 3000 and sometimes as high as 40,000.  Now it does't go above 400ms and usually averages around 60ms.  

Although I ended up figuring it out by myself, I did appreciate your input!
0
 
ITLighthouseAuthor Commented:
figured it out myself
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now