Expected Netapp IOP Performance

Posted on 2015-01-09
Medium Priority
Last Modified: 2015-01-19
Dear All
I am looking for some feedback regarding benchmark testing on our new Netapp FAS2552 running 8.2RC17 Mode 7 with Vmware using Iscsi  

I have connected the Netapp to Vmware ESXi 5.5 with a 10gb connection from the SAN to the switch and from the switch to the host.  I set up iometre to  test using using 4k blocks in sequential reads with outstanding IOPs at 16. I have limited the sample size 100mb, this effectively is making a cache run from the SAN and I have confirmed the read has 100% hit rate.
.  I am surprised to see that the maximum number of iop,s is only 20,000 with less than 90 MB-s being utilised.  I would have thought if the Netapp was serving 100% from cache memory the iop,s level would be significantly higher.  
I have also benchmarked against a LUN from the Netapp directly into the Windows 2008 OS using the software initiator within the Windows operating system.  I am seeing very similar performance.  Given the fact that everything is being served from cache, does it seem right for iop,s at this level?  Your thoughts would be greatly appreciated.
Question by:James-Sillett
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
  • 2
LVL 123

Accepted Solution

Andrew Hancock (VMware vExpert / EE MVE^2) earned 2000 total points
ID: 40541396
(as a point, you should be on 8.2.2. GA at least now!)

How many IOPS do you require ?

do you have jumbo frames enabled?

do you have multipath enabled?

have you applied the recommended settings to the ESXi hosts as per NetApp guidelines.

What disks do you have? how are they configured ?

Author Comment

ID: 40541414
The Filler was only delivered direct from Netapp a couple of days ago, I assumed the latest version of ONYSP  would of been installed at delivery I will look to update before I put this into production, a bit poor on Netapps part.
The required IOPs is not relevant, I am stress testing each part of the setup for benchmarking purposes. I do  not have jumbo frames on at the moment, I will when we go into production however  given the test I am doing is at 4k blocks, and the network path is less then 10% utilised I would not of thought this would of made a difference to this stress test. All guidelines have been followed from the different white papers,. The disk are again not relevant as this is a 100% read only, which is coming directly from cache, ie 100% cache (sysstat) hit almost 0% disk utilisation, this is the reason why I would expected higher IOPS. The points of this these test is to understand the limitation of each part of the setup.
LVL 123
ID: 40541429
We've had the same with us, direct delivered from NetApp without the latest builds! (always!).

We've also trashed the configuration, and set it up again, and also you may want to switch to NFS far better performance than iSCSI, as there is additional overhead, and NetApp now recommends and pushes NFS rather than iSCSI.

Same inside the Filer, NFS is lower overhead, than using iSCSI.

Do you have cache ? SSDs ? (e.g. hybrid storage, or aggregate?)
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!


Author Comment

ID: 40541449
I've been pondering NFS for a while, it just a massive change for our infrastructure but ill do some performance benchmarking and see if it make any difference. We have not got any SSD it a straight aggregate, but as per a white paper on benchmarking I kept the lun drive size below the 18GB of ECC ram so that IO meter would pull everything from cache, so I would expect to be getting the same IOPs or more than I get from my cheap £100 desktop ssd
LVL 123
ID: 40541478
When we queried NetApp support, when we benchmarked our NetApp Filers using IOP meter, and we don't bother any more, they said it would take 2 weeks before the IOPS stabilised!

because of the cache, that what was said!

Are you disks SATA or SAS ?
LVL 22

Expert Comment

ID: 40546320
Benchmark testing should reflect real world situations as closely as possible.

Purely theoretical benchmarks add little value and are essentially a waste of time.

Consider the whole stack that is involved: not only the Netapp, but the networking infrastructure, the physical server, the VMWare ESX software stack and finally the OS on which you are testing.

All of the components can introduce latency and it is the latency that is limiting performance (average 0.69 ms with peaks of 118ms according to your screenshot).

Latency especially matters when using small blocks and no matter how big your bandwidth is, you can never fill up the pipe if latency is high. Both endpoints must wait until outstanding traffic is confirmed before sending more blocks.

Try testing with larger block sizes, 64K, 128K, 512K and you will see that throughput increases and IOPS decrease.

But as I said at the start, doing synthetic benchmarks are mostly a waste of time and give you no idea about real world performance.

Author Comment

ID: 40546349
I agree with what you say, however the point of the tests, are to mathematical  worked out what each component is capable of achieving, then stress test each part to make sure the mathematical performance of the COMPONET matches what is expected. Knowing what each aspect can do, ie Host, NIC, switch, SAN means that when the system is live you can monitor each of those components to see how the respond to real world.
LVL 22

Expert Comment

ID: 40546405
Still you only measure the performance of the entire stack, not of the Netapp alone. Perhaps most of the latency comes from VMWare, who knows? And it is the latency that causes bottlenecks on protocol level.

Featured Post

Building an interactive eFuture classroom

Watch and learn how ATEN provided a total control system solution including seamless switching matrix switch, HDBaseT extenders, PDU, lighting control to build an interactive eFuture classroom.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The question appears often enough, how do I transfer my data from my old server to the new server while preserving file shares, share permissions, and NTFS permisions.  Here are my tips for handling such a transfer.
The business world is becoming increasingly integrated with tech. It’s not just for a select few anymore — but what about if you have a small business? It may be easier than you think to integrate technology into your small business, and it’s likely…
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…
Suggested Courses

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question