Solved

Expected Netapp IOP Performance

Posted on 2015-01-09
8
418 Views
Last Modified: 2015-01-19
Dear All
I am looking for some feedback regarding benchmark testing on our new Netapp FAS2552 running 8.2RC17 Mode 7 with Vmware using Iscsi  

I have connected the Netapp to Vmware ESXi 5.5 with a 10gb connection from the SAN to the switch and from the switch to the host.  I set up iometre to  test using using 4k blocks in sequential reads with outstanding IOPs at 16. I have limited the sample size 100mb, this effectively is making a cache run from the SAN and I have confirmed the read has 100% hit rate.
.  I am surprised to see that the maximum number of iop,s is only 20,000 with less than 90 MB-s being utilised.  I would have thought if the Netapp was serving 100% from cache memory the iop,s level would be significantly higher.  
I have also benchmarked against a LUN from the Netapp directly into the Windows 2008 OS using the software initiator within the Windows operating system.  I am seeing very similar performance.  Given the fact that everything is being served from cache, does it seem right for iop,s at this level?  Your thoughts would be greatly appreciated.
iometer.png
0
Comment
Question by:James-Sillett
  • 3
  • 3
  • 2
8 Comments
 
LVL 117

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
Comment Utility
(as a point, you should be on 8.2.2. GA at least now!)

How many IOPS do you require ?

do you have jumbo frames enabled?

do you have multipath enabled?

have you applied the recommended settings to the ESXi hosts as per NetApp guidelines.

What disks do you have? how are they configured ?
0
 

Author Comment

by:James-Sillett
Comment Utility
Hi,
The Filler was only delivered direct from Netapp a couple of days ago, I assumed the latest version of ONYSP  would of been installed at delivery I will look to update before I put this into production, a bit poor on Netapps part.
The required IOPs is not relevant, I am stress testing each part of the setup for benchmarking purposes. I do  not have jumbo frames on at the moment, I will when we go into production however  given the test I am doing is at 4k blocks, and the network path is less then 10% utilised I would not of thought this would of made a difference to this stress test. All guidelines have been followed from the different white papers,. The disk are again not relevant as this is a 100% read only, which is coming directly from cache, ie 100% cache (sysstat) hit almost 0% disk utilisation, this is the reason why I would expected higher IOPS. The points of this these test is to understand the limitation of each part of the setup.
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
We've had the same with us, direct delivered from NetApp without the latest builds! (always!).

We've also trashed the configuration, and set it up again, and also you may want to switch to NFS far better performance than iSCSI, as there is additional overhead, and NetApp now recommends and pushes NFS rather than iSCSI.

Same inside the Filer, NFS is lower overhead, than using iSCSI.

Do you have cache ? SSDs ? (e.g. hybrid storage, or aggregate?)
0
 

Author Comment

by:James-Sillett
Comment Utility
I've been pondering NFS for a while, it just a massive change for our infrastructure but ill do some performance benchmarking and see if it make any difference. We have not got any SSD it a straight aggregate, but as per a white paper on benchmarking I kept the lun drive size below the 18GB of ECC ram so that IO meter would pull everything from cache, so I would expect to be getting the same IOPs or more than I get from my cheap £100 desktop ssd
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
When we queried NetApp support, when we benchmarked our NetApp Filers using IOP meter, and we don't bother any more, they said it would take 2 weeks before the IOPS stabilised!

because of the cache, that what was said!

Are you disks SATA or SAS ?
0
 
LVL 21

Expert Comment

by:robocat
Comment Utility
Benchmark testing should reflect real world situations as closely as possible.

Purely theoretical benchmarks add little value and are essentially a waste of time.

Consider the whole stack that is involved: not only the Netapp, but the networking infrastructure, the physical server, the VMWare ESX software stack and finally the OS on which you are testing.

All of the components can introduce latency and it is the latency that is limiting performance (average 0.69 ms with peaks of 118ms according to your screenshot).

Latency especially matters when using small blocks and no matter how big your bandwidth is, you can never fill up the pipe if latency is high. Both endpoints must wait until outstanding traffic is confirmed before sending more blocks.

Try testing with larger block sizes, 64K, 128K, 512K and you will see that throughput increases and IOPS decrease.

But as I said at the start, doing synthetic benchmarks are mostly a waste of time and give you no idea about real world performance.
0
 

Author Comment

by:James-Sillett
Comment Utility
I agree with what you say, however the point of the tests, are to mathematical  worked out what each component is capable of achieving, then stress test each part to make sure the mathematical performance of the COMPONET matches what is expected. Knowing what each aspect can do, ie Host, NIC, switch, SAN means that when the system is live you can monitor each of those components to see how the respond to real world.
0
 
LVL 21

Expert Comment

by:robocat
Comment Utility
Still you only measure the performance of the entire stack, not of the Netapp alone. Perhaps most of the latency comes from VMWare, who knows? And it is the latency that causes bottlenecks on protocol level.
0

Featured Post

Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

Join & Write a Comment

Suggested Solutions

How to update Firmware and Bios in Dell Equalogic PS6000 Arrays and Hard Disks firmware update.
This article is an update and follow-up of my previous article:   Storage 101: common concepts in the IT enterprise storage This time, I expand on more frequently used storage concepts.
Teach the user how to rename, unmount, delete and upgrade VMFS datastores. Open vSphere Web Client: Rename VMFS and NFS datastores: Upgrade VMFS-3 volume to VMFS-5: Unmount VMFS datastore: Delete a VMFS datastore:
Teach the user how to install log collectors and how to configure ESXi 5.5 for remote logging Open console session and mount vCenter Server installer: Install vSphere Core Dump Collector: Install vSphere Syslog Collector: Open vSphere Client: Config…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now