Tools to measure IOPS and disk latency

An engineer I was working with used IOMeter to measure IOPs on our T710.

Windows has a built in utility, called perfmon.   Iometer is NOT able to measure IOPs & throughput.  It is a load generator.  It can tell you how much I/O the program creates, and unless there is absolutely nothing else going on, then the numbers aren't correct.

IOmeter is the devil's work...  :-)

Exactly, iometer is often misinterpreted.  The numbers can't possibly be valid if the load is sent to a file system, either, even if nothing else is going on.  This is because of the additional overhead that NTFS has for logging, contiguous files, file system chunk size, just to name a few.

>Exactly, iometer is often misinterpreted
I have a screenshot of IOmeter reporting 180,000 IOPS. Off my laptop's 7200rpm SATA drive...

To the author -- what was the data collection tool the vendor supplied?     But the important thing you need to know is that you don't need a synthetic load generator like iometer or anything else.  You need to use the native O/S utility and measure latency and I/O queue depth WITH YOUR APPLICATIONS running.  Unless queue depth is consistently over 2, then you don't need to buy anything.  The queue depth says how often your programs are waiting for disk I/O to complete.  Low latency is nice, but throughput is irrelevant, as it is a function of block size.  

All you need to worry about is the queue depth measured over time.  When the queue depth is consistently less then two it means the storage is no longer a bottleneck.  You could have all top of the line million dollar SSD RAID array and it won't make your computer any faster then a USB-memory stick storage subsystem once the queue depth for your programs averages under 1.0.

If your vendor didn't explain that and is selling you by touting IOPs and throughput, then kick the vendor out.

Hello dlethe,  

First, I wasn't fair in my initial statement.  Yes the vendor wants me to buy equipment. The vendor suggested we do some analysis, which was reasonable.  Nothing has been decided on what to do and I have alot of questions.  We went through some analsysis and I was disturbed by the high latecny (page 3, output summar right hand column 800ms+).  I am trying to get a better handle on how to measure performance and what it means.

The two questions I really have are "What is the equipment performance?" and "How do I know?".  Currently,  I think things are ok,  because users don't complain.  This is a good measure but I want something more concrete.  To be simple are things green, yellow or red.  Some of this is subjective but I would like to setup performance metrics for Servers, Networks, Internet, etc..   so that I can have some grasp from a technical perspective if thinigs are "normal".

Having said that, could you, or anyone, provide solid advice based on experience, not something that I saw someone do.... and books are reading material is always good.  I really want to have the "how to" and "what it means".  For example I want to say we are traveling at 85mph verus we ware going fast and then to say for this environment 55mph is very good so 85mph is great, etc..


On another note, if 800ms latency is really bad what are the steps of fixing this issue.  Again, someone with experience who has done it and pointing me to "how to" is great.  My conversations with the vendor was I know we have issues because of the environment.  More specifically, we have the following environment:
Windows Server 2008R2; 4gb ram, raid 1 OS, raid 5 data, MS SQL 2000, our database is 6GB.
My initial thoughts are to migrate to SQL 2008, add memory. I would think the SQL issues alone would cause IO thrashing which I think would have a impact on IO latency.  At the same time I would like to capture how is it working now, make changes, and do I have expected results.

Thanks

Attachment now added, page 2 not page 3 as referenced above.
CCF04132012-0000.pdf

You need to model your I/O mix and determine the ratio of reads, writes, random, sequential, and block sizes.   Then you can tune iometer to create a benchmark that models YOUR hardware, so you can benchmark.  Also if you care about performance when hardware is degraded, then yank a disk drive while running benchmarks.

Plenty of things can typically be done to improve performance well before you throw money at it.  Are your partitions aligned?  (Google that).  Since SQL Server does 64KB I/O natively, is your NTFS set up for 64KB chunk size?  These two things alone can give you 25% performance gain, or more.    Now looking at your RAID.  Now that you know the RAID does 64KB I/O at a time, how is  it configured?  If you don't have each disk drive set to do a 64KB chunk, then you are absolutely throwing away I/O.

The partitions are aligned and the chunk size (allocation unit size) is 4096.  

Back to my original question, what is the best way to monitor and see where I am today and when I make changes if I have improvement?