Solved

Determine iSCSI overhead via perfmon?

Posted on 2010-11-12
23
1,174 Views
Last Modified: 2013-11-14
Is there any way to determine how much overhead there is because we're using Microsoft' iSCSI software initiator with an onboard NIC? We're not sure if we want to invest $500+ in an iSCSI HBA card. I've heard it can save 60% of CPU utilization.

0
Comment
Question by:MrVault
  • 12
  • 4
  • 3
  • +2
23 Comments
 
LVL 22

Expert Comment

by:Matt V
ID: 34121000
Trying adding a Process monitor, % of Processor Time with the iSCSI process selected.
0
 

Author Comment

by:MrVault
ID: 34121097
What is the iSCSI process called?
0
 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 34130390
what is your cpu running at now?
0
 

Author Comment

by:MrVault
ID: 34136120
some are running at 40-70%. They are SQL servers running intensive operations, but it's hard to tell if the iSCSI piece is a bottleneck worth investing in.
0
 
LVL 21

Accepted Solution

by:
mastoo earned 100 total points
ID: 34136294
You need to look at something closer to pure i/o.  Maybe run iometer and look at cpu%.  It would be nice to benchmark it against the equivalent system with an iScsi hba but you'll at least get an idea.  If iometer hits the drives and you just show 10% cpu it is easy to say you don't need to spend the money.
0
 

Author Comment

by:MrVault
ID: 34136303
when you say just how 10%, do you mean it increases it by 10% or do you mean it categorizes iSCSI overhead as only 10%?

I've never used iometer before.
0
 
LVL 21

Assisted Solution

by:mastoo
mastoo earned 100 total points
ID: 34138022
The simplest way is with the server doing nothing, then you run iometer and look at total cpu usage.  I just suggested that as a start because if you get a very low cpu number in that scenario, you know the iScsi overhead is even smaller.  Otherwise I'm not sure how you identify overhead specific to running an iScsi software adapter.
0
 

Author Comment

by:MrVault
ID: 34138294
iSCSI overhead, is that tied to activity? Meaning is the overhead percent the same whether there's data going over iSCSI to the bulk storage somewhere or if there's really no data being transferred at all?
0
 
LVL 21

Expert Comment

by:mastoo
ID: 34138680
I won't call myself an expert in this area, but I imagine there is some cpu load attributed to the simulation of a hardware iScsi adapter.  So I would think it only manifests during I/O, which is why I thought iometer would be a good way to exercise it.  You get kind of the simplest i/o you can get.  The trouble with looking at cpu while running sql is that a large portion of the cpu is due to sql activity, which a smaller portion due to i/o, and of that even a smaller portion would be due to having a software iScsi adapter.  On our servers, I've observed that total cpu is fairly low during stress testing of the i/o so I've never worried about needing a hardware adapter.  The main drawback to the test I suggest is that if you get a high cpu, you can't reach any conclusion.  But a low cpu would lead to the conclusion that there would be little benefit to going hardware.
0
 
LVL 16

Assisted Solution

by:Gerald Connolly
Gerald Connolly earned 50 total points
ID: 34143193
If your current CPU utilisation is in the 40-70% range its debateble whether you need a hardware accelerator at present, sure its likely to reduce you CPU utilisation but is that worth $600, i doubt that! You will probably do better to buy some more memory.

Now if you are expecting your workload to increase its probably worth doing due diligence on make, model and pricing so that if you utilisation starts creeping up you can make a quick but informed decision. Obviously it would be better to go through the full system integration route if this is a production system, so think about getting one into one of your UAT systems sooner rather than later.
0
 

Author Comment

by:MrVault
ID: 34145286
LOL. A UAT system? If only I could convince them to spend the $$ to have an extra for testing. In all serious though I am trying to go that route.

Connollyg, I'm curious if you'd ever see the benefit of an iSCSI hba because if you're thinking it's not worth the benefit if it' near 70% utilization, then too much higher and the server is hosed. We've seen spikes easily hold 99% for a long time. the 40-70 is more a typical utilization. How far would you have it go before seeing the benefit? Most documents point to a reduction to 30% total utilization after implementing HBAs. of course YMMV.
0
Save on storage to protect fatherhood memories

You're the dad who has everything. This Father's Day, make sure your family memories are protected. My Passport Ultra has automatic backup and password protection to keep your cherished photos and videos safe. With up to 3TB, you have plenty of room to hold the adventures ahead.

 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 34149173
99%! Well you didnt say that last time!

But seriously, it comes down to a cost benefit! If your server is running short of CPU resource then you have to work though the options of why and what, and then how you could reduce it.
Then you have to balance the resultant actions against cost! Would some other upgrade give you a bigger bang for your buck in overall system performance, or would, what we used to call a ToE (TCP Offload Engine) be cost effective. [ i know a iSCSI HBA isnt quite the same thing as a ToE, but close enough ]

And YES i am in favour of iSCSI HBA's but they have to be cost effective too.
0
 

Author Comment

by:MrVault
ID: 34187772
Sorry for not getting back sooner. We're not always running at 99%, but it does happen on some of our servers for extended periods of time.

Does anyone know how to monitor the process as "mattvmotas" suggested?
0
 
LVL 47

Assisted Solution

by:dlethe
dlethe earned 100 total points
ID: 34188199
The 60% overhead is a valid number, but only on single-core systems.  You will also see big performance penalties in older PCI-based systems where the NIC and the disk controllers are on the same bus.

On a multi-core PCI-e based system, you will see a much smaller impact that may or may not be a factor.   Easiest way to tell, is just take a downtime window when nothing else is going on, and configure iometer to set up a multi-threaded workload that emulates as closely as possible the type of I/O you do real-world.  If you had a modern quad or more processors, and a PCIe-2.0 system, then I would just save myself the trouble and not even worry about the overhead.  

If, however, you don't have such a server, then it is worthwhile doing a test.   I can't tell you the right I/O model, you are just going to have to figure out how many threads you need, and the proper mix of  of read/write/random/sequential I//O and correct block sizes.

The closer you are to the right synthetic load, the more accurate the results.  

If you have a dual processor, or uniprocessor, or don't have a PCIe based motherboard, then by all means, save yourself the time and get a iSCSI with the built-in processor ... you WILL need it.

So only go through the exercise if you are in this grey area


0
 

Author Comment

by:MrVault
ID: 34188349
Thanks. These are all 2 year old servers with 2 quad core processors. I know it's not a guarantee but I'm pretty sure they all support PCIe.

They are all use the PERC6i raid controllers for the non-OS disks.
0
 
LVL 47

Expert Comment

by:dlethe
ID: 34188476
Then personally, I would not worry about it unless I needed to try to squeeze a little more network performance out of the systems. Keep an eye out for a bargain on a used card on ebay in the interim, but I wouldn't make it a priority.
0
 

Author Comment

by:MrVault
ID: 34188528
Do you think the same conclusion should be drawn regarding segregating our network with VLANs and/or physical switches? Right now they have 3 switches on a 4gig backplane all in a single vlan so the wan connections, iscsi connections, san replication, etc are all over the same subnet. best pracitces are to segregate it out both for performance and security reasons. not to dismiss security, but right now their biggest concern is performance. I need to determine if this network config is a bottleneck and if so pitch a solution (like getting separate switches for iscsi traffic or segregating via VLANs).
0
 
LVL 47

Assisted Solution

by:dlethe
dlethe earned 100 total points
ID: 34188560
No, segregating a network and using TOE for iSCSI are completely different.   Do your switches support RMON?  If so, they have a great deal of monitoring capability built-in, and you can see utilization and bottlenecks via any SNMP package, or perhaps through a built-in web interface.

0
 

Author Comment

by:MrVault
ID: 34190486
And which things should I be looking for specifically? Seeing if they are running close to 1Gbps per port? Or looking for collisions or retransmits? They are Foundry managed switches so i'm guessing they support SNMP.
0
 

Author Comment

by:MrVault
ID: 34360588
thanks guys. thoughts were helpful but it appears there's no metric to measure iSCSI overhead on a system without a test system or changing the config which involves downtime. I was hoping there was some perfmon counter. the first guy seemed to think there was a process but I'm guessing he was mistaken. oh well.
0
 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 34367990
There has been too much debate and input from several experts to give away all this expertise without assigning points!
0
 

Author Comment

by:MrVault
ID: 34368908
totally true. didn't mean to imply there'd be no points assigned. just hoping my latest comment might remind someone of a way :-)

Points awarded now.
0
 

Author Closing Comment

by:MrVault
ID: 34368917
no live method was given, but I'm guessing there is none. Only option is to have a like system with an HBA and measure differences.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

A quick step-by-step overview of installing and configuring Carbonite Server Backup.
This article is an update and follow-up of my previous article:   Storage 101: common concepts in the IT enterprise storage This time, I expand on more frequently used storage concepts.
This tutorial will walk an individual through the steps necessary to install and configure the Windows Server Backup Utility. Directly connect an external storage device such as a USB drive, or CD\DVD burner: If the device is a USB drive, ensure i…
This tutorial will walk an individual through the steps necessary to join and promote the first Windows Server 2012 domain controller into an Active Directory environment running on Windows Server 2008. Determine the location of the FSMO roles by lo…

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now