Hi everyone :)
I am responsible for developing a real time software application (running on windows, primarily XP) which appears to be having performance issues and I am trying to find out how to interpret certain MS performance counters but Ive searched all over the internet, and here, to no avail and i was hoping someone could point me in the right direction. I apologise if this question has been answered before or if the answers are obvious, I did make sure i spent a fair amount of time searching first :)
What i would like to know is how to fully interpret the following counters on both single core and dual/multi core boxes. I'd like to know what acceptable values and unacceptable values are. Do I need to combine other counters to interpret them properly? Are there certain patterns that indicate a problem rather than just a single value? etc.
Processor Queue Length
Context switches/sec
Avg.Disk Queue Length
For context, I am already looking at and understanding the following counters ok:
% processor time for each core, including privileged and user, Available MBytes, and Pages/sec
If anyone has any suggestions for additional counters i should be looking at for measuring performance that would also be great :)
Thank you,
Matt :)
This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.
Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.
If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.
Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.
Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.
Access the answers to your technology questions today.
30-day free trial. Register in 60 seconds.
Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.
Try it out and discover for yourself.
30-day free trial. Register in 60 seconds.
Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.
Going further a bit you should monitor your systems with these counters and set your own benchmarks/thresholds for acceptable performance after you have monitored them for a while. Try to capture instances where you have noticeable performance degradation and that should give you a good idea of what to look for. Because each system performs differently its hard to do a cookie cutter approach to monitoring.
Thanks for the quick response Gator :)
What i now realise I should have said is that i understand the meaning of the counters i'm interested in, i just don't know what acceptable values are and therefore how to interpret the statistics i have got. The link you give has got some good reading linked from it though so i'll go through it and see I can learn anything there, thank you.
GATOR420:
[Going further a bit you should monitor your systems with these counters and set your own benchmarks/thresholds for acceptable performance after you have monitored them for a while. Try to capture instances where you have noticeable performance degradation and that should give you a good idea of what to look for. Because each system performs differently its hard to do a cookie cutter approach to monitoring.]
This is very good advice and ultimately may be the only true way to know the answers to my questions although its not an easy process to perform. I was hoping/expecting there would also be some generalised ranges of values for each counter, particularly the queue lengths (Disk and CPU).
My app is a real time trading application so there is no tolerance for delays(theoretically) so i'm really looking for examples of values expected during the "normal" functioning of the PC where neither cpu usage nor HD access are a bottleneck within the system. As soon as they start being a bottleneck I need to know, and i then need to know some kind of scale as to what constitutes the resource being a little overworked and therefore contributing slightly, to what constitunes a resource being majorly overworked and so contributing greatly to a slowing of the system.
Ideally if you have sustained values over 0 for CPU and disk queue that is a definite indicator. Sustained is the key word. If you have peaks where it goes above 0 and drops back down that just indicates that it received a burst of requests, processed them, and is back to "normal." It also varies on multi-processor systems where if the application is not multi-threaded, it could be tying up one CPU where the other is completely idle. So you really have to understand your system and what you are running on it to determine a set of values to benchmark by.
OK cool. So, getting more specific, in the case of the stats i am looking at, my processor queue length idling at 4 minimum for 15 minutes in the run up to my suspected incident whilst hitting levels of 10 a number of times and 16 twice would be a clear indication of CPU resources being overused, which coupled with the 80% cpu usage i have seen would make sense.
I do appreciate what you are saying about multi-core systems and mult-ithreaded applications having a different affect on performance monitoring although if the queue is backing up and at least one core is running at a high usage level then you obviously need more more cycles to do the work you are trying to do. What you then choose to do to solve that problem (i.e. Re-write your code to use more threads or increase the speed of the indiviudal cores) then depends the architecture of your hardware and software, no?
---
My Avg Disk Queue Length values are in the scale of 1/100ths, e.g.
0.000542649
0.001981151
0.000570173
0.000463098
0.00062454
0.000509863
so what is an acceptable value here?
Thanks again for the help. It's good to have someone to discuss this stuff with :)
Hi gator :)
Sorry for the delay. I've got a lot on at work atm with this only being one of the paths i'm following. I've requested the current disk queue lengths be added as well and i'll see what comes back. I don't know what is normal with timescales for questions on the site so i may not get the stats back before i close the topic and accept the solution.
Matt :)
No worries.
Here is some basic information about the disk queue monitor and why I also requested to look at current and not just average. This is from: http://www.oreillynet.com/
The Avg. Disk Queue Length counter is derived from the product of Avg. Disk sec/Transfer multiplied by Disk Transfers/sec, which is the average response of the device times the I/O rate. Again, this corresponds to a well-known theorem of Queuing Theory called Little's Law, which states:
N = A * Sr
where N is the number of outstanding requests in the system, A is the arrival rate of requests, and Sr is the response time. So the Avg. Disk Queue Length counter is an estimate of the number of outstanding requests to the (Logical or Physical) disk. This includes any requests that are currently in service at the device, plus any requests that are waiting for service. If requests are currently waiting for the device inside the SCSI device driver layer of software below the diskperf filter driver, the Current Disk Queue Length counter will have a value greater than 0. If requests are queued in the hardware, which is usual for SCSI disks and RAID controllers, the Current Disk Queue Length counter will show a value of 0, even though requests are queued.
Since the Avg. Disk Queue Length counter value is a derived value and not a direct measurement, you do need to be careful how you interpret it. Little's Law is a very general result that is often used in the field of computer measurement to derive a third result when the other two values are measured directly. However, Little's Law does require an equilibrium assumption in order for it be valid. The equilibrium assumption is that the arrival rate equals the completion rate over the measurement interval. Otherwise, the calculation is meaningless. In practice, this means you should ignore the Ave Disk Queue Length counter value for any interval where the Current Disk Queue Length counter is not equal to the value of Current Disk Queue Length for the previous measurement interval.
Suppose, for example, the Avg. Disk Queue Length counter reads 10.3, and the Current Disk Queue Length counter shows four requests in the disk queue at the end of the measurement interval. If the previous value of Current Disk Queue Length was 0, the equilibrium assumption necessary for Little's Law does not hold. Since the number of arrivals is evidently greater than the number of completions during the interval, there is no valid interpretation for the value in the Avg. Disk Queue Length counter, and you should ignore the counter value. However, if both the present measurement of the Current Disk Queue Length counter and the previous value are equal, then it is safe to interpret the Avg. Disk Queue Length counter as the average number of outstanding I/O requests to the disk over the interval, including both requests currently in service and requests queued for service.
You also need to understand the ramifications of having a total disk roundtrip time measurement instead of a simple disk service time measure. Assuming M/M/1, a disk at 50 percent busy has one request waiting on average and disk response time is 2 times service time. This means that at 50 percent busy--assuming M/M/1 holds--an Avg. Disk Queue Length value of 1.00 is expected. That means that any disk with an Avg. Disk Queue Length value greater than 0.70 probably has a substantial amount of queue time associated with it. The exception, of course, is when M/M/1 does not hold, such as during a back-up operation when there is only a single user of the disk. A single user of the disk can drive a disk to nearly 100 percent utilization without a queue!
Hi Gator :)
I'm too busy at the moment to carry this thread on in a fair emount of time and in fact have a load of other questions that have now become more pressing so i will accept your solutions for now and hopefully we will get the opportunity to continue this discussion another time.
Thank you for all you help :)
Matt :)
Business Accounts
Answer for Membership
by: GATOR420Posted on 2007-06-18 at 04:43:00ID: 19306125
Processor queue length is how many tasks the processor has lined up waiting to be processed. Generally if this is a high number (>5 is the benchmark we typically use depending on the size of the box and what it does. We do have an Oracle cluster server where we have our thresholds set to >25.) your CPU is being heavily tasked and can't process all the tasks being thrown at it. This would be an indicator that the workload is too high and you need to spread out the load or upgrade.
Avg. disk queue length is the same as above but this is for read/writes to the disk. A possible indicator of disk I/O problems. For example if you have one large capacity disk and you are seeing a high disk queue length then you shoould probably consider spreading the I/O across several smaller disks.
Context switches/sec I had to go look this one up, we don't monitor it and I had no idea what it did. Pasted from: http://www.microsoft.com/t
A context switch occurs when the kernel switches the processor from one thread to anotherfor example, when a thread with a higher priority than the running thread becomes ready. Context switching activity is important for several reasons. A program that monopolizes the processor lowers the rate of context switches because it does not allow much processor time for the other processes' threads. A high rate of context switching means that the processor is being shared repeatedlyfor example, by many threads of equal priority. A high context-switch rate often indicates that there are too many threads competing for the processors on the system.
Note
The rate of context switches can also affect performance of multiprocessor computers. For information about how to monitor and tune context-switch activity on multiprocessor systems, see "Measuring Multiprocessor System Activity" in this book.
You can view context switch data in two ways:"
The System\Context Switches/sec counter in System Monitor reports systemwide context switches.
"
The Thread(_Total)\Context Switches/sec counter reports the total number of context switches generated per second by all threads.
Although these counters might vary slightly due to sampling, generally they will be nearly equal.
Figure 7.5 plots System\Context Switches/sec during a transient bottleneck.
Figure 7.5 Systemwide Context Switches During a Processor Bottleneck
In Figure 7.5, Processor(_Total)\% Processor Time jumps to about 60 percent during the sample interval. System\Processor Queue Length (scaled by a factor of 10), shows that the queue varies from 2 to 6, with a mean near 4. System\Context Switches (shown scaled by a factor of 10), reveals an average of about 750 switches per second. A rate of context switches from 500 to 2,000 per second might indicate a problem with a network adapter or a device driver or that you are using an inefficient server-based application that spawns too many threads.
The Pviewer utility on the on the Windows 2000 operating system CDreports context switch data.For information about installing and using the Windows 2000 Support Tools and Support Tools Help, see the file Sreadme.doc in the \Support\Tools folder of the Windows 2000 operating system CD.