Solved

Dual-Core CPU - 100% WAITIO on Primary CPU seems to cause 100% Idle on Secondary CPU

Posted on 2010-08-26
15
1,138 Views
Last Modified: 2013-11-17
We have a database server that has 4 dual-core cpu's.  When the Primary Core's (CPU 0,2,4,6) are in 100% WAITIO, the Secondary Cores (CPU 1,3,5,7) are 100% Idle.  When this occurs, the server acts as if it is 100% busy - does not respond quickly to commands - yet there are 4 CPU cores sitting 100% idle. Why  is this?  

See attached for evidence...

- Greg
nmon.JPG
0
Comment
Question by:gmarino
  • 5
  • 5
  • 3
  • +2
15 Comments
 
LVL 1

Expert Comment

by:bsa_jug
ID: 33533704
what are your disk mapped to other VG ? have you collected I/O statistics  for the Disks , looks like the disk are waiting on i/o operations
0
 
LVL 4

Author Comment

by:gmarino
ID: 33533976
I know the CPU's are waiting on I/O (SAN).  That's not the question.  That is a totally different issue.

The question at hand is - Why are CPUs 1,3,5,7 idle when CPUs 0,2,4,6 are in 100% WAITIO?

- Greg
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 33534680
Hi,

those CPUs  1,3,5,7 are not cores, but are SMT threads.

SMT is "Simultaneous Multi-Threading". SMT capable CPUs show up in topas only "as if" they were cores.

SMT capable CPUs (along with the capable OS, AIX 5.3 and later) are able to store the states of two threads and divide their components (ALU etc.) among them.

Here is kind of an introduction to SMT: http://www.ibm.com/developerworks/aix/library/au-aix5_cpu/index.html

So with heavy I/O wait it might well be that the observed phenomenon is due to the specific SMT implementation, and because an SMT thread looks just like a "core" in topas it seems to be idle.

The above paper has some nice explanations on this, as well as this one:
http://www.aixmind.com/wp-content/uploads/2010/07/Basic_Monitoring_of_IO_on_AIX.pdf

wmp
 
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 33534698
I wrote "topas" - exactly the same is true for "nmon", of course.
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 33539670
In the "c" view of NMON ("CPU by processor", the upmost block in your picture) you can hit "#" to get PURR based values.
 PURR is " Processor Utilization Resource Register", and here is some background -
In a shared-partition environment, you need to understand that there is an unused time slice in each entitled processor capacity. When a virtual processor or SMT thread becomes idle, it is able to cede processor cycle to Hypervisor, and then the Hypervisor can dispatch unused processor cycles for other work. In order to collect CPU utilization at a processor thread level (in an SMT environment), the POWER5 architecture has implemented a new register -- it's called the Processor Utilization Resource Register (PURR). Each thread has its own PURR. The units are the same as the time base register and the sum of the PURR values for both threads is equal to time base register. More traditional methods for measuring processor utilization tend to yield incorrect results in an SMT and SPLAR environment, which is why the PURR registers provide a more accurate realistic measure of processor utilization.
The above quote is from here (another useful paper) -
 http://www.ibm.com/developerworks/aix/library/au-aixoptimization/index.html
Simplified - with TBR you see processor utilization from the LPAR perspective, whereas with PURR you see it from the hypervisor perspective.
Maybe comparing PURR based and TBR based values could shed some more light on your issue.
wmp
0
 
LVL 4

Author Comment

by:gmarino
ID: 33545685
wmp -

Some good info in your posts!  I'm "just a lowly DBA" (yeah right) trying to understand things outside my comfort-zone.  This observation on the CPU's intrigued me and I want to understand it - and at the same time be able to explain why a server with Total CPU reporting 50% WaitIO and 50% idle is really 100% busy/non-responsive.  There are many in DB2/AIX Support that think are crazy when we report that the server is non-responsive to the command line with that CPU utilization.  We are told "There is 50% idle capacity - you must be getting a response."

My guess is that the SMT cores share IO Pathways with the Primary Core on the chip and thus cannot process anything if the Primary Core is 99% waiting for IO.  (Just a guess - looking for evidence/validation.)

I am trying to use the "#" command on nmon ('c' view) and it does nothing. nmon is running as TOPAS_NMON Version 5.3.10.1 TL10.  I will have to research how to get that to work.  I did see that it works in PHYSICAL CPU mode - if you look at my screenshot, I don't have the word PHYSICAL at the top of my screen.

topas = nmon these days (Nigel must be proud that his "illegitimate child" finally got "officially adopted" by AIX.)

Thanks for the insights...

- Greg

0
 
LVL 68

Accepted Solution

by:
woolmilkporc earned 300 total points
ID: 33546257
# only works in an LPAR environment!

You don't have 50% waitio, but near 100% for the concerned threads.

If one single thread has to pass most of its time sitting and waiting for I/O, the other thread could only do some work not involving any I/O at all!

If your system were virtualized, at least the other partitions would be able to continue working (but will most probably also wait for I/O), but if it's physical ...

Will hopefully be back tomorrow to tell you more.

wmp
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 61

Assisted Solution

by:gheist
gheist earned 200 total points
ID: 33574865
Your system does swap, it is a highest priority IO in system, so rest suffers.
It is normal that primary core serves IO, because secondary cannot receive hardware interrupt signals.
To ease your life - vmtune -f 1024 -F 1280
Which will make paging into big blocks without saturating IO busses

Consider compacting your datasets, pinning/mlocking/wlm-ing critical apps, and as an ultimate solution - adding more RAM.
0
 
LVL 4

Author Comment

by:gmarino
ID: 33582722
wmp and gheist,

Great information - confirming and further defining what I suspected to be the case.  

Greg
0
 
LVL 61

Expert Comment

by:gheist
ID: 33584334
apply my quick fix and spend some time configuring WLM and write to your management that you need some extra HW for optimal performance.

0
 
LVL 5

Expert Comment

by:balasundaram_s
ID: 33589965
It looks to me that SMT is not enabled (second logical processor stays IDLE for each CPU)
Confirm it by the following command:
# smtctl

Please ensure(talk to sysadmin) that the SMT is enabled by the following command

# smtctl -m on -w boot  ( and run the 'bosboot' command )
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 33590583
With SMT disabled the secondary SMT threads (appearing as a logical processors) will not show up at all in nmon.
0
 
LVL 4

Author Comment

by:gmarino
ID: 33592942
SMT is enabled.  The 100% WAIT IO on the primary core is effectively shutting off the second core.

This is a DB2 Data Warehouse server.  All this (and 4 other similar servers) is doing is reading data from disk into the DB2  Bufferpools on behalf of the queries being run.  We have long since identified that it's both the queries themselves (tablescans that must read these large mega-GB tables and thus flush the 2GB Bufferpools) and the mix of the various batch jobs and Brio queries that is causing the rampant over utilization of the servers IO resources.

The original question was - why when the Primary Core's (CPU 0,2,4,6) are in 100% WAITIO did the Secondary Cores (CPU 1,3,5,7) are 100% Idle.  I think that has been answered nicely above.  This gives me some confidence when I challenge my colleagues and DB2/AIX support who claim that the box still has 50% capacity in this situation (it obviously does NOT).

Thanks for the discussion!

-Greg
0
 
LVL 4

Author Closing Comment

by:gmarino
ID: 33593538
Thanks
0
 
LVL 61

Expert Comment

by:gheist
ID: 33595049
... DB2 Data Warehouse ...

Advice 1 you need to tune aio with "smitty aio" and allow DB2 to use more than one aio server

Advice 2 secondary cores will not serve hardware, they are not full cpu-s like in nowadays "multicore" PC processors, they are more like "hyperthreading" - a processor microcode, so better avoided to not lose resources for database workload.

Advice 3 to complement 1 )also look at streams parameters in http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds4/no.htm
i.e how many aio request timer you need to serve aio requests...

0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Attention: This article will no longer be maintained. If you have any questions, please feel free to mail me. jgh@FreeBSD.org Please see http://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-update-server/ for the updated article. It is avail…
When you do backups in the Solaris Operating System, the file system must be inactive. Otherwise, the output may be inconsistent. A file system is inactive when it's unmounted or it's write-locked by the operating system. Although the fssnap utility…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now