?
Solved

high load on a websphere server.

Posted on 2012-08-13
8
Medium Priority
?
1,915 Views
Last Modified: 2012-10-16
Hi

Does anybody knows why is load too high when the server has enough CPU/SWAP/MEM (see b5_cpu.JPG?
This server is only running a websphere application server. Does java process needs any tunning?

See files attached.

OS: AIX6.1TL7SP5
websphere 7.0 - SP 23
b5-topas.JPG
b5-lsps.JPG
b5-vio.JPG
b5-uptime.JPG
b5-cpu.JPG
0
Comment
Question by:sminfo
  • 5
  • 3
8 Comments
 

Author Comment

by:sminfo
ID: 38286810
another file... real_cpu_used by the host...
b5-cpu-real.JPG
0
 
LVL 68

Accepted Solution

by:
woolmilkporc earned 2000 total points
ID: 38287089
Hi again,

why do you think that your machine is overloaded? Do you experience any trouble?

topas shows a runqueue size of just 4.6 (uptime shows 6.5 for the 5-minute interval), the lpar2rrd image shows just 0.15 real cores used at max.

The java process uses 258 MB of virtual memory, which is not really much for java.

Load is the sum of all runnable threads, i.e. those running on a CPU or being ready to run.

It really seems that these threads (although there are several of them) use very few CPU resources (see lpar2rrd).

You can issue "ps -m -p 4784340" to see how many threads the java process has forked so far.

You have 4 VPs, i.e. 8 SMT threads configured, so with a load of 6 there are still spare threads left.

I think there is absolutely no reason the get worried.

wmp
0
 

Author Comment

by:sminfo
ID: 38287138
well, I see no trouble on the service to tell the truth ;)

ps -m -p 4784340 shows only one thread....

      PID    TTY  TIME CMD
  4784340      -  7:42 java
        -
        -
        -
        -
        -


What does PgSp means? used pagging memory?

The only odd thing is to see load in 6.0, but if AIX works that way I can accept it ;)
0
Granular recovery for Microsoft Exchange

With Veeam Explorer for Microsoft Exchange you can choose the Exchange Servers and restore points you’re interested in, and Veeam Explorer will present the contents of those mailbox stores for browsing, searching and exporting.

 

Author Comment

by:sminfo
ID: 38287144
also, what's the meaning of Runqueue size? It shows now 8/16 or 32.
0
 
LVL 68

Assisted Solution

by:woolmilkporc
woolmilkporc earned 2000 total points
ID: 38287216
ps -m shows 6 threads. Each "-" represents one forked thread. Use

ps -m -o THREAD -p 4784340

to see details.

PgSp is the size of the paging space allocated to (but not necessarily used) by the process.
It's the working space (footprint) of the process in real and virtual memory.
It does not include space used for executables and shard libraries.

Runqueue size is the current number of runnable threads. It's the base for computing the load (which is the runqueue size averaged over 1-, 5- and 15-minute intervals).

Use this little script to check for processes with more than one thread:
ps -Am -o pid,thcount,comm,args |grep -v defunct |\
awk 'BEGIN {print "\nMultithreaded Processes:\n"}
     /[0-9]/ {num=num+$2; if($2>1) print substr($0,1,60)}
     END {print "\nSum of all Threads:\n\n\t" num}' | sort -k1,n

Open in new window

0
 

Author Comment

by:sminfo
ID: 38287313
well,

ps -m -p 4784340  shows only 6 becuase I cut the output..

(b5):[root] /-> ps -m -o THREAD -p 4784340
    USER      PID     PPID       TID ST  CP PRI SC    WCHAN        F     TT BND COMMAND
    root  4784340        1         - A    1  60 88        *   242001      -   - /usr/IBM/WebSphere/AppServer1/java/bin/java -Declipse.secur
       -        -        -   5439495 S    0  82  1 f1000f0a10005340  8410400      -   - -
       -        -        -   6226039 Z    0  84  1        -   c00001      -   - -
       -        -        -   6291701 S    0  82  1 f1000f0a10006040  8410400      -   - -
       -        -        -   6357085 S    0  82  1 f1000a01001f72b0   410400      -   - -
       -        -        -   6422687 S    0  82  1 f1000f0a10006240  8410400      -   - -
       -        -        -   6488293 S    0  82  1 f1000f0a10006340  8410400      -   - -
       -        -        -   6750429 S    0  82  1 f1000f0a10006740  8410400      -   - -
       -        -        -   7405737 Z    0  82  1        -   c00001      -   - -
       -        -        -   8323081 S    0  82  1 f1000f0a10007f40  8410400      -   - -
       -        -        -   9109543 S    0  82  1 f1000f0a10008b40  8410400      -   - -
       -        -        -   9699519 S    0  82  1 f1000f0a10009440  8410400      -   - -
       -        -        -   9961549 S    0  82  1 f1000f0a10009840  8410400      -   - -
       -        -        -  10289265 S    0  82  1 f1000f0a10009d40  8410400      -   - -
       -        -        -  10354863 S    0  82  1 f1000f0a10009e40  8410400      -   - -
       -        -        -  10420381 S    0  82  1 f1000f0a10009f40  8410400      -   - -
       -        -        -  10485841 S    0  82  1 f1000f0a1000a040  8410400      -   - -
       -        -        -  10616961 S    0  82  1 f1000f0a1000a240  8410400      -   - -
       -        -        -  10747991 S    0  82  1 f1000a05000d0278   410400      -   - -
       -        -        -  10879139 S    0  82  1 f1000f0a1000a640  8410400      -   - -
       -        -        -  11337929 Z    0 110  1        -   c00001      -   - -
       -        -        -  13238499 S    0  82  1 f1000f0a1000ca40  8410400      -   - -
       -        -        -  13369387 S    0  60  1 f1000f0a1000cc40  8410400      -   - -
       -        -        -  13435109 S    0  82  1 f1000f0a1000cd40  8410400      -   - -
       -        -        -  14221433 Z    0  71  1        -   c00001      -   - -
       -        -        -  14680257 Z    0  82  1        -   c00001      -   - -
       -        -        -  14745849 S    0  82  1 f1000f0a1000e140  8410400      -   - -
       -        -        -  14811357 S    0  82  1 f1000f0a1000e240  8410400      -   - -
       -        -        -  14876769 Z    0  82  1        -   c00001      -   - -
       -        -        -  15138957 Z    0  82  1        -   c00001      -   - -
       -        -        -  15532211 Z    0  71  1        -   c00001      -   - -
       -        -        -  15663353 S    0  82  1 f1000f0a1000ef40  8410400      -   - -
       -        -        -  15728885 S    0  82  1 f1000e0000d88a08   410400      -   - -
       -        -        -  15859963 S    0  82  1 f1000f0a1000f240  8410400      -   - -
       -        -        -  15925489 S    0  82  1 f1000f0a1000f340  8410400      -   - -
       -        -        -  15990797 S    0  82  1 f1000a05000d0178   410400      -   - -
       -        -        -  16056565 S    0  82  1 f1000f0a1000f540  8410400      -   - -
       -        -        -  16122101 S    0  60  1 f1000a01002006d8   410400      -   - -
       -        -        -  16187639 S    0  60  1 f1000f0a1000f740  8410400      -   - -
       -        -        -  16253177 S    0  60  1 f1000f0a1000f840  8410400      -   - -
       -        -        -  16384253 S    0  82  1 f1000f0a1000fa40  8410400      -   - -
       -        -        -  16449791 S    0  82  1 f1000f0a1000fb40  8410400      -   - -
       -        -        -  16580609 S    0  82  1 f1000f0a1000fd40  8410400      -   - -
       -        -        -  16646155 Z    0  85  1        -   c00001      -   - -
       -        -        -  16777233 S    0  82  1 f1000f0a10010040  8410400      -   - -
       -        -        -  16842773 S    0  82  1 f1000f0a10010140  8410400      -   - -
       -        -        -  17104917 S    0  82  1 f1000f0a10010540  8410400      -   - -
       -        -        -  17236033 Z    0 162  1        -   c00001      -   - -
       -        -        -  17301531 S    0  82  1 f1000f0a10010840  8410400      -   - -
       -        -        -  17367187 S    0  82  1 f1000f0a10010940  8410400      -   - -
       -        -        -  17629195 S    1  82  1 f1000f0a10010d40  8410400      -   - -
       -        -        -  17825837 S    0  62  1 f1000a01001c2118   410400      -   - -
       -        -        -  17956911 S    0  82  1 f1000f0a10011240  8410400      -   - -
       -        -        -  18022449 S    0  82  1 f1000f0a10011340  8410400      -   - -
       -        -        -  18219101 S    0  82  1 f1000f0a10011640  8410400      -   - -
       -        -        -  18612289 S    0  82  1 f1000f0a10011c40  8410400      -   - -
       -        -        -  19071053 S    0  82  1 f1000f0a10012340  8410400      -   - -
       -        -        -  19202077 S    0  82  1 f1000f0a10012540  8410400      -   - -
       -        -        -  19398837 S    0  82  1 f1000f0a10012840  8410400      -   - -
       -        -        -  19660933 S    0  82  1 f1000f0a10012c40  8410400      -   - -
       -        -        -  19726477 S    0  82  1 f1000f0a10012d40  8410400      -   - -
       -        -        -  19857531 S    0  86  1 f1000f0a10012f40  8410400      -   - -
       -        -        -  20054087 S    0  82  1 f1000f0a10013240  8410400      -   - -
       -        -        -  20119667 S    0  82  1 f1000f0a10013340  8410400      -   - -
       -        -        -  20185227 S    0  82  1 f1000f0a10013440  8410400      -   - -
       -        -        -  20644001 S    0  82  1 f1000f0a10013b40  8410400      -   - -
       -        -        -  20775039 S    0  82  1 f1000f0a10013d40  8410400      -   - -
       -        -        -  20840583 S    0  82  1 f1000f0a10013e40  8410400      -   - -
       -        -        -  20971653 S    0  82  1        -   418400      -   - -
       -        -        -  21037191 S    0  82  1        -   418400      -   - -
       -        -        -  21102729 S    0  82  1        -   418400      -   - -
       -        -        -  21168267 S    0  82  1 f1000f0a10014340  8410400      -   - -
       -        -        -  21299269 Z    0  82  1        -   c00001      -   - -
       -        -        -  21364881 S    0  82  1 f1000f0a10014640  8410400      -   - -
       -        -        -  21495957 S    0  82  1        -   418400      -   - -
       -        -        -  21561487 S    0  82  1 f1000f0a10014940  8410400      -   - -
       -        -        -  21627091 S    0  82  1 f1000f0a10014a40  8410400      -   - -
       -        -        -  21889255 Z    0  71  1        -   c00001      -   - -
       -        -        -  22020273 S    0  82  1 f1000f0a10015040  8410400      -   - -
       -        -        -  22216775 S    0  82  1 f1000f0a10015340  8410400      -   - -
       -        -        -  22544583 S    0  82  1 f1000f0a10015840  8410400      -   - -
       -        -        -  22610035 S    0  82  1 f1000f0a10015940  8410400      -   - -
       -        -        -  22675509 S    0  82  1 f1000f0a10015a40  8410400      -   - -
       -        -        -  22741179 S    0  82  1 f1000f0a10015b40  8410400      -   - -
       -        -        -  22872255 S    0  82  1 f1000f0a10015d40  8410400      -   - -
       -        -        -  23134425 S    0  82  1 f1000f0a10016140  8410400      -   - -
       -        -        -  24838329 S    0  62  1 f1000f0a10017b40  8410400      -   - -
       -        -        -  24969281 Z    0  71  1        -   c00001      -   - -
       -        -        -  25952279 Z    0  82  1        -   c00001      -   - -

This is the output of the script:

Multithreaded Processes:
Sum of all Threads:
        375
  262152     3 lrud     lrud
  393228     3 psmd     psmd
  458766     5 vmmd     vmmd
  589842     3 reaffin  reaffin
  720930    19 j2pg     j2pg
  851994     3 pilegc   pilegc
 1245222     5 gil      gil
 1310760     2 wlmsched wlmsched
 1835128    16 dog      dog
 1900642     8 ldmp_process ldmp_process
 2097258    16 syncd    /usr/sbin/syncd 60
 2359382     2 nfsSM    nfsSM
 2424932     8 nfssync_kproc nfssync_kproc
 2621526     2 kbiod    kbiod
 3145854     2 kdmd     kdmd
 3735678     8 dsmc     /usr/tivoli/tsm/client/ba/bin/dsmc s
 4194440     8 rmcd     /usr/sbin/rsct/bin/rmcd -a IBM.LPCom
 4784340    88 java     /usr/IBM/WebSphere/AppServer1/java/b
 5439654     3 dsmscoutd /usr/tivoli/tsm/client/hsm/bin/dsms
 6553698    41 java     /opt/IBM/cicstg/jvm160/bin/java -Xmx
 7274720    16 IBM.ServiceRMd /usr/sbin/rsct/bin/IBM.Service
 7471340    14 IBM.CSMAgentRMd /usr/sbin/rsct/bin/vac8/IBM.C
 7602424    22 IBM.DRMd /usr/sbin/rsct/bin/IBM.DRMd
 8192070     3 cicsterm cicsterm
10289152    10 cclclnt  cclclnt

see anything wrong?
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 38287429
That seems all quite normal. The only eye-catchers are Websphere and CICS TG.

I'm not familiar with the CICS transaction gateway, but  I assume that 41 threads aren't too many, as well as those 88 threads of Websphere.

Both applications are client-server oriented, and a thread forked by an application of course does not automatically demand for CPU (and thus for a place in the runqueue), it might just sit and wait for a connection.
In case of Websphere there must be a parameter "MinSpareServers" or so, which configures the number of "reserve" threads the server would hold available.
These do not cost CPU, just some memory. They show up as threads, but not in the runqueue.

So if the runqueue grows this could be due to new client connections actively demanding for service.

I see that you're running HSM. A charcteristic here is that (maybe lots of) scanning threads are started in regular intervals, in order to check for files which should be processed.
Could it be that the server load rises and falls along with dsmscoutd's activity?
Afaik you can issue queries like "dsmscoutd statistics" or "dsmscoutd scanplan" to
find out what it did/will do.

Anyway, I'm still convinced that all this is normal activity, more or less.
0
 

Author Closing Comment

by:sminfo
ID: 38287509
thanks wmp.. i'll take a look at dsmscoutd
0

Featured Post

New feature and membership benefit!

New feature! Upgrade and increase expert visibility of your issues with Priority Questions.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This tech tip describes how to install the Solaris Operating System from a tape backup that was created using the Solaris flash archive utility. I have used this procedure on the Solaris 8 and 9 OS, and it shoudl also work well on the Solaris 10 rel…
Installing FreeBSD… FreeBSD is a darling of an operating system. The stability and usability make it a clear choice for servers and desktops (for the cunning). Savvy?  The Ports collection makes available every popular FOSS application and packag…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Suggested Courses
Course of the Month16 days, 15 hours left to enroll

864 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question