vmmd process consuming too much cpu

We currently have an application server that runs AIX 6100-06-05-1115. This Lpar seems to be struggling. It crashed on out of memory errors twice in one week.  When I run the following command, this is the output that I get.
 ps gv | head -n 1; ps gv | egrep -v "RSS" | sort +6b -7 -n -r |pg
 PID       TTY STAT  TIME  PGIN  SIZE        RSS         LIM  TSIZ TRS %CPU %MEM COMMAND
 12320880      - A   20:59 6947  360644 360772    xx    80   128  0.1          2.0 /usr/W
  8585250      - A    69:32  763   333648 333776    xx    80   128  0.2          2.0 /usr/W
 10616940      - A   18:03  830   270472 270600    xx    80   128  0.1          1.0 /usr/W
 11403482      - A   10:13   48    248184 248312    xx    80   128  0.0          1.0 /usr/W
 10748042      - A    9:04   89     242104 242232    xx    80   128  0.0          1.0 /usr/W
  5963878      - A     8:21   45     227192 227320    xx    80   128  0.0          1.0 /usr/W
 13172928      -       8:37   69     221112 221240    xx    80   128  0.0          1.0 /usr/W
 13697138      -       9:52  109    219320 219448    xx    80   128  0.0          1.0 /usr/W
  7143628      - A     3:35    4     193540 193668    xx    80   128  0.0           1.0 /usr/W
 12648552      - A    3:48   96     172732 172860    xx    80   128  0.0          1.0 /usr/W


This is the output when I run lparstat
Type                                       : Shared-SMT-4
Mode                                      : Uncapped
Entitled Capacity                     : 0.40
Partition Group-ID                   : 32784
Shared Pool ID                         : 0
Online Virtual CPUs                  : 2
Maximum Virtual CPU               : 10
Minimum Virtual CPU                : 1
Online Memory                         : 20736 MB
Maximum Memory                    : 36096 MB
Minimum Mem                         : 8448 MB
Variable Capacity Weight           : 128
Minimum Capacity                     : 0.10
Maximum Capacity                     : 5.00
Capacity Increment                     : 0.01

Nmon shows that vmmd is a top processor and uses about 1G of CPU.

How do I stop this process from using so much CPU, and what causes the vmmd process to use so much cpu ?
laingmeliAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

woolmilkporcCommented:
CPU consumption isn't measured in Gigabytes.

Please run

topas

Do you see vmmd on the lower left? What are the values?

What is the corresponding "physc" value? Hit "c" three times to see it in the upper left area below "Idle".

Next, please check errpt. OOM errors should be logged there. Do you see any entries around the date and time when the LPAR crashed?
If so, what are the details? Check with

errpt -a -j <number from first column of errpt>

By, the way, you cannot stop vmmd. It's essential for system operation
0
laingmeliAuthor Commented:
Hi , Thanks for the response. I do see the vmmd , and this is the values
vmmd         458766   0.0   1.2 root

The corresponding physc value varies between 0.10 and 0.12


Errpt output
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Fri Aug 30 09:42:54 USAST 2013
Sequence Number: 1312
Machine Id:      00F727B74C00
Node Id:         psirap2a
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
java
USER'S PROCESS ID:
              17498174
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
0
woolmilkporcCommented:
OK,

so your machine has not been out of real memory, it has been out of paging space, which makes a bit of a difference.

It is quite normal that vmmd would consume some CPU when there are paging space problems - there is a lot of work to do for it to keep the machine running as long as possible.

The CPU consumption of vmmd  (and also the overall CPU consumption) was minuscule at the moment where you collected the above values.

Your "real" memory as seen by the LPAR is 20 GB. How many paging space do you have defined? Run

lsps -s

Check for free space in the VG for a possible paging space extension with

lsps -a

and

lsvg <vgname as displayed with lsps -a>

Check "FREE PPS:" on the upper right.

Java programs tend to use quite a lot of memory. How about reducing this demand by means of the java "-Xmx <nnn>" start option? Please consult your application development team/company.
0
Cloud Class® Course: Python 3 Fundamentals

This course will teach participants about installing and configuring Python, syntax, importing, statements, types, strings, booleans, files, lists, tuples, comprehensions, functions, and classes.

laingmeliAuthor Commented:
$ lsps -s
Total Paging Space   Percent Used
      4096MB               2%


$ lsps -a
Page Space      Physical Volume   Volume Group Size %Used Active  Auto  Type Chksum
hd6             hdisk0            rootvg        4096MB     2   yes   yes    lv     0
$

$ lsvg rootvg
FREE PPs:       295 (18880 megabytes)
0
woolmilkporcCommented:
4 GB paging space is indeed  a bit too little for 20 GB of memory.

I think you should increase it by at least another 6 GB.

First, determine the partition size of rootvg by checking "PP Size" of

lsvg rootvg

Now issue

chps -s $((6144/<PPSIZE>)) hd6

where "<PPSIZE>" is the determined partition size.

This command can be issued while the system is up and running, without worries.

Please note that this extension will cost 6 GB of free space in rootvg, or even 12 GB if hd6 is mirrored.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
laingmeliAuthor Commented:
Thank You for your assistance.
0
woolmilkporcCommented:
You're welcome!
0
woolmilkporcCommented:
The first few comments deal with the problem analysis  and comment https:#a39497204 describes the way how to remedy the issue. Seems the suggested activities have been successful, because the author's final comment was "Thank you for your assistance".
So https:#a39497204 should be accepted.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Unix OS

From novice to tech pro — start learning today.