AIX is a stable operating system and it is unlikely it has memory leak. Capture the output of svmon and ps. Study the storage utilzaton of the highest memory usage PID and report the problem to application developer. According to my experience, usually they are sybase or oracle PID.

Sample output of SVMON
     Pid Command        Inuse      Pin     Pgsp  Virtual   64-bit    Mthrd
   27954 xxedi           7579     1485     7177     9004        N        N

  Vsid     Esid Type Description           Inuse   Pin Pgsp Virtual Addr Range
 38b7c        2 work process private        3069     1 5605  6469   0..6496 :
                                                                    65309..65535
 30018        d work shared library text    2304     0  151   160   0..65535
     0        0 work kernel seg             1810  1484 1372  2326   0..21860 :
                                                                    65474..65535
 26653        - pers /dev/amtrix:55355       210     0    -     -   0..259
 1a74d        1 pers code,/dev/amtrix:230    117     0    -     -   0..151
 36b7b        f work shared library data      40     0   49    49   0..897
 1e7af        - pers /dev/amtrix:4898         16     0    -     -   0..18
 3c7de        - pers /dev/amtrix:4543         13     0    -     -   0..18
 1a80d        - pers /dev/amtrix:12635         0     0    -     -
 32779        - pers /dev/amtrix:10283         0     0    -     -   0..0
   800        - pers /dev/amtrix:426149        0     0    -     -   0..17

Explanation of the output of ps vg | pg
RSS  - Real Memory Resident Size in 1KB
SIZE - Size of Page space in 1KB -à
       SVMON virtual (work process private + work shared library data
    Shmat/mmap) *4
TSIZ - Size of text (shared-program) image.
TRS  - Size of resident set (real memory) of text
TIME - The total execution time of the process
PGIN - (v flag) The number of disk I/Os resulting from references by the process to
       pages not loaded in core.

$svmon Pu > svmon.txt
$ ps vg | pg > vg.txt
           PID  TTY STAT  TIME  PGIN  SIZE   RSS   LIM  TSIZ   TRS %CPU %MEM COMMAND
r4edi
         27954    - A     1:33  6003 26060 13416 65536   546   468  0.0  1.0
         27954    - A     1:34  6096 26072 13716 65536   546   468  0.0  1.0
29 16:12 27954    - A     1:34  6096 26072 13116 65536   546   468  0.0  1.0
29 18:06 27954    - A     1:38  6888 26268  9116 65536   546   432  0.0  0.0
29 18:25 27954    - A     1:39  7110 26272  9796 65536   546   432  0.0  0.0
30 09:17 27954    - A     3:32 11276 36024 22568 65536   546   428  0.1  1.0
30 11:20 27954    - A     3:52 13004 36344 17784 65536   546   472  0.1  1.0
30 11:44 27954    - A     4:04 14081 37708 22340 65536   546   468  0.1  1.0
30 14:28 27954    - A     4:48 16478 37976 25656 65536   546   400  0.1  1.0
30 14:54 27954    - A     4:51 16790 37976 22900 65536   546   456  0.1  1.0
30 16:04 27954    - A     4:53 17634 38144 25700 65536   546   468  0.1  1.0

procengine
30 09:17 53160    - A     4:50  2895 31484  3776 65536    81    72  0.1  0.0
30 09:51 53160    - A     4:52  2946 31508  3792 65536    81    72  0.1  0.0
30 11:20 53160    - A     4:54  3042 31536  3884 65536    81    72  0.1  0.0
30 11:44 53160    - A     4:54  3072 31540  3772 65536    81    72  0.1  0.0
30 14:28 53160    - A     5:05  3121 31792  3944 65536    81    88  0.1  0.0
30 16:04 53160    - A     5:09  3384 31792  4276 65536    81    80  0.1  0.0

xxedi
29 16:12 39996    - A     5:20 10280 27576 15240 65536   546   432  0.1  1.0
29 18:06 39996    - A     5:50 10640 28932 17820 65536   546   432  0.1  1.0
29 18:25 39996    - A     6:19 10691 29756 18672 65536   546   432  0.1  1.0
30 09:17 39996    - A     6:21 14156 29892 17532 65536   546   428  0.1  1.0
30 11:20 39996    - A     9:03 15933 32888 19144 65536   546   472  0.1  1.0
30 11:44 39996    - A     9:13 17607 32888 22836 65536   546   456  0.1  1.0
30 14:28 39996    - A     9:24 17923 32888 21892 65536   546   400  0.1  1.0
30 18:xx 39996    - A    11:12 22041 35544 24712 65536   546   432  0.1  1.0

logger
30 16:09 59858    - A     1:26 19142 11564  6632 65536   270   176  0.0  0.0
30 17:38 59858    - A     1:28 19490 11564  3316 65536   270   176  0.0  0.0

queue
30 16:09 59344    - A    16:51  9023   916   584 65536   147   120  0.2  0.0
30 17:38 59344    - A    17:15  9240   916   516 65536   147   112  0.2  0.0

Hi Angela,

Moving from DBA to Unix Admin, huh?  :^}

Without a memory leak, memory WILL be returned to the O/S free pool as soon as the task hold it ends.  Since your tools are showing no memory leak(s), we need to look somewhere else.

My guess is that a child task is getting hung up.  The child task is mapped shares some of the same memory as its parent, or is mapped over a common region causing the memory to not be freed as THIS task is holding it.  Keep in mind that it may not be the parent's child task.  It may be the parent's child's child so a 'ps -ef' doesn't show a direct link to the original parent.  This suggests that the thread/task model is slightly different in the new version of AIX.  Probably a stray bit not being initialized/cleared.


Good Luck,
Kent

Hi Angela,

Angie, huh?  I'll remember.....

Your description is exactly in line with my expectations.  When you "clear the application" the memory is released.  This amounts to killing the task that is holding the memory.  Now, if we can isolate the problem, we'll still have to convince IBM or Hyperion to correct the issue, but that's down the line.

- How long does it take to clear the application?  sub-second?  seconds?  minutes?  The shorter that interval, the better our test will be.

- Can you clear the application with a script or is there a graceful shutdown procedure that you need to follow?

Basically, what I want to do is "snapshot" the running tasks before and after shutdown.  If you can, try running this two or three times:

Create a script with the following:

# Shutdown and snapshot script
ps -ef > $1.a
shutdown application
ps -ef > $1.b


Then run it as:

  shutdown_script.sh t1

restart the application, create the memory issue

 shutdown_script.sh t2

This will create the file pair t1.a / t1.b and t2.a / t2.b.  We want to do a diff on the pairs and see what tasks dropped.  Then we could have some idea about another task holding the memory.


Kent

Hi Angie,

4-day week job?

That means that you're free the rest of today?  Cool.  I could sure use a Supreme, thin-crust pizza.  As long as you're "free" today, can you deliver?


  :^}

Hi Angela_Wilcox,


You've obviously been non-single for entirely too long.  That's what microwaves are for.  :^}


Men, regardless of actual marital status have the "bachelor gene" and just naturallly understand that.

Plus a microwave is considered "consumer electronics".  Men don't have to understand it.  They just buy it and test all the buttons.


   ;)