?
Solved

/tmp issues - pfiles shows large file usage

Posted on 2005-04-05
25
Medium Priority
?
2,987 Views
Last Modified: 2013-12-27
We have been having an on and off issue with the /tmp directory filling up due to "phantom files." After doing some research on the last occurance, I see the following symptoms:

# swap -l
swapfile             dev  swaplo blocks   free
/dev/vx/dsk/swapvol 271,5      16 16780208 16779952

# swap -s
total: 2240520k bytes allocated + 600536k reserved = 2841056k used, 12066136k available

# df -k /tmp
Filesystem            kbytes    used   avail capacity  Mounted on
swap                 2097152  604640 1492512    29%    /tmp

pfiles command shows the following for one of the known processes contributing to the issue (edited to remove sensitive information):

# pfiles 21004
21004:  ns-httpd -d /opt/netscape/server4/https-instancename/confi
  Current rlimit: 1024 file descriptors
   0: S_IFIFO mode:0000 dev:314,0 ino:1897983 uid:0 gid:0 size:0
      O_RDWR
   1: S_IFREG mode:0644 dev:0,2 ino:37632521 uid:0 gid:0 size:364774539
      O_WRONLY|O_LARGEFILE
   2: S_IFREG mode:0644 dev:0,2 ino:37632521 uid:0 gid:0 size:364774539
      O_WRONLY|O_LARGEFILE
   3: S_IFCHR mode:0666 dev:271,0 ino:421213 uid:0 gid:3 rdev:13,12
      O_RDWR
   4: S_IFREG mode:0644 dev:271,10 ino:183 uid:0 gid:0 size:206
      O_RDONLY
   5: S_IFIFO mode:0000 dev:314,0 ino:1897985 uid:18048 gid:2030 size:0
      O_RDWR
   6: S_IFCHR mode:0666 dev:271,0 ino:421213 uid:0 gid:3 rdev:13,12
      O_RDWR
   7: S_IFREG mode:0644 dev:271,10 ino:2883 uid:18048 gid:2030 size:988828
      O_RDWR|O_APPEND|O_LARGEFILE
   8: S_IFIFO mode:0000 dev:314,0 ino:1897986 uid:18048 gid:2030 size:0
      O_RDWR
   9: S_IFREG mode:0644 dev:271,10 ino:501 uid:0 gid:0 size:229376
      O_RDONLY FD_CLOEXEC
  10: S_IFREG mode:0644 dev:271,10 ino:2884 uid:18048 gid:2030 size:4379676
      O_RDWR|O_APPEND|O_LARGEFILE FD_CLOEXEC
  11: S_IFREG mode:0644 dev:0,2 ino:26330833 uid:18048 gid:2030 size:39964
      O_RDWR|O_LARGEFILE
  12: S_IFCHR mode:0666 dev:271,0 ino:421213 uid:0 gid:3 rdev:13,12
      O_RDONLY
  13: S_IFDIR mode:0511 dev:312,0 ino:12355 uid:18048 gid:2030 size:736
      O_RDONLY
  14: S_IFCHR mode:0666 dev:271,0 ino:421212 uid:0 gid:3 rdev:13,2
      O_RDONLY
  15: S_IFCHR mode:0666 dev:271,0 ino:421212 uid:0 gid:3 rdev:13,2
      O_RDONLY
  16: S_IFREG mode:0644 dev:271,11 ino:42819 uid:18048 gid:2030 size:4821374
      O_WRONLY|O_LARGEFILE
  17: S_IFSOCK mode:0666 dev:313,0 ino:52338 uid:0 gid:0 size:0
      O_RDWR|O_NONBLOCK
        sockname: AF_INET x.x.x.x port: xxxx
        peername: AF_INET x.x.x.x  port: xxxx
  20: S_IFSOCK mode:0666 dev:313,0 ino:52342 uid:0 gid:0 size:0
      O_RDWR|O_NONBLOCK
        sockname: AF_INET x.x.x.x port: xxxx
        peername: AF_INET x.x.x.x port: xxxx
  21: S_IFSOCK mode:0666 dev:313,0 ino:47220 uid:0 gid:0 size:0
      O_RDWR|O_NONBLOCK
        sockname: AF_INET x.x.x.x port: xxxx
        peername: AF_INET x.x.x.x port: xxxx
  23: S_IFSOCK mode:0666 dev:313,0 ino:36864 uid:0 gid:0 size:0
      O_RDWR|O_NONBLOCK
        sockname: AF_INET x.x.x.x port: xxxx
        peername: AF_INET x.x.x.x port: xxxx
 260: S_IFSOCK mode:0666 dev:313,0 ino:4349 uid:0 gid:0 size:0
      O_RDWR|O_NONBLOCK FD_CLOEXEC
        sockname: AF_INET x.x.x.x port: xxxx


I have been monitoring throught the course of the day and have noticed the listed items in pfiles that have large "sizes" on them continue to grow as I monitor them. I believe these are where the issues are at. Also, to note, when the /tmp hits 100%, memory usage is normal (about half of total available memory). Du also shows no large files listed in any directory under /tmp. (please see original question http://www.experts-exchange.com/Operating_Systems/Solaris/Q_21350012.html).  Is there a way for us to tell what the items listed under pfiles reflect and how to identify the "problem" file? We need to determine the cause of the problem, however, I believe I am missing a few items that would help us to do so...

thanks
0
Comment
Question by:RevelationCS
  • 12
  • 7
  • 2
  • +2
24 Comments
 
LVL 10

Expert Comment

by:neteducation
ID: 13713591
# pfiles 21004
21004:  ns-httpd -d /opt/netscape/server4/https-instancename/confi
  Current rlimit: 1024 file descriptors

Which Version of Netscape 4 Webserver ar you using? I remeber of some having a problem opening up temporary files to talk to CGI-Programs, and not cleaning them if the cgi didnt respond in a timely way.

(And for personal opinion... go to the new 6.1 Webserver from sun... much faster and rock-solid)
0
 
LVL 8

Author Comment

by:RevelationCS
ID: 13713954
using 4.1 with the last service pack I believe... this is only one of the webservers running on the servers with this issue... the others are running 6.1 (and I agree and have been bugging the development managers to work on getting the apps upgraded)... does it affect anything other than CGI? all the apps running in the environment are running java/jsps...
0
 

Expert Comment

by:garypen
ID: 13714242
If this was Solaris 10, your question would be very easy to answer since pfiles has been upgraded to include pathnames in it's output where possible:

e.g.

bash-3.00# ps -p 23397
   PID TTY         TIME CMD
 23397 ?           1:20 httpd
bash-3.00# pfiles 23397
23397:  /usr/apache2/bin/httpd -k start
  Current rlimit: 65536 file descriptors
   0: S_IFCHR mode:0666 dev:270,0 ino:6815752 uid:0 gid:3 rdev:13,2
      O_RDONLY
      /devices/pseudo/mm@0:null
   1: S_IFCHR mode:0666 dev:270,0 ino:6815752 uid:0 gid:3 rdev:13,2
      O_WRONLY|O_CREAT|O_TRUNC
      /devices/pseudo/mm@0:null
   2: S_IFREG mode:0644 dev:102,0 ino:114297 uid:0 gid:0 size:322900
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
      /var/apache2/logs/error_log
   3: S_IFSOCK mode:0666 dev:276,0 ino:26677 uid:0 gid:0 size:0
      O_RDWR
        SOCK_STREAM
        SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(49152)
        sockname: AF_INET6 ::  port: 80
   4: S_IFDOOR mode:0444 dev:279,0 ino:57 uid:0 gid:0 size:0
      O_RDONLY|O_LARGEFILE FD_CLOEXEC  door to nscd[125]
      /var/run/name_service_door
   5: S_IFIFO mode:0000 dev:277,0 ino:73320 uid:0 gid:0 size:0
      O_RDWR|O_NONBLOCK
   6: S_IFIFO mode:0000 dev:277,0 ino:73320 uid:0 gid:0 size:0
      O_RDWR
   7: S_IFREG mode:0644 dev:102,0 ino:114297 uid:0 gid:0 size:322900
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
      /var/apache2/logs/error_log
   8: S_IFREG mode:0644 dev:102,0 ino:114298 uid:0 gid:0 size:413909
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
      /var/apache2/logs/access_log
   9: S_IFREG mode:0600 dev:274,2 ino:446347586 uid:0 gid:0 size:0
      O_RDWR|O_CREAT|O_EXCL
  10: S_IFREG mode:0644 dev:274,3 ino:451283270 uid:0 gid:0 size:0
      O_WRONLY|O_CREAT|O_EXCL
bash-3.00#

On older versions of Solaris you are stuck.

You may be able to work out what is happening using truss. You could try to truss the script you use to start up your webserver using something like:

truss -topen -f -o /tmp/trace <command>

This will trace all calls to open, in all sub-processes of command and write the output to /tmp/trace. After starting your server, stop it and then look at the /tmp/trace file to see which files are opened in /tmp. This may give you a clue about which files are being used to hold data in /tmp.

However, I'm not sure that this will help you much. It is probable that your webserver is just not behaving well and knowing that won't help you apart from providing more ammunition to your argument to upgrade the webserver to a newer (presumably less buggy) version.

I think you are barking up the wrong tree. The output above doesn't really have any large files listed apart from fd 1 and 2. These are (probably) the log file output and are clearly the same file (same dev/inodes). I don't expect you are storing your log file output in /tmp and so I don't really think that there is any evidence from the above to show that you have problems with large unidentified files living in /tmp. It's more likely (since you are using Java) that you are simply using large amounts of virtual memory from your Java VMs. Try running:

pmap `pgrep java`

Look at the totals and see if this is your problem. Java is notorious for eating up virtual memory and it's much more likely this is the cause of your problem.


0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
LVL 10

Expert Comment

by:Nukfror
ID: 13723390
Actually not totally stuck.  The dev field in the <Sol10 pfiles output does give valuable information.  A simple example:

bash-2.03$ ps -ef|grep view
 soneill  5024  4987  0 22:21:02 pts/1    0:00 view /etc/hosts
 soneill  5054  5029  0 22:21:42 pts/2    0:00 grep view
bash-2.03$ pfiles 5024
5024:   view /etc/hosts
  Current rlimit: 2048 file descriptors
   0: S_IFCHR mode:0620 dev:85,1 ino:83288 uid:520 gid:7 rdev:24,1
      O_RDWR|O_LARGEFILE
   1: S_IFCHR mode:0620 dev:85,1 ino:83288 uid:520 gid:7 rdev:24,1
      O_RDWR|O_LARGEFILE
   2: S_IFCHR mode:0620 dev:85,1 ino:83288 uid:520 gid:7 rdev:24,1
      O_RDWR|O_LARGEFILE
   3: S_IFCHR mode:0666 dev:85,1 ino:83119 uid:0 gid:3 rdev:13,12
      O_RDWR
   4: S_IFREG mode:0600 dev:85,2 ino:60426 uid:520 gid:520 size:24576
      O_RDWR

Take a look at FD 4. The dev field = First number is the major device # and second is the minor.  So 85,2 is:

cd /devices/pseudo
ls -la | less
.....<snip>
brw-r-----   1 root     sys       85,  2 Feb  4  2001 md@0:0,2,blk
crw-r-----   1 root     sys       85,  2 Feb  4  2001 md@0:0,2,raw

So this is obviously a SVM mount point.

bash-2.03# cd /dev/md/dsk
bash-2.03# ls -la | grep "md@0:0,2"
lrwxrwxrwx   1 root     other         36 Feb  4  2001 d2 -> ../../../devices/pseudo/md@0:0,2,blk
<snip>

d2 is /var mount point:

bash-2.03# df -k
Filesystem            kbytes    used   avail capacity  Mounted on
<snip>
/dev/md/dsk/d2        123231    8317  102591     8%    /var

The pfiles ino field is the inode number.  That inode is related to file:

bash-2.03# ncheck -i 60426 /dev/md/dsk/d2
/dev/md/dsk/d2:
60426   /tmp/ExJjaa0j

The /tmp is a *RELATIVE* reference to the /var mount point.

(You could have also used find to look for the inode number.  Find obeys mount points boundries.  ncheck goes down to the physical partition.  In most cases find is probably a better command to use - but ncheck is rather obscure so kinda kewl ;) )

Notice the size of the file is 24576 as reported by pfiles.    ls -la shows the same mode bits, size, owner, and group (520 is soneill on this box) so this is refering to the same file.

cd /var/tmp
bash-2.03# ls -lai ExJjaa0j
     60426 -rw-------   1 soneill  soneill    24576 Apr  6 22:21 ExJjaa0j

So you aren't totally stuck if you just do a little digging on each file descriptor.
0
 
LVL 10

Expert Comment

by:Nukfror
ID: 13723419
And a comment on your original command output.

# swap -l
swapfile             dev  swaplo blocks   free
/dev/vx/dsk/swapvol 271,5      16 16780208 16779952

# swap -s
total: 2240520k bytes allocated + 600536k reserved = 2841056k used, 12066136k available

You have 4 GB of memory and 8 GB of swap.  PHYSICAL swap (in this output) you using barely 131KBytes.  VIRTUAL swap (e.g. anonymous memory) you've allocated over 2GB.

Run prstat and look for applications with large memory utilizations then run pmap against each of these application over time.

bash-2.03# pmap -x 5024
5024:   view /etc/hosts
 Address   Kbytes Resident Shared Private Permissions       Mapped File
00010000     188       8       8       - read/exec         view
0004E000      40      20       -      20 read/write/exec   view
00058000     432     104       -     104 read/write/exec     [ heap ]
EF380000     688     688     688       - read/exec         libc.so.1
EF43C000      28      28       -      28 read/write/exec   libc.so.1
EF443000       4       4       -       4 read/write/exec   libc.so.1
EF480000      24      24      24       - read/exec         libgen.so.1
EF496000       4       4       -       4 read/write/exec   libgen.so.1
EF500000       8       8       8       - read/exec         libcrypt_i.so.1
EF512000       4       4       -       4 read/write/exec   libcrypt_i.so.1
EF580000     164     124     124       - read/exec         libcurses.so.1
EF5BA000      32      32      12      20 read/write/exec   libcurses.so.1
EF5C2000       8       8       -       8 read/write/exec   libcurses.so.1
EF600000       8       8       8       - read/exec         libmapmalloc.so.1
EF612000       4       4       -       4 read/write/exec   libmapmalloc.so.1
EF680000       4       4       4       - read/exec         libdl.so.1
EF700000       4       4       4       - read/shared       dev:85,2 ino:60443
EF780000     156     156     156       - read/exec         ld.so.1
EF7B8000       8       8       -       8 read/write/exec   ld.so.1
EF7C0000       4       4       -       4 read/write          [ anon ]
EF7D0000       4       4       -       4 read/write/exec     [ anon ]
EF7E0000       4       4       -       4 read/write/exec     [ anon ]
EF7F0000       4       4       -       4 read/write/exec     [ anon ]
EFFFB000      20      20       -      20 read/write          [ stack ]
--------  ------  ------  ------  ------
total Kb    1844    1276    1036     240

You interested in the segments parts the eat anonymous memory like stack/heap space.  Look at the Private column number and look for growing numbers over time.  Notice some segments are flagged as [ anon ], [ heap ], and [ stack ] showing you the candidates to look for.
0
 
LVL 10

Expert Comment

by:neteducation
ID: 13724339
i had the problem only with cgi's, but basically the concept of jsp is pretty much the same as with cgi's... just that you use java as the program.... so I could very well imagine it is the same problem. btw: my "solution" for that problem was to restart the daemon once a night, and to delete the older temporary files in /tmp. (plus in my case kill the cgi-processes that were hanging)
0
 

Expert Comment

by:garypen
ID: 13724570
There is a flaw with Nukfor's approach. AFAIK 'ncheck' only supports UFS filesystems. Since the question seems to be about a tmpfs filesystem, namely /tmp, this approach won't help.

Anyway, it looks like Nukfor agrees with me that the problem isn't caused by unidentified files but by processes consuming large amounts of physical memory which can be best identified using pmap.

If the problem is caused by your JSP processing, then you can try to control the problem by judicious tuning of your webserver/application server to restrict memory usage.

Your webserver is ancient and so it's likely that it fork/execs a separate JVM for each JSP page hit and this is a very inefficient and memory greedy approach likely to cause problems such as you see.

You should upgrade your application server environment to something more modern which can cache JSP pages and also processes page hits in memory and avoids the huge fork/exec overhead I mentioned earlier.

Such work is of course no longer Solaris tuning or a problem with Solaris and you may get better answers to further questions in the "Web Dev" section.

0
 
LVL 10

Expert Comment

by:Nukfror
ID: 13726421
Garypen, true, thanks for the clarification [in the back of my head I knew this - simply didn't type it] - but notice my comment about "find" so this approach still works simply replace ncheck with find.

In most cases, a program reading in large files doesn't eat up large amounts of swap space - it eats up FS Page cache memory.  There are obviously exceptions to every case.  In most costs, lots of anonymous memory could get eaten when a program uses mmap() to pull in a file into its address space - not when using read() or write().

I agree with garypen that this isn't necessarily an issue with something "opening" large files in anonymous memory.  This has something to do with some process eating up anonymous memory with stack/heap.  Assuming Java is the issue, just run:

ps -alxww | grep java

And look for how Java with switch -ms and -mx (see man page for description).  Most folks I've seen specify this in MBytes.  A pmap -x against your java processes would be interesting as well.
0
 
LVL 8

Author Comment

by:RevelationCS
ID: 13727120
for one of the offending processes....

# pmap -x 14224
14224:  ns-httpd -d /opt/netscape/server4/<instance name>/confi
 Address   Kbytes Resident Shared Private Permissions       Mapped File
00010000     584     584     584       - read/exec         ns-httpd
000B0000     128     128      80      48 read/write/exec   ns-httpd
000D0000   25648   25432       -   25432 read/write/exec     [ heap ]
F57B4000       8       8       -       8 read/write          [ anon ]
F57F4000       8       8       -       8 read/write          [ anon ]
F5804000       8       8       -       8 read/write          [ anon ]
F5850000      24      24      24       - read              dev:271,11 ino:29297
F5860000      88      88      88       - read              dev:271,11 ino:39380
F5880000       8       8       8       - read              dev:271,11 ino:18484
F5890000      40      40      40       - read              dev:271,11 ino:18339
F58A0000      40      40      40       - read              dev:271,11 ino:18341
F58B0000      32      32      32       - read              dev:271,11 ino:39323
F58C0000       8       8       8       - read              dev:271,11 ino:18797
F58D0000       8       8       8       - read              dev:271,11 ino:19075
F58E0000       8       8       8       - read              dev:271,11 ino:29265
F58F0000       8       8       8       - read              dev:271,11 ino:37391
F5904000       8       8       -       8 read/write          [ anon ]
F5910000       8       8       8       - read              dev:271,11 ino:18823
F5920000       8       8       8       - read              dev:271,11 ino:40694
F5932000       8       8       -       8 read/write          [ anon ]
F5954000       8       8       -       8 read/write          [ anon ]
F5960000       8       8       8       - read              dev:271,11 ino:18745
F5974000       8       8       -       8 read/write          [ anon ]
F5980000       8       8       8       - read              dev:271,11 ino:18749
F5990000       8       8       8       - read              dev:271,11 ino:37162
F59A0000       8       8       8       - read              dev:271,11 ino:37053
F59B4000       8       8       -       8 read/write          [ anon ]
F59C0000      16      16      16       - read              dev:271,11 ino:30956
F59D0000       8       8       8       - read              dev:271,11 ino:40825
F59E0000       8       8       8       - read              dev:271,11 ino:40828
F59F0000       8       8       8       - read              dev:271,11 ino:18251
F5A00000       8       8       8       - read              dev:271,11 ino:38274
F5A10000       8       8       8       - read              dev:271,11 ino:38247
F5A20000       8       8       8       - read              dev:271,11 ino:38273
F5A30000       8       8       8       - read              dev:271,11 ino:31456
F5A44000       8       8       -       8 read/write          [ anon ]
F5A50000       8       8       8       - read              dev:271,11 ino:40085
F5A70000       8       8       8       - read              dev:271,11 ino:18812
F5A90000      16      16      16       - read              dev:271,11 ino:18821
F5AA0000       8       8       8       - read              dev:271,11 ino:38640
F5AB0000       8       8       8       - read              dev:271,11 ino:37368
F5AC2000       8       8       -       8 read/write          [ anon ]
F5AD0000       8       8       8       - read              dev:271,11 ino:39937
F5AE0000      40      40      40       - read              dev:271,11 ino:33011
F5AF0000      40      40      40       - read              dev:271,11 ino:18574
F5B00000       8       8       8       - read              dev:271,11 ino:42331
F5B10000       8       8       8       - read              dev:271,11 ino:42335
F5B20000       8       8       8       - read              dev:271,11 ino:42437
F5B30000       8       8       8       - read              dev:271,11 ino:29074
F5B40000       8       8       8       - read              dev:271,11 ino:29115
F5B50000       8       8       8       - read              dev:271,11 ino:19329
F5B60000       8       8       8       - read              dev:271,11 ino:41064
F5B70000       8       8       8       - read              dev:271,11 ino:18797
F5B80000       8       8       8       - read              dev:271,11 ino:40571
F5B90000       8       8       8       - read              dev:271,11 ino:18255
F5BA0000       8       8       8       - read              dev:271,11 ino:34271
F5BB0000       8       8       8       - read              dev:271,11 ino:38332
F5BC0000      32      32      32       - read              dev:271,11 ino:33005
F5BD0000       8       8       8       - read              dev:271,11 ino:40261
F5BE0000       8       8       8       - read              dev:271,11 ino:34050
F5BF0000       8       8       8       - read              dev:271,11 ino:34073
F5C00000       8       8       8       - read              dev:271,11 ino:40692
F5C10000       8       8       8       - read              dev:271,11 ino:40690
F5C20000       8       8       8       - read              dev:271,11 ino:18679
F5C30000       8       8       8       - read              dev:271,11 ino:33010
F5C40000     192     192     192       - read              dev:271,11 ino:38594
F5C80000       8       8       8       - read              dev:271,11 ino:28642
F5C90000       8       8       8       - read              dev:271,11 ino:28638
F5CA0000       8       8       8       - read              dev:271,11 ino:28644
F5CB0000       8       8       8       - read              dev:271,11 ino:34306
F5CC0000      24      24      24       - read              dev:271,11 ino:18374
F5CD0000       8       8       8       - read              dev:271,11 ino:35778
F5CE0000      16      16      16       - read              dev:271,11 ino:30806
F5CF0000       8       8       8       - read              dev:271,11 ino:19126
F5D00000       8       8       8       - read              dev:271,11 ino:31213
F5D10000      16      16      16       - read              dev:271,11 ino:38015
F5D3A000      32      32       -      32 read/write          [ anon ]
F5D50000       8       8       8       - read              dev:271,11 ino:38023
F5D7A000      32      32       -      32 read/write          [ anon ]
F5DAA000      32      32       -      32 read/write          [ anon ]
F5DC0000       8       8       8       - read              dev:271,11 ino:39090
F5DD0000       8       8       8       - read              dev:271,11 ino:40598
F5DE0000       8       8       8       - read              dev:271,11 ino:29296
F5DF0000       8       8       8       - read              dev:271,11 ino:34925
F5E00000       8       8       8       - read              dev:271,11 ino:34961
F5E10000      16      16      16       - read              dev:271,11 ino:40687
F5E24000       8       8       -       8 read/write          [ anon ]
F5E30000       8       8       8       - read              dev:271,11 ino:27593
F5E40000      32      32      32       - read              dev:271,11 ino:38601
F5E50000     144     144     144       - read              dev:271,11 ino:18924
F5E80000      24      24      24       - read              dev:271,11 ino:39115
F5E90000      16      16      16       - read              dev:271,11 ino:38602
F5EA0000      16      16      16       - read              dev:271,11 ino:38644
F5EB0000       8       8       8       - read              dev:271,11 ino:40648
F5EC0000       8       8       8       - read              dev:271,11 ino:41869
F5ED0000       8       8       8       - read              dev:271,11 ino:34920
F5EE0000      16      16      16       - read              dev:271,11 ino:39113
F5EF0000       8       8       8       - read              dev:271,11 ino:35393
F5F00000       8       8       8       - read              dev:271,11 ino:18249
F5F10000       8       8       8       - read              dev:271,11 ino:31577
F5F20000       8       8       8       - read              dev:271,11 ino:19110
F5F30000      96      96      96       - read              dev:271,11 ino:28711
F5F50000      16      16      16       - read              dev:271,11 ino:19303
F5F60000      32      32      32       - read              dev:271,11 ino:30855
F5F70000       8       8       8       - read              dev:271,11 ino:18952
F5F80000      16      16      16       - read              dev:271,11 ino:26831
F5F90000       8       8       8       - read              dev:271,11 ino:40865
F5FA0000       8       8       8       - read              dev:271,11 ino:26828
F5FB0000       8       8       8       - read              dev:271,11 ino:38430
F5FC0000       8       8       8       - read              dev:271,11 ino:18918
F5FD0000       8       8       8       - read              dev:271,11 ino:31611
F5FE0000       8       8       8       - read              dev:271,11 ino:19076
F5FF0000       8       8       8       - read              dev:271,11 ino:31581
F6000000       8       8       8       - read              dev:271,11 ino:31609
F6010000       8       8       8       - read              dev:271,11 ino:28674
F6020000     176     176     176       - read              dev:271,11 ino:31315
F6050000       8       8       8       - read              dev:271,11 ino:28590
F6060000       8       8       8       - read              dev:271,11 ino:17860
F6070000       8       8       8       - read              dev:271,11 ino:31202
F6080000       8       8       8       - read              dev:271,11 ino:18221
F6090000       8       8       8       - read              dev:271,11 ino:18743
F60BA000      32      32       -      32 read/write          [ anon ]
F60D0000      16      16      16       - read              dev:271,11 ino:39324
F60FA000      32      32       -      32 read/write          [ anon ]
F6110000       8       8       8       - read              dev:271,11 ino:18330
F613A000      32      32       -      32 read/write          [ anon ]
F6150000       8       8       8       - read              dev:271,11 ino:18713
F6160000       8       8       8       - read              dev:271,11 ino:30932
F6170000       8       8       8       - read              dev:271,11 ino:39094
F6180000       8       8       8       - read              dev:271,11 ino:39321
F6190000      16      16      16       - read              dev:271,11 ino:40404
F61A0000       8       8       8       - read              dev:271,11 ino:39117
F61B0000       8       8       8       - read              dev:271,11 ino:40265
F61C0000      24      24      24       - read              dev:271,11 ino:42246
F61D0000      40      40      40       - read              dev:271,11 ino:42244
F61E0000       8       8       8       - read              dev:271,11 ino:37392
F61F0000       8       8       8       - read              dev:271,11 ino:18956
F6200000       8       8       8       - read              dev:271,11 ino:18950
F6210000       8       8       8       - read              dev:271,11 ino:18920
F6220000       8       8       8       - read              dev:271,11 ino:18568
F6230000      32      32      32       - read              dev:271,11 ino:36869
F6240000       8       8       8       - read              dev:271,11 ino:36858
F6250000      80      80      80       - read              dev:271,11 ino:36847
F6270000      16      16      16       - read              dev:271,11 ino:36776
F6280000       8       8       8       - read              dev:271,11 ino:30901
F6290000       8       8       8       - read              dev:271,11 ino:18638
F62A0000       8       8       8       - read              dev:271,11 ino:18507
F62B0000       8       8       8       - read              dev:271,11 ino:33009
F62DA000      32      32       -      32 read/write          [ anon ]
F62F0000       8       8       8       - read              dev:271,11 ino:30836
F631A000      32      32       -      32 read/write          [ anon ]
F6330000       8       8       8       - read              dev:271,11 ino:39294
F6340000       8       8       8       - read              dev:271,11 ino:42249
F6350000       8       8       8       - read              dev:271,11 ino:18989
F6360000       8       8       8       - read              dev:271,11 ino:18970
F6370000      16      16      16       - read              dev:271,11 ino:42997
F6380000       8       8       8       - read              dev:271,11 ino:30957
F6390000       8       8       8       - read              dev:271,11 ino:42714
F63A0000       8       8       8       - read              dev:271,11 ino:28586
F63B0000       8       8       8       - read              dev:271,11 ino:28592
F63C0000       8       8       8       - read              dev:271,11 ino:28622
F63D0000      16      16      16       - read              dev:271,11 ino:38433
F63E4000       8       8       -       8 read/write          [ anon ]
F63F0000       8       8       8       - read              dev:271,11 ino:38472
F6400000       8       8       8       - read              dev:271,11 ino:28716
F6410000       8       8       8       - read              dev:271,11 ino:39322
F643A000      32      32       -      32 read/write          [ anon ]
F6450000      16      16      16       - read              dev:271,11 ino:40599
F6460000       8       8       8       - read              dev:271,11 ino:40597
F648A000      32      32       -      32 read/write          [ anon ]
F64A0000       8       8       8       - read              dev:271,11 ino:33194
F64B0000       8       8       8       - read              dev:271,11 ino:33175
F64C0000      24      24      24       - read              dev:271,11 ino:42799
F64D0000       8       8       8       - read              dev:271,11 ino:28685
F64FA000      32      32       -      32 read/write          [ anon ]
F6510000       8       8       8       - read              dev:271,11 ino:38485
F653A000      32      32       -      32 read/write          [ anon ]
F6550000       8       8       8       - read              dev:271,11 ino:19741
F6560000      16      16      16       - read              dev:271,11 ino:33215
F6570000      16      16      16       - read              dev:271,11 ino:35780
F6580000       8       8       8       - read              dev:271,11 ino:18365
F6590000      40      40      40       - read              dev:271,11 ino:41868
F65A0000       8       8       8       - read              dev:271,11 ino:42067
F65B0000       8       8       8       - read              dev:271,11 ino:41864
F65C0000       8       8       8       - read              dev:271,11 ino:27622
F65D0000       8       8       8       - read              dev:271,11 ino:38434
F65E0000      24      24      24       - read              dev:271,11 ino:38429
F660A000      32      32       -      32 read/write          [ anon ]
F6620000      16      16      16       - read              dev:271,11 ino:33217
F6630000       8       8       8       - read              dev:271,11 ino:18994
F6640000       8       8       8       - read              dev:271,11 ino:38406
F6650000       8       8       8       - read              dev:271,11 ino:38610
F6660000       8       8       8       - read              dev:271,11 ino:31281
F6670000       8       8       8       - read              dev:271,11 ino:31277
F6680000       8       8       8       - read              dev:271,11 ino:31283
F6690000       8       8       8       - read              dev:271,11 ino:31320
F66A0000       8       8       8       - read              dev:271,11 ino:41863
F66B0000       8       8       8       - read              dev:271,11 ino:19137
F66C0000      16      16      16       - read              dev:271,11 ino:19135
F66D0000       8       8       8       - read              dev:271,11 ino:28726
F66E0000       8       8       8       - read              dev:271,11 ino:31874
F66F0000       8       8       8       - read              dev:271,11 ino:32017
F6700000      32      32      32       - read              dev:271,11 ino:18360
F672A000      32      32       -      32 read/write          [ anon ]
F6740000       8       8       8       - read              dev:271,11 ino:18337
F6750000      16      16      16       - read              dev:271,11 ino:18575
F6760000       8       8       8       - read              dev:271,11 ino:40111
F6770000       8       8       8       - read              dev:271,11 ino:19304
F6780000      16      16      16       - read              dev:271,11 ino:32322
F6790000       8       8       8       - read              dev:271,11 ino:32321
F67A0000       8       8       8       - read              dev:271,11 ino:32325
F67B0000      16      16      16       - read              dev:271,11 ino:32242
F67C0000      16      16      16       - read              dev:271,11 ino:38645
F67D0000      56      56      56       - read              dev:271,11 ino:31324
F67F0000       8       8       8       - read              dev:271,11 ino:38598
F6800000       8       8       8       - read              dev:271,11 ino:38569
F6810000      16      16      16       - read              dev:271,11 ino:38603
F6820000       8       8       8       - read              dev:271,11 ino:33012
F6830000       8       8       8       - read              dev:271,11 ino:18600
F6840000       8       8       8       - read              dev:271,11 ino:30925
F6850000       8       8       8       - read              dev:271,11 ino:18613
F6860000       8       8       8       - read              dev:271,11 ino:38431
F6870000      16      16      16       - read              dev:271,11 ino:18634
F689A000      32      32       -      32 read/write          [ anon ]
F68B0000       8       8       8       - read              dev:271,11 ino:38432
F68DA000      32      32       -      32 read/write          [ anon ]
F68F0000      32      32      32       - read              dev:271,11 ino:18595
F6900000       8       8       8       - read              dev:271,11 ino:31851
F6910000       8       8       8       - read              dev:271,11 ino:42713
F693A000      32      32       -      32 read/write          [ anon ]
F6950000       8       8       8       - read              dev:271,11 ino:18384
F6960000       8       8       8       - read              dev:271,11 ino:18386
F6970000      16      16      16       - read/exec         nss_dns.so.1
F6984000       8       8       -       8 read/write/exec   nss_dns.so.1
F6990000      40      40      40       - read/exec         libnet.so
F69A8000       8       8       -       8 read/write/exec   libnet.so
F69B0000       8       8       8       - read              dev:271,11 ino:18383
F69C0000      24      24      24       - read              dev:271,11 ino:28715
F69EE000      16      16       -      16 read/write          [ anon ]
F6A00000       8       8       8       - read              dev:271,11 ino:32400
F6A2A000      32      32       -      32 read/write          [ anon ]
F6A5A000      32      32       -      32 read/write          [ anon ]
F6A70000       8       8       8       - read              dev:271,11 ino:18376
F6A80000       8       8       8       - read              dev:271,11 ino:18573
F6AAA000      32      32       -      32 read/write          [ anon ]
F6ADA000      32      32       -      32 read/write          [ anon ]
F6B0A000      32      32       -      32 read/write          [ anon ]
F6B3A000      32      32       -      32 read/write          [ anon ]
F6B6A000      32      32       -      32 read/write          [ anon ]
F6B9A000      32      32       -      32 read/write          [ anon ]
F6BCA000      32      32       -      32 read/write          [ anon ]
F6BF8000      40      40       -      40 read/write          [ anon ]
F6C2A000      32      32       -      32 read/write          [ anon ]
F6C5A000      32      32       -      32 read/write          [ anon ]
F6C8A000      32      32       -      32 read/write          [ anon ]
F6CBA000      32      32       -      32 read/write          [ anon ]
F6CEA000      32      32       -      32 read/write          [ anon ]
F6D1A000      32      32       -      32 read/write          [ anon ]
F6D4A000      32      32       -      32 read/write          [ anon ]
F6D7A000      32      32       -      32 read/write          [ anon ]
F6DAA000      32      32       -      32 read/write          [ anon ]
F6DDA000      32      32       -      32 read/write          [ anon ]
F6E0A000      32      32       -      32 read/write          [ anon ]
F6E3A000      32      32       -      32 read/write          [ anon ]
F6E68000      40      40       -      40 read/write          [ anon ]
F6E9A000      32      32       -      32 read/write          [ anon ]
F6ECA000      32      32       -      32 read/write          [ anon ]
F6EFA000      32      32       -      32 read/write          [ anon ]
F6F2A000      32      32       -      32 read/write          [ anon ]
F6F5A000      32      32       -      32 read/write          [ anon ]
F6F8A000      32      32       -      32 read/write          [ anon ]
F6FBA000      32      32       -      32 read/write          [ anon ]
F6FEA000      32      32       -      32 read/write          [ anon ]
F701A000      32      32       -      32 read/write          [ anon ]
F704A000      32      32       -      32 read/write          [ anon ]
F707A000      32      32       -      32 read/write          [ anon ]
F70AA000      32      32       -      32 read/write          [ anon ]
F70DA000      32      32       -      32 read/write          [ anon ]
F710A000      32      32       -      32 read/write          [ anon ]
F713A000      32      32       -      32 read/write          [ anon ]
F716A000      32      32       -      32 read/write          [ anon ]
F719A000      32      32       -      32 read/write          [ anon ]
F71C6000      48      48       -      48 read/write          [ anon ]
F71FA000      32      32       -      32 read/write          [ anon ]
F722A000      32      32       -      32 read/write          [ anon ]
F725A000      32      32       -      32 read/write          [ anon ]
F728A000      32      32       -      32 read/write          [ anon ]
F72BA000      32      32       -      32 read/write          [ anon ]
F72EA000      32      32       -      32 read/write          [ anon ]
F731A000      32      32       -      32 read/write          [ anon ]
F734A000      32      32       -      32 read/write          [ anon ]
F737A000      32      32       -      32 read/write          [ anon ]
F73A0000       8       8       -       8 read/write          [ anon ]
F73B0000       8       8       8       - read              dev:271,11 ino:40262
F73C0000      32      32      32       - read              dev:271,11 ino:37108
F73E0000       8       8       -       8 read/write          [ anon ]
F740E000      16      16       -      16 read/write          [ anon ]
F7420000      56      56      56       - read/exec         libmath.so
F743C000       8       8       -       8 read/write/exec   libmath.so
F7452000       8       8       -       8 read/write          [ anon ]
F7460000      16      16      16       - read              dev:271,11 ino:37126
F7470000      24      24      24       - read              dev:271,11 ino:18612
F7482000       8       8       -       8 read/write          [ anon ]
F7490000      24      24      24       - read              dev:271,11 ino:40157
F74A4000       8       8       -       8 read/write          [ anon ]
F74B0000     176     176     176       - read/shared       dev:271,10 ino:1400
F74E0000       8       8       8       - read/shared       dev:271,10 ino:1401
F74F0000       8       8       8       - read/shared       dev:271,10 ino:1394
F7500000     240     240     240       - read/shared       dev:271,10 ino:1390
F7540000     136     136     136       - read/shared       dev:271,10 ino:1389
F7570000      32      32      32       - read/shared       dev:271,10 ino:1391
F7580000      40      40      40       - read/shared       dev:271,10 ino:1385
F7590000       8       8       8       - read/shared       dev:271,10 ino:1384
F75A0000     456     456     456       - read/shared       dev:271,11 ino:40919
F7620000     368     368     368       - read/shared       dev:271,11 ino:29388
F7680000     624     624     624       - read/shared       dev:271,11 ino:40925
F7730000      24      24      24       - read/shared       dev:271,10 ino:1383
F7740000     192     192     192       - read/shared       dev:271,11 ino:21873
F7780000     584     584     584       - read/shared       dev:271,11 ino:21890
F7820000     104     104     104       - read/shared       dev:271,10 ino:1392
F7840000      24      24      24       - read/shared       dev:271,11 ino:37049
F7850000      32      32      32       - read/shared       dev:271,11 ino:40924
F7860000       8       8       8       - read/shared       dev:271,11 ino:23830
F7870000       8       8       8       - read/shared       dev:271,11 ino:23829
F7880000      16      16      16       - read/shared       dev:271,11 ino:24584
F7890000      24      24      24       - read/shared       dev:271,11 ino:23821
F78A0000      56      56      56       - read/shared       dev:271,11 ino:23827
F78C0000     456     456     456       - read/shared       dev:271,11 ino:24582
F7940000      16      16      16       - read/shared       dev:271,11 ino:23833
F7950000     104     104     104       - read/shared       dev:271,11 ino:23822
F7970000     152     152     152       - read/shared       dev:271,11 ino:23826
F79A0000     280     280     280       - read/shared       dev:271,11 ino:23825
F79F0000      48      48      48       - read/shared       dev:271,11 ino:23806
F7A00000    1608    1608    1608       - read/shared       dev:271,11 ino:19762
F7BA0000      16      16      16       - read/shared       dev:271,11 ino:23831
F7BB0000       8       8       8       - read/shared       dev:271,11 ino:19758
F7BC0000     128     128     128       - read/shared       dev:271,11 ino:19757
F7BF0000       8       8       8       - read/shared       dev:271,11 ino:19755
F7C00000   10752    6832    6832       - read/shared       dev:271,0 ino:54540
F8690000      72      72      72       - read/shared       dev:271,11 ino:19751
F86B0000      32      32      32       - read/shared       dev:271,11 ino:23820
F86C0000     200     200     200       - read/shared       dev:271,11 ino:23823
F8700000     712     712     712       - read/shared       dev:271,11 ino:19761
F87C0000     216     216     216       - read/shared       dev:271,11 ino:23809
F8800000    3936    3936       -    3936 read/write/exec     [ anon ]
F9810000      72      72      72       - read/shared       dev:271,11 ino:23807
F9830000     176     176     176       - read/shared       dev:271,11 ino:23832
F9860000       8       8       8       - read/shared       dev:271,10 ino:1396
F9870000      24      24      24       - read/shared       dev:271,10 ino:1443
F9880000    3432     224     224       - read/shared       dev:271,0 ino:54537
F9BE0000       8       8       8       - read/shared       dev:271,10 ino:1395
F9BF0000      24      24      24       - read/shared       dev:271,10 ino:1464
F9C00000    2048    2048       -    2048 read/write/exec     [ anon ]
FA400000    2048    2048       -    2048 read/write/exec     [ anon ]
FAC00000   36352   36352       -   36352 read/write/exec     [ anon ]
FD820000     104     104     104       - read/shared       dev:271,11 ino:23810
FD85E000      16      16       -      16 read/write          [ anon ]
FD88E000      16      16       -      16 read/write          [ anon ]
FD8C0000       8       8       -       8 read/write          [ anon ]
FD8D0000     136      40      40       - read/shared       dev:271,0 ino:259199
FD900000    1952    1872    1872       - read/shared       dev:271,0 ino:259200
FDAF0000      16      16      16       - read/shared       dev:271,11 ino:23814
FDB00000     112      24      24       - read/shared       dev:271,0 ino:84272
FDB20000      72      72      72       - read/exec         libzip.so
FDB40000      16      16       -      16 read/write/exec   libzip.so
FDB50000     232     232     232       - read/exec         libjava.so
FDB98000      16      16       -      16 read/write/exec   libjava.so
FDBC0000       8       8       -       8 read/write          [ anon ]
FDBF0000       8       8       -       8 read/write          [ anon ]
FDC00000     448     448     448       - read/exec         libnsrwtool.so
FDC7E000      56      56       -      56 read/write/exec   libnsrwtool.so
FDC90000      64      64      64       - read/shared       dev:271,11 ino:23811
FDCB0000     120     120       -     120 read/write/exec     [ anon ]
FDCF0000       8       8       -       8 read/write          [ anon ]
FDD00000    2744    2744    2744       - read/exec         libjvm.so
FDFBC000      88      80       -      80 read/write/exec   libjvm.so
FDFD2000      48      48       -      48 read/write/exec   libjvm.so
FDFF0000       8       8       -       8 read/write/exec     [ anon ]
FE002000       8       8       -       8 read/write          [ anon ]
FE104000       8       8       -       8 read/write          [ anon ]
FE206000       8       8       -       8 read/write          [ anon ]
FE308000       8       8       -       8 read/write          [ anon ]
FE40A000       8       8       -       8 read/write          [ anon ]
FE50C000       8       8       -       8 read/write          [ anon ]
FE60A000       8       8       -       8 read/write          [ anon ]
FE60E000       8       8       -       8 read/write          [ anon ]
FE70C000       8       8       -       8 read/write          [ anon ]
FE710000       8       8       -       8 read/write          [ anon ]
FE80E000       8       8       -       8 read/write          [ anon ]
FE82A000      32      32       -      32 read/write          [ anon ]
FE850000       8       8       -       8 read/write          [ anon ]
FE870000       8       8       -       8 read/write          [ anon ]
FE890000       8       8       -       8 read/write          [ anon ]
FE8B0000       8       8       -       8 read/write          [ anon ]
FE8C0000      40      40      40       - read/write/shared dev:0,2 ino:37726981
FE8D0000       8       8       -       8 read/write/exec     [ anon ]
FE8E0000      16      16      16       - read/exec         libConfigFile.so
FE8F2000       8       8       -       8 read/write/exec   libConfigFile.so
FE900000      16      16      16       - read/exec         libMMapSession.so
FE912000       8       8       -       8 read/write/exec   libMMapSession.so
FE920000       8       8       8       - read/exec         libNSJavaMiscUtil.so
FE930000       8       8       -       8 read/write/exec   libNSJavaMiscUtil.so
FE940000       8       8       -       8 read/write/exec     [ anon ]
FE950000      24      24      24       - read/exec         libNSJvmControl.so
FE964000      16      16       -      16 read/write/exec   libNSJvmControl.so
FE970000      40      40      40       - read/exec         libNSJavaUtil.so
FE988000      16      16       -      16 read/write/exec   libNSJavaUtil.so
FE990000      32      32      32       - read/exec         libNSServletLayer.so
FE9A6000       8       8       -       8 read/write/exec   libNSServletLayer.so
FE9B0000      48      48      48       - read/exec         libNSServletPlugin.so
FE9CA000      16      16       -      16 read/write/exec   libNSServletPlugin.so
FE9E0000      24      24      24       - read/exec         nss_files.so.1
FE9F6000       8       8       -       8 read/write/exec   nss_files.so.1
FEA00000       8       8       8       - read/write/exec/shared   [ anon ]
FEA14000       8       8       -       8 read/write          [ anon ]
FEA20000       8       8       8       - read/write/exec/shared   [ anon ]
FEA30000       8       8       -       8 read/write/exec     [ anon ]
FEA40000       8       8       8       - read/exec         libatomic.so
FEA50000       8       8       8       - read/write/exec   libatomic.so
FEA60000      32      32      32       - read/exec         libaio.so.1
FEA78000       8       8       -       8 read/write/exec   libaio.so.1
FEA80000       8       8       -       8 read/write/exec     [ anon ]
FEA90000      16      16      16       - read/exec         libmp.so.2
FEAA4000       8       8       -       8 read/write/exec   libmp.so.2
FEAB0000      24      24      24       - read/exec         libpthread.so.1
FEAC6000       8       8       -       8 read/write/exec   libpthread.so.1
FEAD0000     208     168     168       - read/exec         libresolv.so.2
FEB14000      24      24       -      24 read/write/exec   libresolv.so.2
FEB20000      16      16      16       - read/exec         libares3.so
FEB32000      16      16       -      16 read/write/exec   libares3.so
FEB40000      24      24      24       - read/exec         libsupport.so
FEB54000       8       8       -       8 read/write/exec   libsupport.so
FEB60000      24      24      24       - read/exec         libnstime.so
FEB74000       8       8       -       8 read/write/exec   libnstime.so
FEB80000       8       8       8       - read/exec         libdirmon4.so
FEB90000       8       8       -       8 read/write/exec   libdirmon4.so
FEBA0000       8       8       8       - read/exec         libnstp.so
FEBB0000       8       8       -       8 read/write/exec   libnstp.so
FEBC0000      32      32      32       - read/exec         libnsfc.so
FEBD6000     136     136     104      32 read/write/exec   libnsfc.so
FEC00000     688     688     688       - read/exec         libc.so.1
FECBC000      32      32       -      32 read/write/exec   libc.so.1
FECD0000       8       8       -       8 read/write/exec     [ anon ]
FECE0000     112     112     112       - read/exec         libthread.so.1
FED0C000       8       8       -       8 read/write/exec   libthread.so.1
FED0E000      48      48       -      48 read/write/exec   libthread.so.1
FED20000       8       8       8       - read/exec         libc_psr.so.1
FED30000     264     216     216       - read/exec         libC.so.5
FED80000      32      32       -      32 read/write/exec   libC.so.5
FED88000      56      16       -      16 read/write/exec   libC.so.5
FEDA0000       8       8       -       8 read/write/exec     [ anon ]
FEDB0000     224      64      64       - read/exec         libm.so.1
FEDF6000       8       8       -       8 read/write/exec   libm.so.1
FEE00000     568     568     568       - read/exec         libnsl.so.1
FEE9E000      40      40       -      40 read/write/exec   libnsl.so.1
FEEA8000      24      16       -      16 read/write/exec   libnsl.so.1
FEEC0000      24      24      24       - read/exec         librt.so.1
FEED6000       8       8       -       8 read/write/exec   librt.so.1
FEEE0000       8       8       -       8 read/write/exec   libdl.so.1
FEEF0000      40      40      40       - read/exec         libsocket.so.1
FEF0A000       8       8       -       8 read/write/exec   libsocket.so.1
FEF10000     176     176     176       - read/exec         libnspr3.so
FEF4A000      16      16       -      16 read/write/exec   libnspr3.so
FEF60000      16      16      16       - read/exec         libplds3.so
FEF72000       8       8       -       8 read/write/exec   libplds3.so
FEF80000      16      16      16       - read/exec         libplc3.so
FEF92000       8       8       -       8 read/write/exec   libplc3.so
FEFA0000      24      24      24       - read/exec         libnsbrk30.so
FEFB4000      16      16       -      16 read/write/exec   libnsbrk30.so
FEFC0000       8       8       -       8 read/write/exec     [ anon ]
FEFD0000      96      96      96       - read/exec         libnscol30.so
FEFF6000      40      40      24      16 read/write/exec   libnscol30.so
FF010000     184     184     184       - read/exec         libnsfmt30.so
FF04C000      32      32       -      32 read/write/exec   libnsfmt30.so
FF060000      80      80      80       - read/exec         libnscnv30.so
FF082000      32      32       -      32 read/write/exec   libnscnv30.so
FF090000     152     152     152       - read/exec         libnsuni30.so
FF0C4000      40      40      16      24 read/write/exec   libnsuni30.so
FF0E0000      96      96      96       - read/exec         libnsres30.so
FF106000      24      24       -      24 read/write/exec   libnsres30.so
FF10C000       8       -       -       - read/write/exec   libnsres30.so
FF120000     168     168     168       - read/exec         libldap30.so
FF158000      16      16       -      16 read/write/exec   libldap30.so
FF160000       8       8       8       - read/exec         libMagnusPostInit.so
FF170000       8       8       -       8 read/write/exec   libMagnusPostInit.so
FF180000      16      16      16       - read/exec         libnsprwrap.so
FF192000       8       8       -       8 read/write/exec   libnsprwrap.so
FF1A0000      40      40      40       - read/exec         liblibdbm.so
FF1B8000       8       8       -       8 read/write/exec   liblibdbm.so
FF1C0000       8       8       -       8 read/write/exec     [ anon ]
FF1D0000      32      32      32       - read/exec         liblibsi18n.so
FF1E6000      72      72      56      16 read/write/exec   liblibsi18n.so
FF200000     936     936     936       - read/exec         libns-httpd40.so
FF2F8000     192     192       8     184 read/write/exec   libns-httpd40.so
FF328000     216      48       -      48 read/write/exec   libns-httpd40.so
FF370000       8       8       8       - read/exec         libgetprop.so
FF380000       8       8       -       8 read/write/exec   libgetprop.so
FF390000       8       8       8       - read/exec         libw.so.1
FF3A0000     176     176     176       - read/exec         ld.so.1
FF3DC000       8       8       -       8 read/write/exec   ld.so.1
FF3DE000       8       8       -       8 read/write/exec   ld.so.1
FFBE2000      56      56       -      56 read/write          [ stack ]
--------  ------  ------  ------  ------
total Kb  110912  102824   28976   73848
0
 
LVL 8

Author Comment

by:RevelationCS
ID: 13727238
Also, to note, based on what I am seeing, swap -l has not changed even though the usage of disk space continues to go up. swap -s is slightly lower than what it had been when I posted above, however, the instance was restarted after some updates were made to some class files so I do not know if this would be a good indicator... I will keep an eye on that today and see if it grows as the disk usage grows...

As it stands from our research, the disk usage is growing at about a rate of 15MB/hr. Also was informed by on of the unix guys that this could have been going on for some time, but they made a change about a month prior to when this issue started to limit /tmp to 2GB, where prior to that all virtual memory on the server would be available for /tmp.....
0
 
LVL 8

Author Comment

by:RevelationCS
ID: 13727258
also, to note..... the server itself has 8GB of physical RAM in the server.... peak usage is about 4.5 GB
0
 
LVL 10

Expert Comment

by:Nukfror
ID: 13727507
RevelationCS,

So df -k shows /tmp not really growing in size ?  If so, then limit on /tmp to 2GB, while a very good idea, doesn't help in this case.  This is what I think you indicated the first time you opened this issue and why I originally said I doubted this had anything to do with /tmp physical utilization of anonymous memory.  If /tmp isn't physical growing into the anonymous memory space, its not the problem.

Which side of the anonymous memory equation is growing: allocated or reserved ?

Reserved mean a program has asked for more memory but hasn't physical touched it yet.  Allocated means the memory has been touched and is pinned in anonymous memory land:  either physical memory or possibly backstored on top physical swap space.

I'm thinking its probably allocated ?

Your comment about swap -l not really growing means that the physical memory currently marked as anonymous memory and actually being used hasn't been exhausted yet so backstoring those pages to physical swap isn't necessary yet.
0
 
LVL 8

Author Comment

by:RevelationCS
ID: 13729055
Nuk,

To clarify... df -k SHOWS /tmp growing in size when it comes to "USED SPACE", however, the allocated size of the partition has remained the same.
0
 
LVL 8

Author Comment

by:RevelationCS
ID: 13729651
running the "/usr/bin/lsof +aL1 /tmp" command, we are a able to view a listing of unused but not unallocated files... It appears that this is common with iPlanet and SunOne from what we can see, and with this, we are noticing the procs associated with typing up disk space having a large size of the files being listed with this command... To relate, we are also unable to locate files of these sizes on the actual /tmp partition itself...
0
 
LVL 10

Expert Comment

by:Nukfror
ID: 13732098
K - sorry. My original impression was /tmp wasn't getting touched and available anonymous space was shrinking from either allocations or reserves.

So is pmap showing these files and associated inode number as residing in /tmp but find can't find them (cuz the link count is 0) ?
0
 
LVL 8

Author Comment

by:RevelationCS
ID: 13740384
after doing further research using the lsof command, it appears that these "phantom" files are related to the logging functions (STDERR/STDOUT) for the JVM on the web instances (files ultimately traced back to /proc/<PID>/fd/1 and /proc/<PID>/fd/2 and both files contain what appears to be contents of logs....

I have addressed the issue with the developers to have them check how they are using the STDERR and STDOUT within their applications. Does anyone know if there is a way to force UNIX/Solaris to clean this up periodically without having to restart the proc? Also, is there a way for us to define in Solaris where STDERR and STDOUT should be going if unable to write to a file?
0
 
LVL 10

Expert Comment

by:Nukfror
ID: 13740634
In your case, STDOUT (fd 1) and STDERR (fd 2) are going to the exact same file.

Cleaning up depends on how the application is written.  You could try simply this:

cp /dev/null /tmp/whatever-the-filename-is

Wait for about a minute or two and see if the file is now 0 bytes+<whatever was written since you ran the cp command> .  If the file comes back to its original size plus some then this won't work.  This sometimes works.  You can play this came with syslog all day.  Doesn't work everytime.

To my knowledge, if you direct STDOUT and STDERR to anything the process can't write to the app will error out immediately.  I've always wished Solaris had a user defined buffer ring log capability like the console buffer.

One idea (but very ugly) is to redirect STDOUT and STDERR to a named pipe.  Have an application you write on the other side of the pipe that will roll the log file it writes when you hit it with say a SIGHUP.  Pretty sure there are examples of doing this with Perl in multiple places on the net.
0
 
LVL 8

Author Comment

by:RevelationCS
ID: 13754033
one thought we are having is with STDERR and STDOUT trying to log to the console... is there a possibility that this can be defined by developers? I know the Log4J logging and a few other applications can be set to log to console, but am unsure if the system outs can do this or not...
0
 
LVL 8

Author Comment

by:RevelationCS
ID: 13819706
still working on testing a fix to the start scripts to resolve this issue... I will keep you posted as to the resolution...
0
 
LVL 10

Expert Comment

by:Nukfror
ID: 13821014
Another idea is to take the logger program and stick it on the end of the named pipe.  See the man page for more information.  Use the -f switch to specify the named pipe as contents to log.  You should redirect both STDOUT and STDERR to the named pipe.
0
 
LVL 8

Author Comment

by:RevelationCS
ID: 13869928
over the weekend we installed the latest service patch for both of the webservers that we were noticing the problems with and it appears to have resolved the issue at this point in time... usage of /tmp has maintained itself at 4% over the last few days... I will continue to monitor for the next week or so and post back if this resolved the issue...
0
 
LVL 8

Author Comment

by:RevelationCS
ID: 13959908
latest results of the lsof command:

ns-httpd   6377 efdadmin    1w  VREG    0,2 215412914     0 49274930 /tmp (swap)
ns-httpd   6377 efdadmin    2w  VREG    0,2 215412914     0 49274930 /tmp (swap)


It appears that we still have one offending process under the webservers, however, things have greatly improved since the issue was noticed. I am planning on returning this information for the remaining process to the developers to have them look at their code for potential causes of the issue...
0
 
LVL 8

Author Comment

by:RevelationCS
ID: 13996193
Still no issues as of yet and it appears that the /tmp directory has stayed under control... I will be posting a request to have the questioned PAQ'd...
0
 

Accepted Solution

by:
OzzMod earned 0 total points
ID: 14076768
Closed, 500 points refunded.
OzzMod
Community Support Moderator (Graveyard shift)
0

Featured Post

[Webinar] Cloud and Mobile-First Strategy

Maybe you’ve fully adopted the cloud since the beginning. Or maybe you started with on-prem resources but are pursuing a “cloud and mobile first” strategy. Getting to that end state has its challenges. Discover how to build out a 100% cloud and mobile IT strategy in this webinar.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Installing FreeBSD… FreeBSD is a darling of an operating system. The stability and usability make it a clear choice for servers and desktops (for the cunning). Savvy?  The Ports collection makes available every popular FOSS application and packag…
Java performance on Solaris - Managing CPUs There are various resource controls in operating system which directly/indirectly influence the performance of application. one of the most important resource controls is "CPU".   In a multithreaded…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.
Suggested Courses

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question