asked on

/tmp issues - pfiles shows large file usage

We have been having an on and off issue with the /tmp directory filling up due to "phantom files." After doing some research on the last occurance, I see the following symptoms:

# swap -l
swapfile dev swaplo blocks free
/dev/vx/dsk/swapvol 271,5 16 16780208 16779952

# swap -s
total: 2240520k bytes allocated + 600536k reserved = 2841056k used, 12066136k available

# df -k /tmp
Filesystem kbytes used avail capacity Mounted on
swap 2097152 604640 1492512 29% /tmp

pfiles command shows the following for one of the known processes contributing to the issue (edited to remove sensitive information):

# pfiles 21004
21004: ns-httpd -d /opt/netscape/server4/https-instancename/confi
Current rlimit: 1024 file descriptors
0: S_IFIFO mode:0000 dev:314,0 ino:1897983 uid:0 gid:0 size:0
O_RDWR
1: S_IFREG mode:0644 dev:0,2 ino:37632521 uid:0 gid:0 size:364774539
O_WRONLY|O_LARGEFILE
2: S_IFREG mode:0644 dev:0,2 ino:37632521 uid:0 gid:0 size:364774539
O_WRONLY|O_LARGEFILE
3: S_IFCHR mode:0666 dev:271,0 ino:421213 uid:0 gid:3 rdev:13,12
O_RDWR
4: S_IFREG mode:0644 dev:271,10 ino:183 uid:0 gid:0 size:206
O_RDONLY
5: S_IFIFO mode:0000 dev:314,0 ino:1897985 uid:18048 gid:2030 size:0
O_RDWR
6: S_IFCHR mode:0666 dev:271,0 ino:421213 uid:0 gid:3 rdev:13,12
O_RDWR
7: S_IFREG mode:0644 dev:271,10 ino:2883 uid:18048 gid:2030 size:988828
O_RDWR|O_APPEND|O_LARGEFILE
8: S_IFIFO mode:0000 dev:314,0 ino:1897986 uid:18048 gid:2030 size:0
O_RDWR
9: S_IFREG mode:0644 dev:271,10 ino:501 uid:0 gid:0 size:229376
O_RDONLY FD_CLOEXEC
10: S_IFREG mode:0644 dev:271,10 ino:2884 uid:18048 gid:2030 size:4379676
O_RDWR|O_APPEND|O_LARGEFILE FD_CLOEXEC
11: S_IFREG mode:0644 dev:0,2 ino:26330833 uid:18048 gid:2030 size:39964
O_RDWR|O_LARGEFILE
12: S_IFCHR mode:0666 dev:271,0 ino:421213 uid:0 gid:3 rdev:13,12
O_RDONLY
13: S_IFDIR mode:0511 dev:312,0 ino:12355 uid:18048 gid:2030 size:736
O_RDONLY
14: S_IFCHR mode:0666 dev:271,0 ino:421212 uid:0 gid:3 rdev:13,2
O_RDONLY
15: S_IFCHR mode:0666 dev:271,0 ino:421212 uid:0 gid:3 rdev:13,2
O_RDONLY
16: S_IFREG mode:0644 dev:271,11 ino:42819 uid:18048 gid:2030 size:4821374
O_WRONLY|O_LARGEFILE
17: S_IFSOCK mode:0666 dev:313,0 ino:52338 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK
sockname: AF_INET x.x.x.x port: xxxx
peername: AF_INET x.x.x.x port: xxxx
20: S_IFSOCK mode:0666 dev:313,0 ino:52342 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK
sockname: AF_INET x.x.x.x port: xxxx
peername: AF_INET x.x.x.x port: xxxx
21: S_IFSOCK mode:0666 dev:313,0 ino:47220 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK
sockname: AF_INET x.x.x.x port: xxxx
peername: AF_INET x.x.x.x port: xxxx
23: S_IFSOCK mode:0666 dev:313,0 ino:36864 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK
sockname: AF_INET x.x.x.x port: xxxx
peername: AF_INET x.x.x.x port: xxxx
260: S_IFSOCK mode:0666 dev:313,0 ino:4349 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK FD_CLOEXEC
sockname: AF_INET x.x.x.x port: xxxx

I have been monitoring throught the course of the day and have noticed the listed items in pfiles that have large "sizes" on them continue to grow as I monitor them. I believe these are where the issues are at. Also, to note, when the /tmp hits 100%, memory usage is normal (about half of total available memory). Du also shows no large files listed in any directory under /tmp. (please see original question https://www.experts-exchange.com/questions/21350012/tmp-directory-issues.html). Is there a way for us to tell what the items listed under pfiles reflect and how to identify the "problem" file? We need to determine the cause of the problem, however, I believe I am missing a few items that would help us to do so...

thanks

neteducation

# pfiles 21004
21004: ns-httpd -d /opt/netscape/server4/https-instancename/confi
Current rlimit: 1024 file descriptors

Which Version of Netscape 4 Webserver ar you using? I remeber of some having a problem opening up temporary files to talk to CGI-Programs, and not cleaning them if the cgi didnt respond in a timely way.

(And for personal opinion... go to the new 6.1 Webserver from sun... much faster and rock-solid)

RevelationCS

ASKER

using 4.1 with the last service pack I believe... this is only one of the webservers running on the servers with this issue... the others are running 6.1 (and I agree and have been bugging the development managers to work on getting the apps upgraded)... does it affect anything other than CGI? all the apps running in the environment are running java/jsps...

garypen

If this was Solaris 10, your question would be very easy to answer since pfiles has been upgraded to include pathnames in it's output where possible:

e.g.

bash-3.00# ps -p 23397
PID TTY TIME CMD
23397 ? 1:20 httpd
bash-3.00# pfiles 23397
23397: /usr/apache2/bin/httpd -k start
Current rlimit: 65536 file descriptors
0: S_IFCHR mode:0666 dev:270,0 ino:6815752 uid:0 gid:3 rdev:13,2
O_RDONLY
/devices/pseudo/mm@0:null
1: S_IFCHR mode:0666 dev:270,0 ino:6815752 uid:0 gid:3 rdev:13,2
O_WRONLY|O_CREAT|O_TRUNC
/devices/pseudo/mm@0:null
2: S_IFREG mode:0644 dev:102,0 ino:114297 uid:0 gid:0 size:322900
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
/var/apache2/logs/error_log
3: S_IFSOCK mode:0666 dev:276,0 ino:26677 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(49152)
sockname: AF_INET6 :: port: 80
4: S_IFDOOR mode:0444 dev:279,0 ino:57 uid:0 gid:0 size:0
O_RDONLY|O_LARGEFILE FD_CLOEXEC door to nscd[125]
/var/run/name_service_door
5: S_IFIFO mode:0000 dev:277,0 ino:73320 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK
6: S_IFIFO mode:0000 dev:277,0 ino:73320 uid:0 gid:0 size:0
O_RDWR
7: S_IFREG mode:0644 dev:102,0 ino:114297 uid:0 gid:0 size:322900
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
/var/apache2/logs/error_log
8: S_IFREG mode:0644 dev:102,0 ino:114298 uid:0 gid:0 size:413909
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
/var/apache2/logs/access_log
9: S_IFREG mode:0600 dev:274,2 ino:446347586 uid:0 gid:0 size:0
O_RDWR|O_CREAT|O_EXCL
10: S_IFREG mode:0644 dev:274,3 ino:451283270 uid:0 gid:0 size:0
O_WRONLY|O_CREAT|O_EXCL
bash-3.00#

On older versions of Solaris you are stuck.

You may be able to work out what is happening using truss. You could try to truss the script you use to start up your webserver using something like:

truss -topen -f -o /tmp/trace <command>

This will trace all calls to open, in all sub-processes of command and write the output to /tmp/trace. After starting your server, stop it and then look at the /tmp/trace file to see which files are opened in /tmp. This may give you a clue about which files are being used to hold data in /tmp.

However, I'm not sure that this will help you much. It is probable that your webserver is just not behaving well and knowing that won't help you apart from providing more ammunition to your argument to upgrade the webserver to a newer (presumably less buggy) version.

I think you are barking up the wrong tree. The output above doesn't really have any large files listed apart from fd 1 and 2. These are (probably) the log file output and are clearly the same file (same dev/inodes). I don't expect you are storing your log file output in /tmp and so I don't really think that there is any evidence from the above to show that you have problems with large unidentified files living in /tmp. It's more likely (since you are using Java) that you are simply using large amounts of virtual memory from your Java VMs. Try running:

pmap `pgrep java`

Look at the totals and see if this is your problem. Java is notorious for eating up virtual memory and it's much more likely this is the cause of your problem.

Nukfror

Actually not totally stuck. The dev field in the <Sol10 pfiles output does give valuable information. A simple example:

bash-2.03$ ps -ef|grep view
soneill 5024 4987 0 22:21:02 pts/1 0:00 view /etc/hosts
soneill 5054 5029 0 22:21:42 pts/2 0:00 grep view
bash-2.03$ pfiles 5024
5024: view /etc/hosts
Current rlimit: 2048 file descriptors
0: S_IFCHR mode:0620 dev:85,1 ino:83288 uid:520 gid:7 rdev:24,1
O_RDWR|O_LARGEFILE
1: S_IFCHR mode:0620 dev:85,1 ino:83288 uid:520 gid:7 rdev:24,1
O_RDWR|O_LARGEFILE
2: S_IFCHR mode:0620 dev:85,1 ino:83288 uid:520 gid:7 rdev:24,1
O_RDWR|O_LARGEFILE
3: S_IFCHR mode:0666 dev:85,1 ino:83119 uid:0 gid:3 rdev:13,12
O_RDWR
4: S_IFREG mode:0600 dev:85,2 ino:60426 uid:520 gid:520 size:24576
O_RDWR

Take a look at FD 4. The dev field = First number is the major device # and second is the minor. So 85,2 is:

cd /devices/pseudo
ls -la | less
.....<snip>
brw-r----- 1 root sys 85, 2 Feb 4 2001 md@0:0,2,blk
crw-r----- 1 root sys 85, 2 Feb 4 2001 md@0:0,2,raw

So this is obviously a SVM mount point.

bash-2.03# cd /dev/md/dsk
bash-2.03# ls -la | grep "md@0:0,2"
lrwxrwxrwx 1 root other 36 Feb 4 2001 d2 -> ../../../devices/pseudo/md@0:0,2,blk
<snip>

d2 is /var mount point:

bash-2.03# df -k
Filesystem kbytes used avail capacity Mounted on
<snip>
/dev/md/dsk/d2 123231 8317 102591 8% /var

The pfiles ino field is the inode number. That inode is related to file:

bash-2.03# ncheck -i 60426 /dev/md/dsk/d2
/dev/md/dsk/d2:
60426 /tmp/ExJjaa0j

The /tmp is a *RELATIVE* reference to the /var mount point.

(You could have also used find to look for the inode number. Find obeys mount points boundries. ncheck goes down to the physical partition. In most cases find is probably a better command to use - but ncheck is rather obscure so kinda kewl ;) )

Notice the size of the file is 24576 as reported by pfiles. ls -la shows the same mode bits, size, owner, and group (520 is soneill on this box) so this is refering to the same file.

cd /var/tmp
bash-2.03# ls -lai ExJjaa0j
60426 -rw------- 1 soneill soneill 24576 Apr 6 22:21 ExJjaa0j

So you aren't totally stuck if you just do a little digging on each file descriptor.

Nukfror

And a comment on your original command output.

# swap -l
swapfile dev swaplo blocks free
/dev/vx/dsk/swapvol 271,5 16 16780208 16779952

# swap -s
total: 2240520k bytes allocated + 600536k reserved = 2841056k used, 12066136k available

You have 4 GB of memory and 8 GB of swap. PHYSICAL swap (in this output) you using barely 131KBytes. VIRTUAL swap (e.g. anonymous memory) you've allocated over 2GB.

Run prstat and look for applications with large memory utilizations then run pmap against each of these application over time.

bash-2.03# pmap -x 5024
5024: view /etc/hosts
Address Kbytes Resident Shared Private Permissions Mapped File
00010000 188 8 8 - read/exec view
0004E000 40 20 - 20 read/write/exec view
00058000 432 104 - 104 read/write/exec [ heap ]
EF380000 688 688 688 - read/exec libc.so.1
EF43C000 28 28 - 28 read/write/exec libc.so.1
EF443000 4 4 - 4 read/write/exec libc.so.1
EF480000 24 24 24 - read/exec libgen.so.1
EF496000 4 4 - 4 read/write/exec libgen.so.1
EF500000 8 8 8 - read/exec libcrypt_i.so.1
EF512000 4 4 - 4 read/write/exec libcrypt_i.so.1
EF580000 164 124 124 - read/exec libcurses.so.1
EF5BA000 32 32 12 20 read/write/exec libcurses.so.1
EF5C2000 8 8 - 8 read/write/exec libcurses.so.1
EF600000 8 8 8 - read/exec libmapmalloc.so.1
EF612000 4 4 - 4 read/write/exec libmapmalloc.so.1
EF680000 4 4 4 - read/exec libdl.so.1
EF700000 4 4 4 - read/shared dev:85,2 ino:60443
EF780000 156 156 156 - read/exec ld.so.1
EF7B8000 8 8 - 8 read/write/exec ld.so.1
EF7C0000 4 4 - 4 read/write [ anon ]
EF7D0000 4 4 - 4 read/write/exec [ anon ]
EF7E0000 4 4 - 4 read/write/exec [ anon ]
EF7F0000 4 4 - 4 read/write/exec [ anon ]
EFFFB000 20 20 - 20 read/write [ stack ]
-------- ------ ------ ------ ------
total Kb 1844 1276 1036 240

You interested in the segments parts the eat anonymous memory like stack/heap space. Look at the Private column number and look for growing numbers over time. Notice some segments are flagged as [ anon ], [ heap ], and [ stack ] showing you the candidates to look for.

neteducation

i had the problem only with cgi's, but basically the concept of jsp is pretty much the same as with cgi's... just that you use java as the program.... so I could very well imagine it is the same problem. btw: my "solution" for that problem was to restart the daemon once a night, and to delete the older temporary files in /tmp. (plus in my case kill the cgi-processes that were hanging)

garypen

There is a flaw with Nukfor's approach. AFAIK 'ncheck' only supports UFS filesystems. Since the question seems to be about a tmpfs filesystem, namely /tmp, this approach won't help.

Anyway, it looks like Nukfor agrees with me that the problem isn't caused by unidentified files but by processes consuming large amounts of physical memory which can be best identified using pmap.

If the problem is caused by your JSP processing, then you can try to control the problem by judicious tuning of your webserver/application server to restrict memory usage.

Your webserver is ancient and so it's likely that it fork/execs a separate JVM for each JSP page hit and this is a very inefficient and memory greedy approach likely to cause problems such as you see.

You should upgrade your application server environment to something more modern which can cache JSP pages and also processes page hits in memory and avoids the huge fork/exec overhead I mentioned earlier.

Such work is of course no longer Solaris tuning or a problem with Solaris and you may get better answers to further questions in the "Web Dev" section.

Nukfror

Garypen, true, thanks for the clarification [in the back of my head I knew this - simply didn't type it] - but notice my comment about "find" so this approach still works simply replace ncheck with find.

In most cases, a program reading in large files doesn't eat up large amounts of swap space - it eats up FS Page cache memory. There are obviously exceptions to every case. In most costs, lots of anonymous memory could get eaten when a program uses mmap() to pull in a file into its address space - not when using read() or write().

I agree with garypen that this isn't necessarily an issue with something "opening" large files in anonymous memory. This has something to do with some process eating up anonymous memory with stack/heap. Assuming Java is the issue, just run:

ps -alxww | grep java

And look for how Java with switch -ms and -mx (see man page for description). Most folks I've seen specify this in MBytes. A pmap -x against your java processes would be interesting as well.

RevelationCS

ASKER

for one of the offending processes....

# pmap -x 14224
14224: ns-httpd -d /opt/netscape/server4/<instance name>/confi
Address Kbytes Resident Shared Private Permissions Mapped File
00010000 584 584 584 - read/exec ns-httpd
000B0000 128 128 80 48 read/write/exec ns-httpd
000D0000 25648 25432 - 25432 read/write/exec [ heap ]
F57B4000 8 8 - 8 read/write [ anon ]
F57F4000 8 8 - 8 read/write [ anon ]
F5804000 8 8 - 8 read/write [ anon ]
F5850000 24 24 24 - read dev:271,11 ino:29297
F5860000 88 88 88 - read dev:271,11 ino:39380
F5880000 8 8 8 - read dev:271,11 ino:18484
F5890000 40 40 40 - read dev:271,11 ino:18339
F58A0000 40 40 40 - read dev:271,11 ino:18341
F58B0000 32 32 32 - read dev:271,11 ino:39323
F58C0000 8 8 8 - read dev:271,11 ino:18797
F58D0000 8 8 8 - read dev:271,11 ino:19075
F58E0000 8 8 8 - read dev:271,11 ino:29265
F58F0000 8 8 8 - read dev:271,11 ino:37391
F5904000 8 8 - 8 read/write [ anon ]
F5910000 8 8 8 - read dev:271,11 ino:18823
F5920000 8 8 8 - read dev:271,11 ino:40694
F5932000 8 8 - 8 read/write [ anon ]
F5954000 8 8 - 8 read/write [ anon ]
F5960000 8 8 8 - read dev:271,11 ino:18745
F5974000 8 8 - 8 read/write [ anon ]
F5980000 8 8 8 - read dev:271,11 ino:18749
F5990000 8 8 8 - read dev:271,11 ino:37162
F59A0000 8 8 8 - read dev:271,11 ino:37053
F59B4000 8 8 - 8 read/write [ anon ]
F59C0000 16 16 16 - read dev:271,11 ino:30956
F59D0000 8 8 8 - read dev:271,11 ino:40825
F59E0000 8 8 8 - read dev:271,11 ino:40828
F59F0000 8 8 8 - read dev:271,11 ino:18251
F5A00000 8 8 8 - read dev:271,11 ino:38274
F5A10000 8 8 8 - read dev:271,11 ino:38247
F5A20000 8 8 8 - read dev:271,11 ino:38273
F5A30000 8 8 8 - read dev:271,11 ino:31456
F5A44000 8 8 - 8 read/write [ anon ]
F5A50000 8 8 8 - read dev:271,11 ino:40085
F5A70000 8 8 8 - read dev:271,11 ino:18812
F5A90000 16 16 16 - read dev:271,11 ino:18821
F5AA0000 8 8 8 - read dev:271,11 ino:38640
F5AB0000 8 8 8 - read dev:271,11 ino:37368
F5AC2000 8 8 - 8 read/write [ anon ]
F5AD0000 8 8 8 - read dev:271,11 ino:39937
F5AE0000 40 40 40 - read dev:271,11 ino:33011
F5AF0000 40 40 40 - read dev:271,11 ino:18574
F5B00000 8 8 8 - read dev:271,11 ino:42331
F5B10000 8 8 8 - read dev:271,11 ino:42335
F5B20000 8 8 8 - read dev:271,11 ino:42437
F5B30000 8 8 8 - read dev:271,11 ino:29074
F5B40000 8 8 8 - read dev:271,11 ino:29115
F5B50000 8 8 8 - read dev:271,11 ino:19329
F5B60000 8 8 8 - read dev:271,11 ino:41064
F5B70000 8 8 8 - read dev:271,11 ino:18797
F5B80000 8 8 8 - read dev:271,11 ino:40571
F5B90000 8 8 8 - read dev:271,11 ino:18255
F5BA0000 8 8 8 - read dev:271,11 ino:34271
F5BB0000 8 8 8 - read dev:271,11 ino:38332
F5BC0000 32 32 32 - read dev:271,11 ino:33005
F5BD0000 8 8 8 - read dev:271,11 ino:40261
F5BE0000 8 8 8 - read dev:271,11 ino:34050
F5BF0000 8 8 8 - read dev:271,11 ino:34073
F5C00000 8 8 8 - read dev:271,11 ino:40692
F5C10000 8 8 8 - read dev:271,11 ino:40690
F5C20000 8 8 8 - read dev:271,11 ino:18679
F5C30000 8 8 8 - read dev:271,11 ino:33010
F5C40000 192 192 192 - read dev:271,11 ino:38594
F5C80000 8 8 8 - read dev:271,11 ino:28642
F5C90000 8 8 8 - read dev:271,11 ino:28638
F5CA0000 8 8 8 - read dev:271,11 ino:28644
F5CB0000 8 8 8 - read dev:271,11 ino:34306
F5CC0000 24 24 24 - read dev:271,11 ino:18374
F5CD0000 8 8 8 - read dev:271,11 ino:35778
F5CE0000 16 16 16 - read dev:271,11 ino:30806
F5CF0000 8 8 8 - read dev:271,11 ino:19126
F5D00000 8 8 8 - read dev:271,11 ino:31213
F5D10000 16 16 16 - read dev:271,11 ino:38015
F5D3A000 32 32 - 32 read/write [ anon ]
F5D50000 8 8 8 - read dev:271,11 ino:38023
F5D7A000 32 32 - 32 read/write [ anon ]
F5DAA000 32 32 - 32 read/write [ anon ]
F5DC0000 8 8 8 - read dev:271,11 ino:39090
F5DD0000 8 8 8 - read dev:271,11 ino:40598
F5DE0000 8 8 8 - read dev:271,11 ino:29296
F5DF0000 8 8 8 - read dev:271,11 ino:34925
F5E00000 8 8 8 - read dev:271,11 ino:34961
F5E10000 16 16 16 - read dev:271,11 ino:40687
F5E24000 8 8 - 8 read/write [ anon ]
F5E30000 8 8 8 - read dev:271,11 ino:27593
F5E40000 32 32 32 - read dev:271,11 ino:38601
F5E50000 144 144 144 - read dev:271,11 ino:18924
F5E80000 24 24 24 - read dev:271,11 ino:39115
F5E90000 16 16 16 - read dev:271,11 ino:38602
F5EA0000 16 16 16 - read dev:271,11 ino:38644
F5EB0000 8 8 8 - read dev:271,11 ino:40648
F5EC0000 8 8 8 - read dev:271,11 ino:41869
F5ED0000 8 8 8 - read dev:271,11 ino:34920
F5EE0000 16 16 16 - read dev:271,11 ino:39113
F5EF0000 8 8 8 - read dev:271,11 ino:35393
F5F00000 8 8 8 - read dev:271,11 ino:18249
F5F10000 8 8 8 - read dev:271,11 ino:31577
F5F20000 8 8 8 - read dev:271,11 ino:19110
F5F30000 96 96 96 - read dev:271,11 ino:28711
F5F50000 16 16 16 - read dev:271,11 ino:19303
F5F60000 32 32 32 - read dev:271,11 ino:30855
F5F70000 8 8 8 - read dev:271,11 ino:18952
F5F80000 16 16 16 - read dev:271,11 ino:26831
F5F90000 8 8 8 - read dev:271,11 ino:40865
F5FA0000 8 8 8 - read dev:271,11 ino:26828
F5FB0000 8 8 8 - read dev:271,11 ino:38430
F5FC0000 8 8 8 - read dev:271,11 ino:18918
F5FD0000 8 8 8 - read dev:271,11 ino:31611
F5FE0000 8 8 8 - read dev:271,11 ino:19076
F5FF0000 8 8 8 - read dev:271,11 ino:31581
F6000000 8 8 8 - read dev:271,11 ino:31609
F6010000 8 8 8 - read dev:271,11 ino:28674
F6020000 176 176 176 - read dev:271,11 ino:31315
F6050000 8 8 8 - read dev:271,11 ino:28590
F6060000 8 8 8 - read dev:271,11 ino:17860
F6070000 8 8 8 - read dev:271,11 ino:31202
F6080000 8 8 8 - read dev:271,11 ino:18221
F6090000 8 8 8 - read dev:271,11 ino:18743
F60BA000 32 32 - 32 read/write [ anon ]
F60D0000 16 16 16 - read dev:271,11 ino:39324
F60FA000 32 32 - 32 read/write [ anon ]
F6110000 8 8 8 - read dev:271,11 ino:18330
F613A000 32 32 - 32 read/write [ anon ]
F6150000 8 8 8 - read dev:271,11 ino:18713
F6160000 8 8 8 - read dev:271,11 ino:30932
F6170000 8 8 8 - read dev:271,11 ino:39094
F6180000 8 8 8 - read dev:271,11 ino:39321
F6190000 16 16 16 - read dev:271,11 ino:40404
F61A0000 8 8 8 - read dev:271,11 ino:39117
F61B0000 8 8 8 - read dev:271,11 ino:40265
F61C0000 24 24 24 - read dev:271,11 ino:42246
F61D0000 40 40 40 - read dev:271,11 ino:42244
F61E0000 8 8 8 - read dev:271,11 ino:37392
F61F0000 8 8 8 - read dev:271,11 ino:18956
F6200000 8 8 8 - read dev:271,11 ino:18950
F6210000 8 8 8 - read dev:271,11 ino:18920
F6220000 8 8 8 - read dev:271,11 ino:18568
F6230000 32 32 32 - read dev:271,11 ino:36869
F6240000 8 8 8 - read dev:271,11 ino:36858
F6250000 80 80 80 - read dev:271,11 ino:36847
F6270000 16 16 16 - read dev:271,11 ino:36776
F6280000 8 8 8 - read dev:271,11 ino:30901
F6290000 8 8 8 - read dev:271,11 ino:18638
F62A0000 8 8 8 - read dev:271,11 ino:18507
F62B0000 8 8 8 - read dev:271,11 ino:33009
F62DA000 32 32 - 32 read/write [ anon ]
F62F0000 8 8 8 - read dev:271,11 ino:30836
F631A000 32 32 - 32 read/write [ anon ]
F6330000 8 8 8 - read dev:271,11 ino:39294
F6340000 8 8 8 - read dev:271,11 ino:42249
F6350000 8 8 8 - read dev:271,11 ino:18989
F6360000 8 8 8 - read dev:271,11 ino:18970
F6370000 16 16 16 - read dev:271,11 ino:42997
F6380000 8 8 8 - read dev:271,11 ino:30957
F6390000 8 8 8 - read dev:271,11 ino:42714
F63A0000 8 8 8 - read dev:271,11 ino:28586
F63B0000 8 8 8 - read dev:271,11 ino:28592
F63C0000 8 8 8 - read dev:271,11 ino:28622
F63D0000 16 16 16 - read dev:271,11 ino:38433
F63E4000 8 8 - 8 read/write [ anon ]
F63F0000 8 8 8 - read dev:271,11 ino:38472
F6400000 8 8 8 - read dev:271,11 ino:28716
F6410000 8 8 8 - read dev:271,11 ino:39322
F643A000 32 32 - 32 read/write [ anon ]
F6450000 16 16 16 - read dev:271,11 ino:40599
F6460000 8 8 8 - read dev:271,11 ino:40597
F648A000 32 32 - 32 read/write [ anon ]
F64A0000 8 8 8 - read dev:271,11 ino:33194
F64B0000 8 8 8 - read dev:271,11 ino:33175
F64C0000 24 24 24 - read dev:271,11 ino:42799
F64D0000 8 8 8 - read dev:271,11 ino:28685
F64FA000 32 32 - 32 read/write [ anon ]
F6510000 8 8 8 - read dev:271,11 ino:38485
F653A000 32 32 - 32 read/write [ anon ]
F6550000 8 8 8 - read dev:271,11 ino:19741
F6560000 16 16 16 - read dev:271,11 ino:33215
F6570000 16 16 16 - read dev:271,11 ino:35780
F6580000 8 8 8 - read dev:271,11 ino:18365
F6590000 40 40 40 - read dev:271,11 ino:41868
F65A0000 8 8 8 - read dev:271,11 ino:42067
F65B0000 8 8 8 - read dev:271,11 ino:41864
F65C0000 8 8 8 - read dev:271,11 ino:27622
F65D0000 8 8 8 - read dev:271,11 ino:38434
F65E0000 24 24 24 - read dev:271,11 ino:38429
F660A000 32 32 - 32 read/write [ anon ]
F6620000 16 16 16 - read dev:271,11 ino:33217
F6630000 8 8 8 - read dev:271,11 ino:18994
F6640000 8 8 8 - read dev:271,11 ino:38406
F6650000 8 8 8 - read dev:271,11 ino:38610
F6660000 8 8 8 - read dev:271,11 ino:31281
F6670000 8 8 8 - read dev:271,11 ino:31277
F6680000 8 8 8 - read dev:271,11 ino:31283
F6690000 8 8 8 - read dev:271,11 ino:31320
F66A0000 8 8 8 - read dev:271,11 ino:41863
F66B0000 8 8 8 - read dev:271,11 ino:19137
F66C0000 16 16 16 - read dev:271,11 ino:19135
F66D0000 8 8 8 - read dev:271,11 ino:28726
F66E0000 8 8 8 - read dev:271,11 ino:31874
F66F0000 8 8 8 - read dev:271,11 ino:32017
F6700000 32 32 32 - read dev:271,11 ino:18360
F672A000 32 32 - 32 read/write [ anon ]
F6740000 8 8 8 - read dev:271,11 ino:18337
F6750000 16 16 16 - read dev:271,11 ino:18575
F6760000 8 8 8 - read dev:271,11 ino:40111
F6770000 8 8 8 - read dev:271,11 ino:19304
F6780000 16 16 16 - read dev:271,11 ino:32322
F6790000 8 8 8 - read dev:271,11 ino:32321
F67A0000 8 8 8 - read dev:271,11 ino:32325
F67B0000 16 16 16 - read dev:271,11 ino:32242
F67C0000 16 16 16 - read dev:271,11 ino:38645
F67D0000 56 56 56 - read dev:271,11 ino:31324
F67F0000 8 8 8 - read dev:271,11 ino:38598
F6800000 8 8 8 - read dev:271,11 ino:38569
F6810000 16 16 16 - read dev:271,11 ino:38603
F6820000 8 8 8 - read dev:271,11 ino:33012
F6830000 8 8 8 - read dev:271,11 ino:18600
F6840000 8 8 8 - read dev:271,11 ino:30925
F6850000 8 8 8 - read dev:271,11 ino:18613
F6860000 8 8 8 - read dev:271,11 ino:38431
F6870000 16 16 16 - read dev:271,11 ino:18634
F689A000 32 32 - 32 read/write [ anon ]
F68B0000 8 8 8 - read dev:271,11 ino:38432
F68DA000 32 32 - 32 read/write [ anon ]
F68F0000 32 32 32 - read dev:271,11 ino:18595
F6900000 8 8 8 - read dev:271,11 ino:31851
F6910000 8 8 8 - read dev:271,11 ino:42713
F693A000 32 32 - 32 read/write [ anon ]
F6950000 8 8 8 - read dev:271,11 ino:18384
F6960000 8 8 8 - read dev:271,11 ino:18386
F6970000 16 16 16 - read/exec nss_dns.so.1
F6984000 8 8 - 8 read/write/exec nss_dns.so.1
F6990000 40 40 40 - read/exec libnet.so
F69A8000 8 8 - 8 read/write/exec libnet.so
F69B0000 8 8 8 - read dev:271,11 ino:18383
F69C0000 24 24 24 - read dev:271,11 ino:28715
F69EE000 16 16 - 16 read/write [ anon ]
F6A00000 8 8 8 - read dev:271,11 ino:32400
F6A2A000 32 32 - 32 read/write [ anon ]
F6A5A000 32 32 - 32 read/write [ anon ]
F6A70000 8 8 8 - read dev:271,11 ino:18376
F6A80000 8 8 8 - read dev:271,11 ino:18573
F6AAA000 32 32 - 32 read/write [ anon ]
F6ADA000 32 32 - 32 read/write [ anon ]
F6B0A000 32 32 - 32 read/write [ anon ]
F6B3A000 32 32 - 32 read/write [ anon ]
F6B6A000 32 32 - 32 read/write [ anon ]
F6B9A000 32 32 - 32 read/write [ anon ]
F6BCA000 32 32 - 32 read/write [ anon ]
F6BF8000 40 40 - 40 read/write [ anon ]
F6C2A000 32 32 - 32 read/write [ anon ]
F6C5A000 32 32 - 32 read/write [ anon ]
F6C8A000 32 32 - 32 read/write [ anon ]
F6CBA000 32 32 - 32 read/write [ anon ]
F6CEA000 32 32 - 32 read/write [ anon ]
F6D1A000 32 32 - 32 read/write [ anon ]
F6D4A000 32 32 - 32 read/write [ anon ]
F6D7A000 32 32 - 32 read/write [ anon ]
F6DAA000 32 32 - 32 read/write [ anon ]
F6DDA000 32 32 - 32 read/write [ anon ]
F6E0A000 32 32 - 32 read/write [ anon ]
F6E3A000 32 32 - 32 read/write [ anon ]
F6E68000 40 40 - 40 read/write [ anon ]
F6E9A000 32 32 - 32 read/write [ anon ]
F6ECA000 32 32 - 32 read/write [ anon ]
F6EFA000 32 32 - 32 read/write [ anon ]
F6F2A000 32 32 - 32 read/write [ anon ]
F6F5A000 32 32 - 32 read/write [ anon ]
F6F8A000 32 32 - 32 read/write [ anon ]
F6FBA000 32 32 - 32 read/write [ anon ]
F6FEA000 32 32 - 32 read/write [ anon ]
F701A000 32 32 - 32 read/write [ anon ]
F704A000 32 32 - 32 read/write [ anon ]
F707A000 32 32 - 32 read/write [ anon ]
F70AA000 32 32 - 32 read/write [ anon ]
F70DA000 32 32 - 32 read/write [ anon ]
F710A000 32 32 - 32 read/write [ anon ]
F713A000 32 32 - 32 read/write [ anon ]
F716A000 32 32 - 32 read/write [ anon ]
F719A000 32 32 - 32 read/write [ anon ]
F71C6000 48 48 - 48 read/write [ anon ]
F71FA000 32 32 - 32 read/write [ anon ]
F722A000 32 32 - 32 read/write [ anon ]
F725A000 32 32 - 32 read/write [ anon ]
F728A000 32 32 - 32 read/write [ anon ]
F72BA000 32 32 - 32 read/write [ anon ]
F72EA000 32 32 - 32 read/write [ anon ]
F731A000 32 32 - 32 read/write [ anon ]
F734A000 32 32 - 32 read/write [ anon ]
F737A000 32 32 - 32 read/write [ anon ]
F73A0000 8 8 - 8 read/write [ anon ]
F73B0000 8 8 8 - read dev:271,11 ino:40262
F73C0000 32 32 32 - read dev:271,11 ino:37108
F73E0000 8 8 - 8 read/write [ anon ]
F740E000 16 16 - 16 read/write [ anon ]
F7420000 56 56 56 - read/exec libmath.so
F743C000 8 8 - 8 read/write/exec libmath.so
F7452000 8 8 - 8 read/write [ anon ]
F7460000 16 16 16 - read dev:271,11 ino:37126
F7470000 24 24 24 - read dev:271,11 ino:18612
F7482000 8 8 - 8 read/write [ anon ]
F7490000 24 24 24 - read dev:271,11 ino:40157
F74A4000 8 8 - 8 read/write [ anon ]
F74B0000 176 176 176 - read/shared dev:271,10 ino:1400
F74E0000 8 8 8 - read/shared dev:271,10 ino:1401
F74F0000 8 8 8 - read/shared dev:271,10 ino:1394
F7500000 240 240 240 - read/shared dev:271,10 ino:1390
F7540000 136 136 136 - read/shared dev:271,10 ino:1389
F7570000 32 32 32 - read/shared dev:271,10 ino:1391
F7580000 40 40 40 - read/shared dev:271,10 ino:1385
F7590000 8 8 8 - read/shared dev:271,10 ino:1384
F75A0000 456 456 456 - read/shared dev:271,11 ino:40919
F7620000 368 368 368 - read/shared dev:271,11 ino:29388
F7680000 624 624 624 - read/shared dev:271,11 ino:40925
F7730000 24 24 24 - read/shared dev:271,10 ino:1383
F7740000 192 192 192 - read/shared dev:271,11 ino:21873
F7780000 584 584 584 - read/shared dev:271,11 ino:21890
F7820000 104 104 104 - read/shared dev:271,10 ino:1392
F7840000 24 24 24 - read/shared dev:271,11 ino:37049
F7850000 32 32 32 - read/shared dev:271,11 ino:40924
F7860000 8 8 8 - read/shared dev:271,11 ino:23830
F7870000 8 8 8 - read/shared dev:271,11 ino:23829
F7880000 16 16 16 - read/shared dev:271,11 ino:24584
F7890000 24 24 24 - read/shared dev:271,11 ino:23821
F78A0000 56 56 56 - read/shared dev:271,11 ino:23827
F78C0000 456 456 456 - read/shared dev:271,11 ino:24582
F7940000 16 16 16 - read/shared dev:271,11 ino:23833
F7950000 104 104 104 - read/shared dev:271,11 ino:23822
F7970000 152 152 152 - read/shared dev:271,11 ino:23826
F79A0000 280 280 280 - read/shared dev:271,11 ino:23825
F79F0000 48 48 48 - read/shared dev:271,11 ino:23806
F7A00000 1608 1608 1608 - read/shared dev:271,11 ino:19762
F7BA0000 16 16 16 - read/shared dev:271,11 ino:23831
F7BB0000 8 8 8 - read/shared dev:271,11 ino:19758
F7BC0000 128 128 128 - read/shared dev:271,11 ino:19757
F7BF0000 8 8 8 - read/shared dev:271,11 ino:19755
F7C00000 10752 6832 6832 - read/shared dev:271,0 ino:54540
F8690000 72 72 72 - read/shared dev:271,11 ino:19751
F86B0000 32 32 32 - read/shared dev:271,11 ino:23820
F86C0000 200 200 200 - read/shared dev:271,11 ino:23823
F8700000 712 712 712 - read/shared dev:271,11 ino:19761
F87C0000 216 216 216 - read/shared dev:271,11 ino:23809
F8800000 3936 3936 - 3936 read/write/exec [ anon ]
F9810000 72 72 72 - read/shared dev:271,11 ino:23807
F9830000 176 176 176 - read/shared dev:271,11 ino:23832
F9860000 8 8 8 - read/shared dev:271,10 ino:1396
F9870000 24 24 24 - read/shared dev:271,10 ino:1443
F9880000 3432 224 224 - read/shared dev:271,0 ino:54537
F9BE0000 8 8 8 - read/shared dev:271,10 ino:1395
F9BF0000 24 24 24 - read/shared dev:271,10 ino:1464
F9C00000 2048 2048 - 2048 read/write/exec [ anon ]
FA400000 2048 2048 - 2048 read/write/exec [ anon ]
FAC00000 36352 36352 - 36352 read/write/exec [ anon ]
FD820000 104 104 104 - read/shared dev:271,11 ino:23810
FD85E000 16 16 - 16 read/write [ anon ]
FD88E000 16 16 - 16 read/write [ anon ]
FD8C0000 8 8 - 8 read/write [ anon ]
FD8D0000 136 40 40 - read/shared dev:271,0 ino:259199
FD900000 1952 1872 1872 - read/shared dev:271,0 ino:259200
FDAF0000 16 16 16 - read/shared dev:271,11 ino:23814
FDB00000 112 24 24 - read/shared dev:271,0 ino:84272
FDB20000 72 72 72 - read/exec libzip.so
FDB40000 16 16 - 16 read/write/exec libzip.so
FDB50000 232 232 232 - read/exec libjava.so
FDB98000 16 16 - 16 read/write/exec libjava.so
FDBC0000 8 8 - 8 read/write [ anon ]
FDBF0000 8 8 - 8 read/write [ anon ]
FDC00000 448 448 448 - read/exec libnsrwtool.so
FDC7E000 56 56 - 56 read/write/exec libnsrwtool.so
FDC90000 64 64 64 - read/shared dev:271,11 ino:23811
FDCB0000 120 120 - 120 read/write/exec [ anon ]
FDCF0000 8 8 - 8 read/write [ anon ]
FDD00000 2744 2744 2744 - read/exec libjvm.so
FDFBC000 88 80 - 80 read/write/exec libjvm.so
FDFD2000 48 48 - 48 read/write/exec libjvm.so
FDFF0000 8 8 - 8 read/write/exec [ anon ]
FE002000 8 8 - 8 read/write [ anon ]
FE104000 8 8 - 8 read/write [ anon ]
FE206000 8 8 - 8 read/write [ anon ]
FE308000 8 8 - 8 read/write [ anon ]
FE40A000 8 8 - 8 read/write [ anon ]
FE50C000 8 8 - 8 read/write [ anon ]
FE60A000 8 8 - 8 read/write [ anon ]
FE60E000 8 8 - 8 read/write [ anon ]
FE70C000 8 8 - 8 read/write [ anon ]
FE710000 8 8 - 8 read/write [ anon ]
FE80E000 8 8 - 8 read/write [ anon ]
FE82A000 32 32 - 32 read/write [ anon ]
FE850000 8 8 - 8 read/write [ anon ]
FE870000 8 8 - 8 read/write [ anon ]
FE890000 8 8 - 8 read/write [ anon ]
FE8B0000 8 8 - 8 read/write [ anon ]
FE8C0000 40 40 40 - read/write/shared dev:0,2 ino:37726981
FE8D0000 8 8 - 8 read/write/exec [ anon ]
FE8E0000 16 16 16 - read/exec libConfigFile.so
FE8F2000 8 8 - 8 read/write/exec libConfigFile.so
FE900000 16 16 16 - read/exec libMMapSession.so
FE912000 8 8 - 8 read/write/exec libMMapSession.so
FE920000 8 8 8 - read/exec libNSJavaMiscUtil.so
FE930000 8 8 - 8 read/write/exec libNSJavaMiscUtil.so
FE940000 8 8 - 8 read/write/exec [ anon ]
FE950000 24 24 24 - read/exec libNSJvmControl.so
FE964000 16 16 - 16 read/write/exec libNSJvmControl.so
FE970000 40 40 40 - read/exec libNSJavaUtil.so
FE988000 16 16 - 16 read/write/exec libNSJavaUtil.so
FE990000 32 32 32 - read/exec libNSServletLayer.so
FE9A6000 8 8 - 8 read/write/exec libNSServletLayer.so
FE9B0000 48 48 48 - read/exec libNSServletPlugin.so
FE9CA000 16 16 - 16 read/write/exec libNSServletPlugin.so
FE9E0000 24 24 24 - read/exec nss_files.so.1
FE9F6000 8 8 - 8 read/write/exec nss_files.so.1
FEA00000 8 8 8 - read/write/exec/shared [ anon ]
FEA14000 8 8 - 8 read/write [ anon ]
FEA20000 8 8 8 - read/write/exec/shared [ anon ]
FEA30000 8 8 - 8 read/write/exec [ anon ]
FEA40000 8 8 8 - read/exec libatomic.so
FEA50000 8 8 8 - read/write/exec libatomic.so
FEA60000 32 32 32 - read/exec libaio.so.1
FEA78000 8 8 - 8 read/write/exec libaio.so.1
FEA80000 8 8 - 8 read/write/exec [ anon ]
FEA90000 16 16 16 - read/exec libmp.so.2
FEAA4000 8 8 - 8 read/write/exec libmp.so.2
FEAB0000 24 24 24 - read/exec libpthread.so.1
FEAC6000 8 8 - 8 read/write/exec libpthread.so.1
FEAD0000 208 168 168 - read/exec libresolv.so.2
FEB14000 24 24 - 24 read/write/exec libresolv.so.2
FEB20000 16 16 16 - read/exec libares3.so
FEB32000 16 16 - 16 read/write/exec libares3.so
FEB40000 24 24 24 - read/exec libsupport.so
FEB54000 8 8 - 8 read/write/exec libsupport.so
FEB60000 24 24 24 - read/exec libnstime.so
FEB74000 8 8 - 8 read/write/exec libnstime.so
FEB80000 8 8 8 - read/exec libdirmon4.so
FEB90000 8 8 - 8 read/write/exec libdirmon4.so
FEBA0000 8 8 8 - read/exec libnstp.so
FEBB0000 8 8 - 8 read/write/exec libnstp.so
FEBC0000 32 32 32 - read/exec libnsfc.so
FEBD6000 136 136 104 32 read/write/exec libnsfc.so
FEC00000 688 688 688 - read/exec libc.so.1
FECBC000 32 32 - 32 read/write/exec libc.so.1
FECD0000 8 8 - 8 read/write/exec [ anon ]
FECE0000 112 112 112 - read/exec libthread.so.1
FED0C000 8 8 - 8 read/write/exec libthread.so.1
FED0E000 48 48 - 48 read/write/exec libthread.so.1
FED20000 8 8 8 - read/exec libc_psr.so.1
FED30000 264 216 216 - read/exec libC.so.5
FED80000 32 32 - 32 read/write/exec libC.so.5
FED88000 56 16 - 16 read/write/exec libC.so.5
FEDA0000 8 8 - 8 read/write/exec [ anon ]
FEDB0000 224 64 64 - read/exec libm.so.1
FEDF6000 8 8 - 8 read/write/exec libm.so.1
FEE00000 568 568 568 - read/exec libnsl.so.1
FEE9E000 40 40 - 40 read/write/exec libnsl.so.1
FEEA8000 24 16 - 16 read/write/exec libnsl.so.1
FEEC0000 24 24 24 - read/exec librt.so.1
FEED6000 8 8 - 8 read/write/exec librt.so.1
FEEE0000 8 8 - 8 read/write/exec libdl.so.1
FEEF0000 40 40 40 - read/exec libsocket.so.1
FEF0A000 8 8 - 8 read/write/exec libsocket.so.1
FEF10000 176 176 176 - read/exec libnspr3.so
FEF4A000 16 16 - 16 read/write/exec libnspr3.so
FEF60000 16 16 16 - read/exec libplds3.so
FEF72000 8 8 - 8 read/write/exec libplds3.so
FEF80000 16 16 16 - read/exec libplc3.so
FEF92000 8 8 - 8 read/write/exec libplc3.so
FEFA0000 24 24 24 - read/exec libnsbrk30.so
FEFB4000 16 16 - 16 read/write/exec libnsbrk30.so
FEFC0000 8 8 - 8 read/write/exec [ anon ]
FEFD0000 96 96 96 - read/exec libnscol30.so
FEFF6000 40 40 24 16 read/write/exec libnscol30.so
FF010000 184 184 184 - read/exec libnsfmt30.so
FF04C000 32 32 - 32 read/write/exec libnsfmt30.so
FF060000 80 80 80 - read/exec libnscnv30.so
FF082000 32 32 - 32 read/write/exec libnscnv30.so
FF090000 152 152 152 - read/exec libnsuni30.so
FF0C4000 40 40 16 24 read/write/exec libnsuni30.so
FF0E0000 96 96 96 - read/exec libnsres30.so
FF106000 24 24 - 24 read/write/exec libnsres30.so
FF10C000 8 - - - read/write/exec libnsres30.so
FF120000 168 168 168 - read/exec libldap30.so
FF158000 16 16 - 16 read/write/exec libldap30.so
FF160000 8 8 8 - read/exec libMagnusPostInit.so
FF170000 8 8 - 8 read/write/exec libMagnusPostInit.so
FF180000 16 16 16 - read/exec libnsprwrap.so
FF192000 8 8 - 8 read/write/exec libnsprwrap.so
FF1A0000 40 40 40 - read/exec liblibdbm.so
FF1B8000 8 8 - 8 read/write/exec liblibdbm.so
FF1C0000 8 8 - 8 read/write/exec [ anon ]
FF1D0000 32 32 32 - read/exec liblibsi18n.so
FF1E6000 72 72 56 16 read/write/exec liblibsi18n.so
FF200000 936 936 936 - read/exec libns-httpd40.so
FF2F8000 192 192 8 184 read/write/exec libns-httpd40.so
FF328000 216 48 - 48 read/write/exec libns-httpd40.so
FF370000 8 8 8 - read/exec libgetprop.so
FF380000 8 8 - 8 read/write/exec libgetprop.so
FF390000 8 8 8 - read/exec libw.so.1
FF3A0000 176 176 176 - read/exec ld.so.1
FF3DC000 8 8 - 8 read/write/exec ld.so.1
FF3DE000 8 8 - 8 read/write/exec ld.so.1
FFBE2000 56 56 - 56 read/write [ stack ]
-------- ------ ------ ------ ------
total Kb 110912 102824 28976 73848

RevelationCS

ASKER

Also, to note, based on what I am seeing, swap -l has not changed even though the usage of disk space continues to go up. swap -s is slightly lower than what it had been when I posted above, however, the instance was restarted after some updates were made to some class files so I do not know if this would be a good indicator... I will keep an eye on that today and see if it grows as the disk usage grows...

As it stands from our research, the disk usage is growing at about a rate of 15MB/hr. Also was informed by on of the unix guys that this could have been going on for some time, but they made a change about a month prior to when this issue started to limit /tmp to 2GB, where prior to that all virtual memory on the server would be available for /tmp.....

RevelationCS

ASKER

also, to note..... the server itself has 8GB of physical RAM in the server.... peak usage is about 4.5 GB

Nukfror

RevelationCS,

So df -k shows /tmp not really growing in size ? If so, then limit on /tmp to 2GB, while a very good idea, doesn't help in this case. This is what I think you indicated the first time you opened this issue and why I originally said I doubted this had anything to do with /tmp physical utilization of anonymous memory. If /tmp isn't physical growing into the anonymous memory space, its not the problem.

Which side of the anonymous memory equation is growing: allocated or reserved ?

Reserved mean a program has asked for more memory but hasn't physical touched it yet. Allocated means the memory has been touched and is pinned in anonymous memory land: either physical memory or possibly backstored on top physical swap space.

I'm thinking its probably allocated ?

Your comment about swap -l not really growing means that the physical memory currently marked as anonymous memory and actually being used hasn't been exhausted yet so backstoring those pages to physical swap isn't necessary yet.

RevelationCS

ASKER

Nuk,

To clarify... df -k SHOWS /tmp growing in size when it comes to "USED SPACE", however, the allocated size of the partition has remained the same.

RevelationCS

ASKER

running the "/usr/bin/lsof +aL1 /tmp" command, we are a able to view a listing of unused but not unallocated files... It appears that this is common with iPlanet and SunOne from what we can see, and with this, we are noticing the procs associated with typing up disk space having a large size of the files being listed with this command... To relate, we are also unable to locate files of these sizes on the actual /tmp partition itself...

Nukfror

K - sorry. My original impression was /tmp wasn't getting touched and available anonymous space was shrinking from either allocations or reserves.

So is pmap showing these files and associated inode number as residing in /tmp but find can't find them (cuz the link count is 0) ?

RevelationCS

ASKER

after doing further research using the lsof command, it appears that these "phantom" files are related to the logging functions (STDERR/STDOUT) for the JVM on the web instances (files ultimately traced back to /proc/<PID>/fd/1 and /proc/<PID>/fd/2 and both files contain what appears to be contents of logs....

I have addressed the issue with the developers to have them check how they are using the STDERR and STDOUT within their applications. Does anyone know if there is a way to force UNIX/Solaris to clean this up periodically without having to restart the proc? Also, is there a way for us to define in Solaris where STDERR and STDOUT should be going if unable to write to a file?

Nukfror

In your case, STDOUT (fd 1) and STDERR (fd 2) are going to the exact same file.

Cleaning up depends on how the application is written. You could try simply this:

cp /dev/null /tmp/whatever-the-filename-is

Wait for about a minute or two and see if the file is now 0 bytes+<whatever was written since you ran the cp command> . If the file comes back to its original size plus some then this won't work. This sometimes works. You can play this came with syslog all day. Doesn't work everytime.

To my knowledge, if you direct STDOUT and STDERR to anything the process can't write to the app will error out immediately. I've always wished Solaris had a user defined buffer ring log capability like the console buffer.

One idea (but very ugly) is to redirect STDOUT and STDERR to a named pipe. Have an application you write on the other side of the pipe that will roll the log file it writes when you hit it with say a SIGHUP. Pretty sure there are examples of doing this with Perl in multiple places on the net.

RevelationCS

ASKER

one thought we are having is with STDERR and STDOUT trying to log to the console... is there a possibility that this can be defined by developers? I know the Log4J logging and a few other applications can be set to log to console, but am unsure if the system outs can do this or not...

RevelationCS

ASKER

still working on testing a fix to the start scripts to resolve this issue... I will keep you posted as to the resolution...

Nukfror

Another idea is to take the logger program and stick it on the end of the named pipe. See the man page for more information. Use the -f switch to specify the named pipe as contents to log. You should redirect both STDOUT and STDERR to the named pipe.

RevelationCS

ASKER

over the weekend we installed the latest service patch for both of the webservers that we were noticing the problems with and it appears to have resolved the issue at this point in time... usage of /tmp has maintained itself at 4% over the last few days... I will continue to monitor for the next week or so and post back if this resolved the issue...

RevelationCS

ASKER

latest results of the lsof command:

ns-httpd 6377 efdadmin 1w VREG 0,2 215412914 0 49274930 /tmp (swap)
ns-httpd 6377 efdadmin 2w VREG 0,2 215412914 0 49274930 /tmp (swap)

It appears that we still have one offending process under the webservers, however, things have greatly improved since the issue was noticed. I am planning on returning this information for the remaining process to the developers to have them look at their code for potential causes of the issue...

RevelationCS

ASKER

Still no issues as of yet and it appears that the /tmp directory has stayed under control... I will be posting a request to have the questioned PAQ'd...

ASKER CERTIFIED SOLUTION

OzzMod

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial