Solved

What's causing my disk usage to creep up?

Posted on 2006-11-12
5
228 Views
Last Modified: 2013-12-16
I have a relatively complicated Java process that runs as a daemon on my server. I know that it makes substantial use of temporary files, which are always created in /tmp.

While it runs, `df` reports less and less available space, and over 24 hours it goes from 45% -> 90% disk usage on a 72GB file system.

    root@thb-mta-t1:~# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda1              71G   60G  7.5G  89% /

However, running `du` on the directories (excliding an NFS mount nfs/)

    root@thb-mta-t1:~# cd /
    root@thb-mta-t1:/# du -sh `ls | perl -ne 'if (!/^nfs/){print $_;}'`

Does not show anything that could account for the extra disk usage. I see only 88M in /tmp.

Here's the odd bit.... I can kill the java process with kill -9 (SIGKILL - i.e. no allowing any shutdown hooks to do clean-up) and the disk space is reclaimed. A more polite kill (SIGTERM) gradually relinquishes a good deal of the disk space, but hangs.

Any ideas?
0
Comment
Question by:rstaveley
  • 3
  • 2
5 Comments
 
LVL 43

Accepted Solution

by:
ravenpl earned 500 total points
ID: 17924879
> Here's the odd bit....
Noting odd, usual case.
The java process creates temporary files, which fill the disk space. They are not seen to du command, cause they are already removed(unlinked).
Process can open file, then unlink it, and untill it close the file, the file occupies disk space. The process may write to such file enlarging the occupied space.
To find such files, use
lsof +L1 #command
0
 
LVL 17

Author Comment

by:rstaveley
ID: 17925438
How odd. I'm sure you are right, but can you really unlink a file that's not closed?

Off to try that lsof command...

Many thanks, ravenpl.
0
 
LVL 17

Author Comment

by:rstaveley
ID: 17925454
You're right of course. I'm seeing reems of files marked as deleted with lsof. That's thrown me though. Would you hazard a guess that one thread has unlinked the file while another has retained a reference to it, or do you reckon this is symptomatic of simply failing to close a file before deleting it?
0
 
LVL 43

Expert Comment

by:ravenpl
ID: 17926129
I never thought of the situation where one thread keeps using file deleted by other.
But it's normal situation, where process needs some temporary file, so it opens/creates the file, then unlinks immediatelly. This way the file resources will be freed no matter the application finishes or crashes or gets killed.
If the application forgets to close the file, but unlinks - it's a bug! Especially for server applications.
0
 
LVL 17

Author Comment

by:rstaveley
ID: 17926177
I think it is a bug (mine!), but it is related now - I realise - to the fact that I have two processes accessing the same files. Specifically, the files are a database index (if that's the right thing to call a Lucene index). One process reads and the other process writes. The process which writes is supposed to close the index intermittently and the process that reads is supposed to pick up on the fact that the index has bee modified and close its file handles and then reopen. It looks like the writer is doing its stuff, but the reader is holding the orginal files open, when it reloads. Armed with my new found better understanding now about file unlinks 8-), I reckon that the bug is a failure to close files in the reader, when the files have been deleted by the writer. Thank you so much for your patient explanation, ravenpl. Java let's you get away with being sloppy about resource management.... on the whole.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Using 'screen' for session sharing, The Simple Edition Step 1: user starts session with command: screen Step 2: other user (logged in with same user account) connects with command: screen -x Done. Both users are connected to the same CLI sessio…
Linux users are sometimes dumbfounded by the severe lack of documentation on a topic. Sometimes, the documentation is copious, but other times, you end up with some obscure "it varies depending on your distribution" over and over when searching for …
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

920 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now