What's causing my disk usage to creep up?

I have a relatively complicated Java process that runs as a daemon on my server. I know that it makes substantial use of temporary files, which are always created in /tmp.

While it runs, `df` reports less and less available space, and over 24 hours it goes from 45% -> 90% disk usage on a 72GB file system.

    root@thb-mta-t1:~# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda1              71G   60G  7.5G  89% /

However, running `du` on the directories (excliding an NFS mount nfs/)

    root@thb-mta-t1:~# cd /
    root@thb-mta-t1:/# du -sh `ls | perl -ne 'if (!/^nfs/){print $_;}'`

Does not show anything that could account for the extra disk usage. I see only 88M in /tmp.

Here's the odd bit.... I can kill the java process with kill -9 (SIGKILL - i.e. no allowing any shutdown hooks to do clean-up) and the disk space is reclaimed. A more polite kill (SIGTERM) gradually relinquishes a good deal of the disk space, but hangs.

Any ideas?
LVL 17
rstaveleyAsked:
Who is Participating?
 
ravenplCommented:
> Here's the odd bit....
Noting odd, usual case.
The java process creates temporary files, which fill the disk space. They are not seen to du command, cause they are already removed(unlinked).
Process can open file, then unlink it, and untill it close the file, the file occupies disk space. The process may write to such file enlarging the occupied space.
To find such files, use
lsof +L1 #command
0
 
rstaveleyAuthor Commented:
How odd. I'm sure you are right, but can you really unlink a file that's not closed?

Off to try that lsof command...

Many thanks, ravenpl.
0
 
rstaveleyAuthor Commented:
You're right of course. I'm seeing reems of files marked as deleted with lsof. That's thrown me though. Would you hazard a guess that one thread has unlinked the file while another has retained a reference to it, or do you reckon this is symptomatic of simply failing to close a file before deleting it?
0
 
ravenplCommented:
I never thought of the situation where one thread keeps using file deleted by other.
But it's normal situation, where process needs some temporary file, so it opens/creates the file, then unlinks immediatelly. This way the file resources will be freed no matter the application finishes or crashes or gets killed.
If the application forgets to close the file, but unlinks - it's a bug! Especially for server applications.
0
 
rstaveleyAuthor Commented:
I think it is a bug (mine!), but it is related now - I realise - to the fact that I have two processes accessing the same files. Specifically, the files are a database index (if that's the right thing to call a Lucene index). One process reads and the other process writes. The process which writes is supposed to close the index intermittently and the process that reads is supposed to pick up on the fact that the index has bee modified and close its file handles and then reopen. It looks like the writer is doing its stuff, but the reader is holding the orginal files open, when it reloads. Armed with my new found better understanding now about file unlinks 8-), I reckon that the bug is a failure to close files in the reader, when the files have been deleted by the writer. Thank you so much for your patient explanation, ravenpl. Java let's you get away with being sloppy about resource management.... on the whole.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.