?
Solved

What's causing my disk usage to creep up?

Posted on 2006-11-12
5
Medium Priority
?
234 Views
Last Modified: 2013-12-16
I have a relatively complicated Java process that runs as a daemon on my server. I know that it makes substantial use of temporary files, which are always created in /tmp.

While it runs, `df` reports less and less available space, and over 24 hours it goes from 45% -> 90% disk usage on a 72GB file system.

    root@thb-mta-t1:~# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda1              71G   60G  7.5G  89% /

However, running `du` on the directories (excliding an NFS mount nfs/)

    root@thb-mta-t1:~# cd /
    root@thb-mta-t1:/# du -sh `ls | perl -ne 'if (!/^nfs/){print $_;}'`

Does not show anything that could account for the extra disk usage. I see only 88M in /tmp.

Here's the odd bit.... I can kill the java process with kill -9 (SIGKILL - i.e. no allowing any shutdown hooks to do clean-up) and the disk space is reclaimed. A more polite kill (SIGTERM) gradually relinquishes a good deal of the disk space, but hangs.

Any ideas?
0
Comment
Question by:rstaveley
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
5 Comments
 
LVL 43

Accepted Solution

by:
ravenpl earned 2000 total points
ID: 17924879
> Here's the odd bit....
Noting odd, usual case.
The java process creates temporary files, which fill the disk space. They are not seen to du command, cause they are already removed(unlinked).
Process can open file, then unlink it, and untill it close the file, the file occupies disk space. The process may write to such file enlarging the occupied space.
To find such files, use
lsof +L1 #command
0
 
LVL 17

Author Comment

by:rstaveley
ID: 17925438
How odd. I'm sure you are right, but can you really unlink a file that's not closed?

Off to try that lsof command...

Many thanks, ravenpl.
0
 
LVL 17

Author Comment

by:rstaveley
ID: 17925454
You're right of course. I'm seeing reems of files marked as deleted with lsof. That's thrown me though. Would you hazard a guess that one thread has unlinked the file while another has retained a reference to it, or do you reckon this is symptomatic of simply failing to close a file before deleting it?
0
 
LVL 43

Expert Comment

by:ravenpl
ID: 17926129
I never thought of the situation where one thread keeps using file deleted by other.
But it's normal situation, where process needs some temporary file, so it opens/creates the file, then unlinks immediatelly. This way the file resources will be freed no matter the application finishes or crashes or gets killed.
If the application forgets to close the file, but unlinks - it's a bug! Especially for server applications.
0
 
LVL 17

Author Comment

by:rstaveley
ID: 17926177
I think it is a bug (mine!), but it is related now - I realise - to the fact that I have two processes accessing the same files. Specifically, the files are a database index (if that's the right thing to call a Lucene index). One process reads and the other process writes. The process which writes is supposed to close the index intermittently and the process that reads is supposed to pick up on the fact that the index has bee modified and close its file handles and then reopen. It looks like the writer is doing its stuff, but the reader is holding the orginal files open, when it reloads. Armed with my new found better understanding now about file unlinks 8-), I reckon that the bug is a failure to close files in the reader, when the files have been deleted by the writer. Thank you so much for your patient explanation, ravenpl. Java let's you get away with being sloppy about resource management.... on the whole.
0

Featured Post

Get proactive database performance tuning online

At Percona’s web store you can order full Percona Database Performance Audit in minutes. Find out the health of your database, and how to improve it. Pay online with a credit card. Improve your database performance now!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Fine Tune your automatic Updates for Ubuntu / Debian
Google Drive is extremely cheap offsite storage, and it's even possible to get extra storage for free for two years.  You can use the free account 15GB, and if you have an Android device..when you install Google Drive for the first time it will give…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
Suggested Courses
Course of the Month8 days, 14 hours left to enroll

764 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question