Solved

What's causing my disk usage to creep up?

Posted on 2006-11-12
5
227 Views
Last Modified: 2013-12-16
I have a relatively complicated Java process that runs as a daemon on my server. I know that it makes substantial use of temporary files, which are always created in /tmp.

While it runs, `df` reports less and less available space, and over 24 hours it goes from 45% -> 90% disk usage on a 72GB file system.

    root@thb-mta-t1:~# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda1              71G   60G  7.5G  89% /

However, running `du` on the directories (excliding an NFS mount nfs/)

    root@thb-mta-t1:~# cd /
    root@thb-mta-t1:/# du -sh `ls | perl -ne 'if (!/^nfs/){print $_;}'`

Does not show anything that could account for the extra disk usage. I see only 88M in /tmp.

Here's the odd bit.... I can kill the java process with kill -9 (SIGKILL - i.e. no allowing any shutdown hooks to do clean-up) and the disk space is reclaimed. A more polite kill (SIGTERM) gradually relinquishes a good deal of the disk space, but hangs.

Any ideas?
0
Comment
Question by:rstaveley
  • 3
  • 2
5 Comments
 
LVL 43

Accepted Solution

by:
ravenpl earned 500 total points
Comment Utility
> Here's the odd bit....
Noting odd, usual case.
The java process creates temporary files, which fill the disk space. They are not seen to du command, cause they are already removed(unlinked).
Process can open file, then unlink it, and untill it close the file, the file occupies disk space. The process may write to such file enlarging the occupied space.
To find such files, use
lsof +L1 #command
0
 
LVL 17

Author Comment

by:rstaveley
Comment Utility
How odd. I'm sure you are right, but can you really unlink a file that's not closed?

Off to try that lsof command...

Many thanks, ravenpl.
0
 
LVL 17

Author Comment

by:rstaveley
Comment Utility
You're right of course. I'm seeing reems of files marked as deleted with lsof. That's thrown me though. Would you hazard a guess that one thread has unlinked the file while another has retained a reference to it, or do you reckon this is symptomatic of simply failing to close a file before deleting it?
0
 
LVL 43

Expert Comment

by:ravenpl
Comment Utility
I never thought of the situation where one thread keeps using file deleted by other.
But it's normal situation, where process needs some temporary file, so it opens/creates the file, then unlinks immediatelly. This way the file resources will be freed no matter the application finishes or crashes or gets killed.
If the application forgets to close the file, but unlinks - it's a bug! Especially for server applications.
0
 
LVL 17

Author Comment

by:rstaveley
Comment Utility
I think it is a bug (mine!), but it is related now - I realise - to the fact that I have two processes accessing the same files. Specifically, the files are a database index (if that's the right thing to call a Lucene index). One process reads and the other process writes. The process which writes is supposed to close the index intermittently and the process that reads is supposed to pick up on the fact that the index has bee modified and close its file handles and then reopen. It looks like the writer is doing its stuff, but the reader is holding the orginal files open, when it reloads. Armed with my new found better understanding now about file unlinks 8-), I reckon that the bug is a failure to close files in the reader, when the files have been deleted by the writer. Thank you so much for your patient explanation, ravenpl. Java let's you get away with being sloppy about resource management.... on the whole.
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Network Interface Card (NIC) bonding, also known as link aggregation, NIC teaming and trunking, is an important concept to understand and implement in any environment where high availability is of concern. Using this feature, a server administrator …
SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

8 Experts available now in Live!

Get 1:1 Help Now