Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
Solved

What's causing my disk usage to creep up?

Posted on 2006-11-12
5
230 Views
Last Modified: 2013-12-16
I have a relatively complicated Java process that runs as a daemon on my server. I know that it makes substantial use of temporary files, which are always created in /tmp.

While it runs, `df` reports less and less available space, and over 24 hours it goes from 45% -> 90% disk usage on a 72GB file system.

    root@thb-mta-t1:~# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda1              71G   60G  7.5G  89% /

However, running `du` on the directories (excliding an NFS mount nfs/)

    root@thb-mta-t1:~# cd /
    root@thb-mta-t1:/# du -sh `ls | perl -ne 'if (!/^nfs/){print $_;}'`

Does not show anything that could account for the extra disk usage. I see only 88M in /tmp.

Here's the odd bit.... I can kill the java process with kill -9 (SIGKILL - i.e. no allowing any shutdown hooks to do clean-up) and the disk space is reclaimed. A more polite kill (SIGTERM) gradually relinquishes a good deal of the disk space, but hangs.

Any ideas?
0
Comment
Question by:rstaveley
  • 3
  • 2
5 Comments
 
LVL 43

Accepted Solution

by:
ravenpl earned 500 total points
ID: 17924879
> Here's the odd bit....
Noting odd, usual case.
The java process creates temporary files, which fill the disk space. They are not seen to du command, cause they are already removed(unlinked).
Process can open file, then unlink it, and untill it close the file, the file occupies disk space. The process may write to such file enlarging the occupied space.
To find such files, use
lsof +L1 #command
0
 
LVL 17

Author Comment

by:rstaveley
ID: 17925438
How odd. I'm sure you are right, but can you really unlink a file that's not closed?

Off to try that lsof command...

Many thanks, ravenpl.
0
 
LVL 17

Author Comment

by:rstaveley
ID: 17925454
You're right of course. I'm seeing reems of files marked as deleted with lsof. That's thrown me though. Would you hazard a guess that one thread has unlinked the file while another has retained a reference to it, or do you reckon this is symptomatic of simply failing to close a file before deleting it?
0
 
LVL 43

Expert Comment

by:ravenpl
ID: 17926129
I never thought of the situation where one thread keeps using file deleted by other.
But it's normal situation, where process needs some temporary file, so it opens/creates the file, then unlinks immediatelly. This way the file resources will be freed no matter the application finishes or crashes or gets killed.
If the application forgets to close the file, but unlinks - it's a bug! Especially for server applications.
0
 
LVL 17

Author Comment

by:rstaveley
ID: 17926177
I think it is a bug (mine!), but it is related now - I realise - to the fact that I have two processes accessing the same files. Specifically, the files are a database index (if that's the right thing to call a Lucene index). One process reads and the other process writes. The process which writes is supposed to close the index intermittently and the process that reads is supposed to pick up on the fact that the index has bee modified and close its file handles and then reopen. It looks like the writer is doing its stuff, but the reader is holding the orginal files open, when it reloads. Armed with my new found better understanding now about file unlinks 8-), I reckon that the bug is a failure to close files in the reader, when the files have been deleted by the writer. Thank you so much for your patient explanation, ravenpl. Java let's you get away with being sloppy about resource management.... on the whole.
0

Featured Post

Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
bash file 10 49
How to mount nfs share on this CentOS server? 6 42
Migrating a Linux server to VMware 3 53
Linux MD5 Hash 7 36
How many times have you wanted to quickly do the same thing to a list but found yourself typing it again and again? I first figured out a small time saver with the up arrow to recall the last command but that can only get you so far if you have a bi…
It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

839 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question