Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Something is eating disk space

Posted on 2011-03-11
13
Medium Priority
?
1,157 Views
Last Modified: 2013-12-16
Hello,  
 
New to Linux.  Something is eating disk space at about 0.82 GB an hour.  This will continue until disk is at 100%.  A reboot frees all of the phantom disk usage and we are back to 18% used.

I check the disk space using WebMin.  It is the Disk mounted as
/ (Root filesystem)       Linux Native Filesystem (ext3)       Partition labelled /1

We are running:
CentOS Linux 5.5
Webmin version 1.510
MySQL version 5.1.52
Apache version 2.2.17
PHP - don't know the version
Zimbra email server

I've tried restarting MySQL, Apache, VNC & Zimbra.  No change
Rebooting is the only thing that will help.

I know you cannot diagnose the issue without access to the server, but has anyone run into this issue?  Any ideas of where to look?  

Thanks

0
Comment
Question by:Waterstone
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
  • 2
  • +4
13 Comments
 
LVL 79

Expert Comment

by:arnold
ID: 35110299
Look in /var/tmp
/tmp
Those are the two locations for temporary storage I think /tmp
/var/log
do
cd /
ls | while read a; do
du -ks $a
done
This will narrow the path down to where the growth is.
You can then repeat the above to further narrow down where the growth is seen
cd 'to the directory where the increase is seen"
repeat the above.
post df -k to see the mount points that you have


0
 
LVL 40

Accepted Solution

by:
omarfarid earned 2000 total points
ID: 35110454
you can do the following

touch /tmp/myfile
sleep 30
find / -newer -exec ls -l {} \;

this should give you files that are growing in size

you may run

cd /
du -k | sort -rn | more

0
 
LVL 79

Expert Comment

by:arnold
ID: 35110552
a slight correction, you omitted the reference to the file in the find command which should be
find / -newer /tmp/myfile -exec ls -l {} \;

The problem I see with the above is it will scan the entire system while my suggestion would first narrow down to where the growth is seen, and then you can look to see what is writting/creating those entries.
0
Survive A High-Traffic Event with Percona

Your application or website rely on your database to deliver information about products and services to your customers. You can’t afford to have your database lose performance, lose availability or become unresponsive – even for just a few minutes.

 
LVL 40

Assisted Solution

by:omarfarid
omarfarid earned 2000 total points
ID: 35110586
thanks for the correction :)

Yes, but this command will run few times till find the file(s) growing
0
 
LVL 7

Expert Comment

by:jackiechen858
ID: 35111591
Another approach:

you can run
lsof  | gawk '{if (($4 ~ /[0-9]+w/) && (  $5 ~ /REG/ ) ) print $0}'

This will tell you which files are being written at this moment.
0
 
LVL 7

Expert Comment

by:jackiechen858
ID: 35111628
ls -l `lsof  | gawk '{if (($4 ~ /[0-9]+w/) && (  $5 ~ /REG/ ) ) print $0}' |gawk '{print $NF}' | sort -u` | sort -n -k5

the above command will tell you all the currently written file's size, and sort them by their size. You should check the bottom line file
0
 
LVL 12

Expert Comment

by:hfraser
ID: 35124368
It's unusual that the space os restored after a simple reboot; most files would still exist after the reboot, even those in /var/log or /tmp. Try stopping services (not restart) and checking after each to see if the space is restored.

Sometimes installations of databases like mysql include an optimize step as part of the startup that might recover some disk space, but even that;s a stretch unless you have an app that's doing a lot of insert/delete traffic between restarts. .82G/hour would imply a lot of traffic to the database.
0
 
LVL 57

Expert Comment

by:giltjr
ID: 35134974
I've had a problem like this before and it was a real pain to figure out.  To tell the truth I'm not sure how we figured out what file it was.

In our situation somebody created a script that was exeucted at boot time to delete a file.  The problem is that the file was a log file that was in use.  So, you can't ls it any more, but it continunes to grow and grow.  When you re-boot, actually stop the service that was writting to the log file, the file file is then really deleted and the space freed up.

Again, I know we narrowed it down to the directory, but I can't remember how we found the exact file.
0
 
LVL 79

Expert Comment

by:arnold
ID: 35135093
The lsof method that was referenced in post http:#35111628 by jackiechen858 is one.
The other is look at all the services that start.
Another is narrow down the list to the directory where the space if being taken up and then run lsof +d <directoryName> this will tell you want process has what files open.
i.e. if the space you are missing is in /var/tmp
lsof +d /var/tmp will display all the processes that have an open filehandle into that directory
Then you could look at which file is not showing up in /var/tmp from the list.
I've seen issues similar to this, but the space did not came back on reboot i.e. the reference to the filename was being cleared from the filesystem, but it was incomplete i.e. it was not freeing the space.
0
 
LVL 12

Expert Comment

by:hfraser
ID: 35138609
It's certainly possible an app is creating and extending a file quickly, deleting it when it exits in response to the signal it receives when the system's shutting down. Stopping services one at a time, folowed by remaining processes, might identify the culprit.

You could also install a tool like iotop to see what process is doing the most I/O. Process accounting can also help with similar stats.
0
 
LVL 19

Expert Comment

by:jools
ID: 35176715
it sounds like /tmp is filling up which is why the reboot "fixes" the problem as /tmp is cleared on bootup this is likely the case if /tmp is not a separate file system.

You could create a separate file system for /tmp and mount this as /tmp.

If you need assistance you need to respond to some of the questions already posted.
0
 

Author Closing Comment

by:Waterstone
ID: 35226414
Sorry I let this question dangle.  Been fighting fires.  We found the file was a VNC error file that was filling up because of a syntax error in the VNC config file.  Fixed that and we are good.  

Thanks for all the replies.
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 35229240
Welcome
0

Featured Post

Migrating Your Company's PCs

To keep pace with competitors, businesses must keep employees productive, and that means providing them with the latest technology. This document provides the tips and tricks you need to help you migrate an outdated PC fleet to new desktops, laptops, and tablets.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Setting up Secure Ubuntu server on VMware 1.      Insert the Ubuntu Server distribution CD or attach the ISO of the CD which is in the “Datastore”. Note that it is important to install the x64 edition on servers, not the X86 editions. 2.      Power on th…
In the first part of this tutorial we will cover the prerequisites for installing SQL Server vNext on Linux.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Suggested Courses

670 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question