We help IT Professionals succeed at work.

Always 100% disk use while disk is not full

bt33
bt33 asked
on
Medium Priority
2,302 Views
Last Modified: 2012-06-27
A friend of mine asked me to look at his server because after a crash it seemed that the websites hosted on the server would not be available (couldn't find server). After some searching I'm fairly sure that Apache just crashes because if I reboot Apache it works again for a little while. I searched google for some answers and found one but I couldn't do anything because the server keeps telling me there is no disk space left. After deleting a few 100MB I still got the message there is no disk space left. I'm not that great with linux so I googled some more to find out the command to check disk space, I used this:

df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/vzfs             2.0G  633M     0 100% /
simfs                 2.0G  633M     0 100% /tmp
simfs                 2.0G  633M     0 100% /var/tmp

Nothing I do seems to fix this, the Use is always 100%. When I delete files the 'Used' amount decreases but the 'Use' perentage stays 100%. So it seems that linux thinks the server has no space left but that's not true. What can it be?
Comment
Watch Question

Are you on Virtuozzo, or OpenVZ?
CERTIFIED EXPERT
Top Expert 2007

Commented:
can you take system to single user mode (this will make it unavailable on the network till you take it to multi user mode again)? You may run then

init s
fsck -y

then reboot the system :

reboot

Author

Commented:
ai_ja_nai: I log into the server through Secure Shell Client, I'm not sure what you mean by Virtuozzo or OpenVZ, sorry.

omarfarid: I'm not behind my own PC now so can't test what you recommended but I will try when I get home (might be tomorrow). When I reboot the server do I need to put it back in multi user again?
Hugh FraserConsultant

Commented:
Deleting files that are currently open does not release the space until the files are closed. You can check for processes with open files using "lsof | grep deleted", checking for processes holding open the files you've deleted.

This doesn't however, answer why the "%" value remains the same, especially if you've removed 200M on a 2G volume, which should free up 10%.
Duncan RoeSoftware Developer
CERTIFIED EXPERT

Commented:
It is possible that the reserved space for root has been set way too high. This would lead to what you observe - I think df always reports % available to normal users. Use tune2fs to tell you if that is the case.
CERTIFIED EXPERT
Top Expert 2007

Commented:
after rebooting system will go to default run level which multi-user by default.

Author

Commented:
I use the root login and I think the server is running fedora, if that would be of any help.

omarfarid: The 'fsck -y' command gives back "-bash: fsck: command not found".

hfraser: The 'lsof | grep deleted' command gives back:

httpd     12043   root   35w   REG       0,28        0  271774316 (deleted) /var/log/httpd/ssl_mutex.12033
httpd     12162 apache   35w   REG       0,28        0  271774316 (deleted) /var/log/httpd/ssl_mutex.12033
httpd     12163 apache   35w   REG       0,28        0  271774316 (deleted) /var/log/httpd/ssl_mutex.12033

duncan_roe: The 'tune2fs' gives back "-bash: tune2fs: command not found"
So you don't even have the very basic tools for administration. Install them as soon as possible,or at the first crash you won't even be able to boot :/

Stop all the services and remove everything in /tmp then see if df reports some changes
Hugh FraserConsultant

Commented:
As Duncan_roe points out, there is about 5% of the disk reserved for root, to ensure you can still boot and log in as root even if the rest of the disk is filled. If you do a "df", you'll notice the sum of used and available doesn't add up to 1-K blocks column. That's the reserved part.

It's possible to delete files and still have the "Use%" column show up as 100% because of rounding of the value (ie. 99.6% used shows as 100%). Keep in mind that the "Use%" column includes the 5% reserved component.

Even so, freeing up some space on the volumes (I assume you've freed up some on / and /tmp) should allow Apache to start. Can you post the error message it returns?

Author

Commented:
hfraser: Apache starts up and for now it even keeps running. The problem is that the disk is full while it shouldn't be.

ai_ja_nai: I have removed everything in /tmp and /var/tmp but I don't know how to shut down all the services (and how to restart them). df report is still the same.
# Check if a service is running:
service servicename status

# Starting a service:
service servicename start

# Stopping a service:
service servicename stop

stop apache, mysql and other things running. install fsck to perform a... scandisk of your filesystem, in case the reported size is because of corruption
Duncan RoeSoftware Developer
CERTIFIED EXPERT

Commented:
Possibly your bash can't find tune2fs, fsck &c because your PATH is deficient. Try:
locate fsck
/sbin/fsck
/usr/sbin/fsck
Commented:
Damnit, sorry for not answering sooner, have had 3 really long days. The friend I was helping with this problem called and said everything was working again, I still have no clue what the cause was but thanks for all you're efforts. Everything is working fine again.

Not the solution you were looking for? Getting a personalized solution is easy.

Ask the Experts
Access more of Experts Exchange with a free account
Thanks for using Experts Exchange.

Create a free account to continue.

Limited access with a free account allows you to:

  • View three pieces of content (articles, solutions, posts, and videos)
  • Ask the experts questions (counted toward content limit)
  • Customize your dashboard and profile

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.