?
Solved

Always 100% disk use while disk is not full

Posted on 2009-02-14
13
Medium Priority
?
2,057 Views
Last Modified: 2012-06-27
A friend of mine asked me to look at his server because after a crash it seemed that the websites hosted on the server would not be available (couldn't find server). After some searching I'm fairly sure that Apache just crashes because if I reboot Apache it works again for a little while. I searched google for some answers and found one but I couldn't do anything because the server keeps telling me there is no disk space left. After deleting a few 100MB I still got the message there is no disk space left. I'm not that great with linux so I googled some more to find out the command to check disk space, I used this:

df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/vzfs             2.0G  633M     0 100% /
simfs                 2.0G  633M     0 100% /tmp
simfs                 2.0G  633M     0 100% /var/tmp

Nothing I do seems to fix this, the Use is always 100%. When I delete files the 'Used' amount decreases but the 'Use' perentage stays 100%. So it seems that linux thinks the server has no space left but that's not true. What can it be?
0
Comment
Question by:bt33
  • 4
  • 3
  • 2
  • +2
13 Comments
 
LVL 16

Expert Comment

by:ai_ja_nai
ID: 23641633
Are you on Virtuozzo, or OpenVZ?
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 23641859
can you take system to single user mode (this will make it unavailable on the network till you take it to multi user mode again)? You may run then

init s
fsck -y

then reboot the system :

reboot
0
 

Author Comment

by:bt33
ID: 23642148
ai_ja_nai: I log into the server through Secure Shell Client, I'm not sure what you mean by Virtuozzo or OpenVZ, sorry.

omarfarid: I'm not behind my own PC now so can't test what you recommended but I will try when I get home (might be tomorrow). When I reboot the server do I need to put it back in multi user again?
0
Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

 
LVL 12

Expert Comment

by:Hugh Fraser
ID: 23642701
Deleting files that are currently open does not release the space until the files are closed. You can check for processes with open files using "lsof | grep deleted", checking for processes holding open the files you've deleted.

This doesn't however, answer why the "%" value remains the same, especially if you've removed 200M on a 2G volume, which should free up 10%.
0
 
LVL 35

Expert Comment

by:Duncan Roe
ID: 23642931
It is possible that the reserved space for root has been set way too high. This would lead to what you observe - I think df always reports % available to normal users. Use tune2fs to tell you if that is the case.
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 23643373
after rebooting system will go to default run level which multi-user by default.
0
 

Author Comment

by:bt33
ID: 23644738
I use the root login and I think the server is running fedora, if that would be of any help.

omarfarid: The 'fsck -y' command gives back "-bash: fsck: command not found".

hfraser: The 'lsof | grep deleted' command gives back:

httpd     12043   root   35w   REG       0,28        0  271774316 (deleted) /var/log/httpd/ssl_mutex.12033
httpd     12162 apache   35w   REG       0,28        0  271774316 (deleted) /var/log/httpd/ssl_mutex.12033
httpd     12163 apache   35w   REG       0,28        0  271774316 (deleted) /var/log/httpd/ssl_mutex.12033

duncan_roe: The 'tune2fs' gives back "-bash: tune2fs: command not found"
0
 
LVL 16

Expert Comment

by:ai_ja_nai
ID: 23644875
So you don't even have the very basic tools for administration. Install them as soon as possible,or at the first crash you won't even be able to boot :/

Stop all the services and remove everything in /tmp then see if df reports some changes
0
 
LVL 12

Expert Comment

by:Hugh Fraser
ID: 23645269
As Duncan_roe points out, there is about 5% of the disk reserved for root, to ensure you can still boot and log in as root even if the rest of the disk is filled. If you do a "df", you'll notice the sum of used and available doesn't add up to 1-K blocks column. That's the reserved part.

It's possible to delete files and still have the "Use%" column show up as 100% because of rounding of the value (ie. 99.6% used shows as 100%). Keep in mind that the "Use%" column includes the 5% reserved component.

Even so, freeing up some space on the volumes (I assume you've freed up some on / and /tmp) should allow Apache to start. Can you post the error message it returns?
0
 

Author Comment

by:bt33
ID: 23645527
hfraser: Apache starts up and for now it even keeps running. The problem is that the disk is full while it shouldn't be.

ai_ja_nai: I have removed everything in /tmp and /var/tmp but I don't know how to shut down all the services (and how to restart them). df report is still the same.
0
 
LVL 16

Expert Comment

by:ai_ja_nai
ID: 23645594
# Check if a service is running:
service servicename status

# Starting a service:
service servicename start

# Stopping a service:
service servicename stop

stop apache, mysql and other things running. install fsck to perform a... scandisk of your filesystem, in case the reported size is because of corruption
0
 
LVL 35

Expert Comment

by:Duncan Roe
ID: 23646389
Possibly your bash can't find tune2fs, fsck &c because your PATH is deficient. Try:
locate fsck
/sbin/fsck
/usr/sbin/fsck
0
 

Accepted Solution

by:
bt33 earned 0 total points
ID: 23672931
Damnit, sorry for not answering sooner, have had 3 really long days. The friend I was helping with this problem called and said everything was working again, I still have no clue what the cause was but thanks for all you're efforts. Everything is working fine again.
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction This article is intended for those who are new to PHP error handling (https://www.experts-exchange.com/articles/11769/And-by-the-way-I-am-New-to-PHP.html).  It addresses one of the most common problems that plague beginning PHP develop…
The title says it all. Writing any type of PHP Application or API code that provides high throughput, while under a heavy load, seems to be an arcane art form (Black Magic). This article aims to provide some general guidelines for producing this typ…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
Suggested Courses
Course of the Month16 days, 9 hours left to enroll

864 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question