Linux show's 138 GB of 145 Used but 0% disk space avilable

I recently had a linux server where I ran out of disk space and wasn't able to get any services started. It's onthe LVM2 filesystem... I fixed the previous problems and was able to dump a few of the databases. However, now I've deleted more files and got over 7GB free. However, it says that there is 0% available. I have 138 out of 145 used but I don't have any disk space available.

Everytime I delete files it shows less being used but still none available.

I researched around and people told me it's because services are using the files etc etc... So I killed all the services and still nothing. so then I booted into a rescue CD and tried to delete files that way. But still nothing frees up.

Any suggestions or what I'm doing wrong?

I'm deleting folders with the rm -rf command.

Thanks a lot in advance.
Who is Participating?
ReN501Connect With a Mentor Commented:
yeah , i'd say your into overhead territory.... basicall boot as single user , goto / and type "du -h"
minus the "" , this will give you the folders and folders sizes , navigate though to find whats taking up all the space and goto town , you need to free up space. its pretty much that simple.
What output do the following commands produce?
Did you run fsck ?
A proven path to a career in data science

At Springboard, we know how to get you a job in data science. With Springboard’s Data Science Career Track, you’ll master data science  with a curriculum built by industry experts. You’ll work on real projects, and get 1-on-1 mentorship from a data scientist.

It is the reserved space for root (by default 5%) in ext2 and ext3 volumes. For detailed information see explanation of -m  in man mke2fs

To check  reserved space"
tune2fs -l /dev/sda3
Block count:              4194973
Reserved block count:     209748
indeed ,  both the above experts are correct, to fre up some space temporarily , if your system isn't using the swap file , you can disable it for a quick fix ( although it is strongly recommaned if you dont have ram in your machine to support such a move , not to ) log files are bad for space also , deleting old or outdated logs or even moving them temporarily would also be a temp fix until you sort your space requirements out.
keith_opswatAuthor Commented:
I can't delete any files. And even though there are several gb free there is no usable space. I can't delete any files and Ive tried adding space to the volgroup through an external drive but it doesn't work. I can add space to boot but not / in the volume group. I've even booted into single user mode, rescue mode as well as some other bootable drive utilities. I've created a new swap file... I've added space to the existing swap file as well as disabled the swap file. I can delete files and it shows the used space getting lower, but the free space still stays at zero even though I've deleted several GB of data.

I have 300mb of memory free, 800mb free cache, and over 2gb worth of swap space free. I did a free -k command for that.

Df -h returns

[root@database botv2]# df -h
Filesystem                                             Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00     145G  144G     0 100% /
/dev/sda1                                              927M   13M  866M   2% /boot
none                                                      501M     0  501M   0% /dev/shm
/dev/hda                                                 76M   76M     0 100% /media/cdrom

A few days ago I had the Used space down to 137GB and still 0 available.

Any suggestions after receiving all this info?? I apologize for such a long delay in my response but I had a bunch of things come up. I was able to retrieve all the most vital information off this server but when I put it back up for everyone else to get files they need they started using the damn thing again and filled the space all the way up. Now I can't get any free space or anything...

Thanks for your time.
keith_opswatAuthor Commented:
Yeh that's what I've been doing.. Going through with du -h --max-depth=1 and finding what I can get rid of. But like I said even though I was deleting like 10GB of stuff it never showed anymore free space.

So you're telling me Single User Mode should fix this? I will try it here in the next little bit and let you know how that works out for me.

Getting what I was able too off this server is amazing enough because originally even the Rescue CD couldn't detect a linux partition on the drive. I was able to completely rebuild the partitions and the filesystem with minimal loss... Just have had this issue because I have people here at work bugging me cause they want more stuff off there. EVen though the only reason the server filled up is because they did what I said not to do... And that was to run more Bot tests on our new software releases and fill up the drive with test results.

But thanks again... I hope your solution is the answer. Be back in a bit...
i'm saying linux using drive space really efficeiantly , 10gig might not be enough , backup save or delete more
keith_opswatAuthor Commented:
Nothing has worked... I'm done messing with this and closing. I appreciate all your attempts to help... This Linux server has been through a lot. It's possible when the filesystem went to hell and I rebuilt it something weird happened. I'm sure all your guys' info was correct, just wasn't gonna work on this one!!
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.