can't find why my / file system is 100% full on a DC/file server

samkeen
samkeen used Ask the Experts™
on
Hi,

My server's config

DELL PE 2970 : 4GB RAM,RAID 1 with 2x500GB sata HDD
1 primary partition /boot - 250MB - EXT2
1 logical volume - LVM - 495GB

The logical volume is partitioned as the following :
/var 31GB - EXT3
/home 250 GB - EXT3
/ 50 GB - EXT3
swap 12GB
/usr - 30GB - EXT3
/tmp - 2GB - EXT3

remain free around 120GB
-------------------------------------------------------------------------------------------------
Last week when checking the disk space usage on the server, I found the  / file system is full

root@gege:/# df -kh

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/gege-root

                                        46G   45G     0 100% /

tmpfs                               2,0G     0  2,0G   0% /lib/init/rw

varrun                             2,0G  1,5M  2,0G   1% /var/run

varlock                           2,0G     0  2,0G   0% /var/lock

udev                               2,0G  160K  2,0G   1% /dev

tmpfs                              2,0G     0  2,0G   0% /dev/shm

lrm                                  2,0G  2,5M  2,0G   1% /lib/modules/2.6.28-19-server/volatile

/dev/sda1                       236M   34M  190M  16% /boot

/dev/mapper/gege-home

                                       231G   84G  135G  39% /home

/dev/mapper/gege-temp

                                       1,9G   35M  1,8G   2% /tmp

/dev/mapper/gege-user

                                       29G  1,2G   26G   5% /usr

/dev/mapper/gege-var   29G  2,6G   25G  10% /var
---------------------------------------------------------------------------------------------------
To find which file or directory that is consuming all that space disk I tried  the following on / but nothing in the result can help.

root@gege:/# du -kshc * |sort -n



0      cdrom

0      initrd.img

0      initrd.img.old

0      lib64

0      proc

0      sys

0      vmlinuz

0      vmlinuz.old


4,0K        mnt

4,0K       opt

4,0K       selinux

4,0K               srv

6,9M       etc

7,2M      bin

8,0K       media

8,2M      sbin

16K              lost+found

16K         tmp

28M       boot

38M              root

84G              home

87G              total

160K      dev

240M      lib

467M      var
963M      usr

------------------------------------------------------------------------------------------------
If someone has any idea on what happens I will really appreciate.

Many
thanks in advance.

Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
If you want to check the existence of some very large files you can use find command to do that

find / -type f -size +100000k


You can change the sizes accordingly and see if you find some big files.
This will give you usefui info on the top 10 files as well as who owns them and date created.  no need to guess at a size


 find / -type f -exec ls -ls {} \; | sort -n -r | head -10 | awk ' { print $10" " $4" " $6" "$7" "$8}'
Or something like
find / -type f -xdev -size 10G
Acronis in Gartner 2019 MQ for datacenter backup

It is an honor to be featured in Gartner 2019 Magic Quadrant for Datacenter Backup and Recovery Solutions. Gartner’s MQ sets a high standard and earning a place on their grid is a great affirmation that Acronis is delivering on our mission to protect all data, apps, and systems.

Make sure to use -xdev because you would want to stay on the same file system.
Try to restrict the size with -size +xxxG something like this otherwise it would not be efficient trying to list all files on the system and then sorting them all, etc.

Author

Commented:
I did try all that kind of search but no retrurn with a huge file.

The point is that file system like /home, /tmp, /usr, /tmp got their own partition.
on / partition we can find :/lib, /dev, /root, /lost+found, /sbin, /media,/bin, /etc, /srv, /sys,/proc and the total of all give around 300MB but df command still return that / file system is full.

Commented:
This usually meant there were some runaway process/es which used up the / space. Only way to fix this is a "reboot".
I guess that under your mounted partition are hidden many dir and files (may be that the system was not "cleany" transported from a disk to another disk or similar operation).

try this :
1) as root go in single user mode
# init 1
2) umount your mounted files system
# umount /home
# umount /var
# umount /usr
3) check now the mount point that if all is correct may be empty
# ls -la /home
total 16
drwxr-xr-x  2 root root  4096 May 17 09:50 .
drwxrwxrwt 14 root root 12288 May 17 09:50 ..
#ls -la /var
total 16
drwxr-xr-x  2 root root  4096 May 17 09:50 .
drwxrwxrwt 14 root root 12288 May 17 09:50 ..
#ls -la /usr
total 16
drwxr-xr-x  2 root root  4096 May 17 09:50 .
drwxrwxrwt 14 root root 12288 May 17 09:50 ..
4) if you see other files and dir (assure that are surely umounted and check with
the mount command) than my guess was right. Then clean the mount point.
# rm -r /home/*
or
# rm -r /usr/*
or
# rm -r /var/*


guess 2: if it's not the case 3 (and all the mount point are correctly empty) then
5) already in single user, try to do a forced check of your / filesystem. First remount the / in readonly with
# mount -o remount,ro /dev/mapper/gege-root /
# fsck.ext3 -y /dev/mapper/gege-root
6) restart your system and check




Author

Commented:
Hello I have tried guess 2 and it still the same and I still working on

Do you have also tryed the first guess ? Do you have checked under the mount point if the directory are empty ?
please run also this command (as root) and give me the result
sfdisk -l


Author

Commented:
No I did not try the first gess

Here is the result for the command sfdisk -l
 
root@gege:~# sfdisk -l

Disk /dev/sda: 60734 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1   *      0+     31-     31-    248832   83  Linux
/dev/sda2         31+  60734-  60704- 487598080+   5  Extended
/dev/sda3          0       -       0          0    0  Empty
/dev/sda4          0       -       0          0    0  Empty
/dev/sda5         31+  60734-  60704- 487598080   8e  Linux LVM


please try as i've suggested from step 1 to 4 ... at the moment it's the most probable situation

Author

Commented:
I did not try the first guess because I did not moved the system from a disk to another disk .
So I will try it and let you know

Author

Commented:
Ok stetor it works, I have try your fisrt guess and it 's OK

thanks

it can't be nothing else ...

well, pleased to be helpful ;-)

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial