Link to home
Start Free TrialLog in
Avatar of samkeen
samkeen

asked on

can't find why my / file system is 100% full on a DC/file server

Hi,

My server's config

DELL PE 2970 : 4GB RAM,RAID 1 with 2x500GB sata HDD
1 primary partition /boot - 250MB - EXT2
1 logical volume - LVM - 495GB

The logical volume is partitioned as the following :
/var 31GB - EXT3
/home 250 GB - EXT3
/ 50 GB - EXT3
swap 12GB
/usr - 30GB - EXT3
/tmp - 2GB - EXT3

remain free around 120GB
-------------------------------------------------------------------------------------------------
Last week when checking the disk space usage on the server, I found the  / file system is full

root@gege:/# df -kh

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/gege-root

                                        46G   45G     0 100% /

tmpfs                               2,0G     0  2,0G   0% /lib/init/rw

varrun                             2,0G  1,5M  2,0G   1% /var/run

varlock                           2,0G     0  2,0G   0% /var/lock

udev                               2,0G  160K  2,0G   1% /dev

tmpfs                              2,0G     0  2,0G   0% /dev/shm

lrm                                  2,0G  2,5M  2,0G   1% /lib/modules/2.6.28-19-server/volatile

/dev/sda1                       236M   34M  190M  16% /boot

/dev/mapper/gege-home

                                       231G   84G  135G  39% /home

/dev/mapper/gege-temp

                                       1,9G   35M  1,8G   2% /tmp

/dev/mapper/gege-user

                                       29G  1,2G   26G   5% /usr

/dev/mapper/gege-var   29G  2,6G   25G  10% /var
---------------------------------------------------------------------------------------------------
To find which file or directory that is consuming all that space disk I tried  the following on / but nothing in the result can help.

root@gege:/# du -kshc * |sort -n



0      cdrom

0      initrd.img

0      initrd.img.old

0      lib64

0      proc

0      sys

0      vmlinuz

0      vmlinuz.old


4,0K        mnt

4,0K       opt

4,0K       selinux

4,0K               srv

6,9M       etc

7,2M      bin

8,0K       media

8,2M      sbin

16K              lost+found

16K         tmp

28M       boot

38M              root

84G              home

87G              total

160K      dev

240M      lib

467M      var
963M      usr

------------------------------------------------------------------------------------------------
If someone has any idea on what happens I will really appreciate.

Many
thanks in advance.

Avatar of farzanj
farzanj
Flag of Canada image

If you want to check the existence of some very large files you can use find command to do that

find / -type f -size +100000k


You can change the sizes accordingly and see if you find some big files.
This will give you usefui info on the top 10 files as well as who owns them and date created.  no need to guess at a size


 find / -type f -exec ls -ls {} \; | sort -n -r | head -10 | awk ' { print $10" " $4" " $6" "$7" "$8}'
Or something like
find / -type f -xdev -size 10G
Make sure to use -xdev because you would want to stay on the same file system.
Try to restrict the size with -size +xxxG something like this otherwise it would not be efficient trying to list all files on the system and then sorting them all, etc.
Avatar of samkeen
samkeen

ASKER

I did try all that kind of search but no retrurn with a huge file.

The point is that file system like /home, /tmp, /usr, /tmp got their own partition.
on / partition we can find :/lib, /dev, /root, /lost+found, /sbin, /media,/bin, /etc, /srv, /sys,/proc and the total of all give around 300MB but df command still return that / file system is full.
This usually meant there were some runaway process/es which used up the / space. Only way to fix this is a "reboot".
ASKER CERTIFIED SOLUTION
Avatar of stetor
stetor
Flag of Italy image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of samkeen

ASKER

Hello I have tried guess 2 and it still the same and I still working on

Do you have also tryed the first guess ? Do you have checked under the mount point if the directory are empty ?
please run also this command (as root) and give me the result
sfdisk -l


Avatar of samkeen

ASKER

No I did not try the first gess

Here is the result for the command sfdisk -l
 
root@gege:~# sfdisk -l

Disk /dev/sda: 60734 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1   *      0+     31-     31-    248832   83  Linux
/dev/sda2         31+  60734-  60704- 487598080+   5  Extended
/dev/sda3          0       -       0          0    0  Empty
/dev/sda4          0       -       0          0    0  Empty
/dev/sda5         31+  60734-  60704- 487598080   8e  Linux LVM


please try as i've suggested from step 1 to 4 ... at the moment it's the most probable situation
Avatar of samkeen

ASKER

I did not try the first guess because I did not moved the system from a disk to another disk .
So I will try it and let you know
Avatar of samkeen

ASKER

Ok stetor it works, I have try your fisrt guess and it 's OK

thanks

it can't be nothing else ...

well, pleased to be helpful ;-)