It is an honor to be featured in Gartner 2019 Magic Quadrant for Datacenter Backup and Recovery Solutions. Gartner’s MQ sets a high standard and earning a place on their grid is a great affirmation that Acronis is delivering on our mission to protect all data, apps, and systems.
can't find why my / file system is 100% full on a DC/file server
Hi,
My server's config
DELL PE 2970 : 4GB RAM,RAID 1 with 2x500GB sata HDD
1 primary partition /boot - 250MB - EXT2
1 logical volume - LVM - 495GB
The logical volume is partitioned as the following :
/var 31GB - EXT3
/home 250 GB - EXT3
/ 50 GB - EXT3
swap 12GB
/usr - 30GB - EXT3
/tmp - 2GB - EXT3
remain free around 120GB
--------------------------
Last week when checking the disk space usage on the server, I found the / file system is full
root@gege:/# df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/gege-root
46G 45G 0 100% /
tmpfs 2,0G 0 2,0G 0% /lib/init/rw
varrun 2,0G 1,5M 2,0G 1% /var/run
varlock 2,0G 0 2,0G 0% /var/lock
udev 2,0G 160K 2,0G 1% /dev
tmpfs 2,0G 0 2,0G 0% /dev/shm
lrm 2,0G 2,5M 2,0G 1% /lib/modules/2.6.28-19-ser
/dev/sda1 236M 34M 190M 16% /boot
/dev/mapper/gege-home
231G 84G 135G 39% /home
/dev/mapper/gege-temp
1,9G 35M 1,8G 2% /tmp
/dev/mapper/gege-user
29G 1,2G 26G 5% /usr
/dev/mapper/gege-var 29G 2,6G 25G 10% /var
--------------------------
To find which file or directory that is consuming all that space disk I tried the following on / but nothing in the result can help.
root@gege:/# du -kshc * |sort -n
0 cdrom
0 initrd.img
0 initrd.img.old
0 lib64
0 proc
0 sys
0 vmlinuz
0 vmlinuz.old
4,0K mnt
4,0K opt
4,0K selinux
4,0K srv
6,9M etc
7,2M bin
8,0K media
8,2M sbin
16K lost+found
16K tmp
28M boot
38M root
84G home
87G total
160K dev
240M lib
467M var
963M usr
--------------------------
If someone has any idea on what happens I will really appreciate.
Many
thanks in advance.
My server's config
DELL PE 2970 : 4GB RAM,RAID 1 with 2x500GB sata HDD
1 primary partition /boot - 250MB - EXT2
1 logical volume - LVM - 495GB
The logical volume is partitioned as the following :
/var 31GB - EXT3
/home 250 GB - EXT3
/ 50 GB - EXT3
swap 12GB
/usr - 30GB - EXT3
/tmp - 2GB - EXT3
remain free around 120GB
--------------------------
Last week when checking the disk space usage on the server, I found the / file system is full
root@gege:/# df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/gege-root
46G 45G 0 100% /
tmpfs 2,0G 0 2,0G 0% /lib/init/rw
varrun 2,0G 1,5M 2,0G 1% /var/run
varlock 2,0G 0 2,0G 0% /var/lock
udev 2,0G 160K 2,0G 1% /dev
tmpfs 2,0G 0 2,0G 0% /dev/shm
lrm 2,0G 2,5M 2,0G 1% /lib/modules/2.6.28-19-ser
/dev/sda1 236M 34M 190M 16% /boot
/dev/mapper/gege-home
231G 84G 135G 39% /home
/dev/mapper/gege-temp
1,9G 35M 1,8G 2% /tmp
/dev/mapper/gege-user
29G 1,2G 26G 5% /usr
/dev/mapper/gege-var 29G 2,6G 25G 10% /var
--------------------------
To find which file or directory that is consuming all that space disk I tried the following on / but nothing in the result can help.
root@gege:/# du -kshc * |sort -n
0 cdrom
0 initrd.img
0 initrd.img.old
0 lib64
0 proc
0 sys
0 vmlinuz
0 vmlinuz.old
4,0K mnt
4,0K opt
4,0K selinux
4,0K srv
6,9M etc
7,2M bin
8,0K media
8,2M sbin
16K lost+found
16K tmp
28M boot
38M root
84G home
87G total
160K dev
240M lib
467M var
963M usr
--------------------------
If someone has any idea on what happens I will really appreciate.
Many
thanks in advance.
Do more with
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
If you want to check the existence of some very large files you can use find command to do that
find / -type f -size +100000k
You can change the sizes accordingly and see if you find some big files.
find / -type f -size +100000k
You can change the sizes accordingly and see if you find some big files.
This will give you usefui info on the top 10 files as well as who owns them and date created. no need to guess at a size
find / -type f -exec ls -ls {} \; | sort -n -r | head -10 | awk ' { print $10" " $4" " $6" "$7" "$8}'
find / -type f -exec ls -ls {} \; | sort -n -r | head -10 | awk ' { print $10" " $4" " $6" "$7" "$8}'
Try to restrict the size with -size +xxxG something like this otherwise it would not be efficient trying to list all files on the system and then sorting them all, etc.
I did try all that kind of search but no retrurn with a huge file.
The point is that file system like /home, /tmp, /usr, /tmp got their own partition.
on / partition we can find :/lib, /dev, /root, /lost+found, /sbin, /media,/bin, /etc, /srv, /sys,/proc and the total of all give around 300MB but df command still return that / file system is full.
The point is that file system like /home, /tmp, /usr, /tmp got their own partition.
on / partition we can find :/lib, /dev, /root, /lost+found, /sbin, /media,/bin, /etc, /srv, /sys,/proc and the total of all give around 300MB but df command still return that / file system is full.
This usually meant there were some runaway process/es which used up the / space. Only way to fix this is a "reboot".
I guess that under your mounted partition are hidden many dir and files (may be that the system was not "cleany" transported from a disk to another disk or similar operation).
try this :
1) as root go in single user mode
# init 1
2) umount your mounted files system
# umount /home
# umount /var
# umount /usr
3) check now the mount point that if all is correct may be empty
# ls -la /home
total 16
drwxr-xr-x 2 root root 4096 May 17 09:50 .
drwxrwxrwt 14 root root 12288 May 17 09:50 ..
#ls -la /var
total 16
drwxr-xr-x 2 root root 4096 May 17 09:50 .
drwxrwxrwt 14 root root 12288 May 17 09:50 ..
#ls -la /usr
total 16
drwxr-xr-x 2 root root 4096 May 17 09:50 .
drwxrwxrwt 14 root root 12288 May 17 09:50 ..
4) if you see other files and dir (assure that are surely umounted and check with
the mount command) than my guess was right. Then clean the mount point.
# rm -r /home/*
or
# rm -r /usr/*
or
# rm -r /var/*
guess 2: if it's not the case 3 (and all the mount point are correctly empty) then
5) already in single user, try to do a forced check of your / filesystem. First remount the / in readonly with
# mount -o remount,ro /dev/mapper/gege-root /
# fsck.ext3 -y /dev/mapper/gege-root
6) restart your system and check
try this :
1) as root go in single user mode
# init 1
2) umount your mounted files system
# umount /home
# umount /var
# umount /usr
3) check now the mount point that if all is correct may be empty
# ls -la /home
total 16
drwxr-xr-x 2 root root 4096 May 17 09:50 .
drwxrwxrwt 14 root root 12288 May 17 09:50 ..
#ls -la /var
total 16
drwxr-xr-x 2 root root 4096 May 17 09:50 .
drwxrwxrwt 14 root root 12288 May 17 09:50 ..
#ls -la /usr
total 16
drwxr-xr-x 2 root root 4096 May 17 09:50 .
drwxrwxrwt 14 root root 12288 May 17 09:50 ..
4) if you see other files and dir (assure that are surely umounted and check with
the mount command) than my guess was right. Then clean the mount point.
# rm -r /home/*
or
# rm -r /usr/*
or
# rm -r /var/*
guess 2: if it's not the case 3 (and all the mount point are correctly empty) then
5) already in single user, try to do a forced check of your / filesystem. First remount the / in readonly with
# mount -o remount,ro /dev/mapper/gege-root /
# fsck.ext3 -y /dev/mapper/gege-root
6) restart your system and check
Do you have also tryed the first guess ? Do you have checked under the mount point if the directory are empty ?
please run also this command (as root) and give me the result
sfdisk -l
No I did not try the first gess
Here is the result for the command sfdisk -l
root@gege:~# sfdisk -l
Disk /dev/sda: 60734 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sda1 * 0+ 31- 31- 248832 83 Linux
/dev/sda2 31+ 60734- 60704- 487598080+ 5 Extended
/dev/sda3 0 - 0 0 0 Empty
/dev/sda4 0 - 0 0 0 Empty
/dev/sda5 31+ 60734- 60704- 487598080 8e Linux LVM
Here is the result for the command sfdisk -l
root@gege:~# sfdisk -l
Disk /dev/sda: 60734 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sda1 * 0+ 31- 31- 248832 83 Linux
/dev/sda2 31+ 60734- 60704- 487598080+ 5 Extended
/dev/sda3 0 - 0 0 0 Empty
/dev/sda4 0 - 0 0 0 Empty
/dev/sda5 31+ 60734- 60704- 487598080 8e Linux LVM
Premium Content
You need an Expert Office subscription to comment.Start Free Trial