[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 374
  • Last Modified:

Centos disk space doesn't add up

Hi,

I have a small VPS that I use as a development server, it's running Centos and Cpanel.

The problem I'm having is that I'm quickly running out of disk space but I can't seem to figure out where it's all going. I've ran the du -sh * command to see which folders are taking up space on the server but the numbers being returned don't add up to the amount of disk space that's being used.

root@server [/home]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       38G   34G  1.9G  95% /
tmpfs           497M     0  497M   0% /dev/shm
/usr/tmpDSK     4.0G  139M  3.7G   4% /tmp

Open in new window


du -sh * on the root

4.0K	backup
7.0M	bin
37M	boot
124K	dev
28K	error_log
26M	etc
34G	home
224M	lib
23M	lib64
16K	lost+found
4.0K	media
4.0K	mnt
41M	opt
0	proc
16K	quota.user
4.0K	razor-agent.log
173M	root
9.3M	sbin
0	scripts
4.0K	selinux
4.0K	srv
0	sys
2.5M	tmp
8.3G	usr
4.1G	var

Open in new window


du -sh on the /home folder (which also contains a virtfs folder)

4.0K	0_README_BEFORE_DELETING_VIRTFS
61M	backup-12.28.2012_11-23-31_deskdraw.tar.gz
653M	cdevsvr
69M	cds
583M	cmsdev
16K	cpbackuptmp
331M	cpeasyapache
4.0K	cprestore
5.5M	deskapp
121M	digcms
891M	durham
55M	hutton
156M	installd
63M	inthols
35M	kgibbon
32K	latest
431M	libraifa
85M	mg
19M	oplog
12M	oplogdev
6.3M	postcode
189M	sheppard
497M	smem
71M	sr8
107M	tacs
339M	tagdoors
28G	virtfs
1.6G	webdev
190M	xcloud

Open in new window


By my calculations the following folders are taking up most of the space....

/usr = 8.3GB
/var = 4.1GB
/home = 6.5GB (I'm not counting the virtfs folder as I believe this is just a link to other folders)
That adds up to around 19GB, so why does the OS say I'm using more than that?
0
SheppardDigital
Asked:
SheppardDigital
  • 6
  • 4
  • 4
1 Solution
 
gheistCommented:
lsof +L1
will show you deleted files still open by applications.
0
 
SheppardDigitalAuthor Commented:
There doesn't seem to be anything showing that's large

COMMAND     PID     USER   FD   TYPE DEVICE SIZE/OFF NLINK    NODE NAME
mysqld     1337    mysql    4u   REG    7,0        0     0     228 /tmp/ibY2RlUs (deleted)
mysqld     1337    mysql    5u   REG    7,0        0     0     230 /tmp/ibwQH7Cg (deleted)
mysqld     1337    mysql    6u   REG    7,0        0     0     231 /tmp/iba3Y3l4 (deleted)
mysqld     1337    mysql    7u   REG    7,0        0     0     232 /tmp/ibh64RXF (deleted)
mysqld     1337    mysql   11u   REG    7,0        0     0     235 /tmp/ibyh3dlu (deleted)
dovecot    1466     root  111u   REG  202,1        0     0 3458261 /var/run/dovecot/login-master-notify6e3294739088d79c (deleted)
dovecot    1466     root  128u   REG  202,1        0     0 3458264 /var/run/dovecot/login-master-notifybd7a590ada995cdf (deleted)
dovecot    1466     root  131u   REG  202,1        0     0 3458265 /var/run/dovecot/login-master-notifyf255e36aa9280017 (deleted)
pop3-logi  1475 dovenull    4u   REG  202,1        0     0 3458261 /var/run/dovecot/login-master-notify6e3294739088d79c (deleted)
imap-logi  1476 dovenull    4u   REG  202,1        0     0 3458265 /var/run/dovecot/login-master-notifyf255e36aa9280017 (deleted)
pop3-logi  1480 dovenull    4u   REG  202,1        0     0 3458261 /var/run/dovecot/login-master-notify6e3294739088d79c (deleted)
imap-logi  1481 dovenull    4u   REG  202,1        0     0 3458265 /var/run/dovecot/login-master-notifyf255e36aa9280017 (deleted)
httpd      1490     root  140w   REG  202,1        0     0 4113078 /usr/local/apache/logs/ssl_mutex (deleted)
httpd      1500   nobody  140w   REG  202,1        0     0 4113078 /usr/local/apache/logs/ssl_mutex (deleted)
httpd      6351   nobody  140w   REG  202,1        0     0 4113078 /usr/local/apache/logs/ssl_mutex (deleted)
httpd      7911   nobody  140w   REG  202,1        0     0 4113078 /usr/local/apache/logs/ssl_mutex (deleted)
httpd      7915   nobody  140w   REG  202,1        0     0 4113078 /usr/local/apache/logs/ssl_mutex (deleted)
httpd      9473   nobody  140w   REG  202,1        0     0 4113078 /usr/local/apache/logs/ssl_mutex (deleted)
httpd      9477   nobody  140w   REG  202,1        0     0 4113078 /usr/local/apache/logs/ssl_mutex (deleted)
httpd     11026   nobody  140w   REG  202,1        0     0 4113078 /usr/local/apache/logs/ssl_mutex (deleted)
httpd     20697   nobody  140w   REG  202,1        0     0 4113078 /usr/local/apache/logs/ssl_mutex (deleted)
httpd     20698   nobody  140w   REG  202,1        0     0 4113078 /usr/local/apache/logs/ssl_mutex (deleted)
httpd     21023   nobody  140w   REG  202,1        0     0 4113078 /usr/local/apache/logs/ssl_mutex (deleted)
httpd     21026   nobody  140w   REG  202,1        0     0 4113078 /usr/local/apache/logs/ssl_mutex (deleted)

Open in new window

0
 
gheistCommented:
Do you have some non-standard log cleanup script deleting files in /tmp ?
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
SheppardDigitalAuthor Commented:
I don't believe so no.
0
 
gheistCommented:
/usr is big. I have <2GB there on headless server.
Do you have heaps of useless packages installed? i.e can I see "rpm -qa" and "pstree" output as attachment?
0
 
SheppardDigitalAuthor Commented:
pstree output

init─┬─/usr/local/cpan───spamd\040child
     ├─agetty
     ├─atd
     ├─auditd───{auditd}
     ├─cPhulkd\040-\040proce
     ├─cpanellogd\040-\040sl
     ├─cpdavd\040-\040accept
     ├─cpsrvd-ssl
     ├─crond
     ├─dbus-daemon
     ├─dovecot─┬─anvil
     │         ├─config
     │         ├─2*[imap-login]
     │         ├─log
     │         ├─2*[pop3-login]
     │         └─ssl-params
     ├─exim
     ├─httpd─┬─11*[httpd]
     │       └─leechprotect
     ├─6*[mingetty]
     ├─mysqld_safe───mysqld───31*[{mysqld}]
     ├─named───6*[{named}]
     ├─nova-agent───{nova-agent}
     ├─nrsysmond───nrsysmond───{nrsysmond}
     ├─pure-authd
     ├─pure-ftpd
     ├─queueprocd\040-\040wa
     ├─rackspace-monit───4*[{rackspace-moni}]
     ├─rsyslogd───3*[{rsyslogd}]
     ├─sshd───sshd───bash───pstree
     ├─tailwatchd
     ├─udevd───2*[udevd]
     └─xe-daemon───sleep

Open in new window


rpm -qa attached
rpmqa.txt
0
 
gheistCommented:
cpanel is very prevalent in that rpm -qa with full beef of php and perl. Removing cpanel-perl will leave you with single centos perl where you can add modules from CentOS and EPEL when needed (and save gigabyte or two)
btw you are one openssl and 10 kernel updates behind safe system (but upgrading them will not add or free disk space)
0
 
SandyCommented:
Check for /var/spool/ directories it has mailqueue for deferred mails also which might be eating up your system space.  Apart from this i don't think anything in /usr can be treated for housekeeping purpose..

Check spool directory usage.. this might can help you

Ty/SA
0
 
SheppardDigitalAuthor Commented:
My main problem is though, that if i add up the space being used it doesn't add up to the total that df -h is telling me.

I can only account for around 18GB of the used space and can't seem to find the missing 16GB, unless I'm missing something totally obvious?
0
 
SandyCommented:
Ok.. So virtfs is not the one who only contains links to junk files...

it is in act whenever jailed ssh comes in picture . (http://www.docs.cpanel.net/twiki/bin/view/AllDocumentation/WHMDocs/VirtFS)

To see if any user has virtfs mounted, use the following command.

#cat /proc/mounts | grep virtfs

If you see any you can umount them. That will solve the issue.

TY/SA
0
 
SheppardDigitalAuthor Commented:
It looks like two users...

/dev/xvda1 /home/virtfs/webdev/lib ext3 ro,nosuid,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/webdev/bin ext3 ro,nosuid,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/webdev/sbin ext3 ro,nosuid,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/webdev/opt ext3 ro,nosuid,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/webdev/lib64 ext3 ro,nosuid,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/webdev/usr ext3 ro,nosuid,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/webdev/var ext3 ro,nosuid,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/webdev/var/spool ext3 rw,nosuid,noexec,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/webdev/var/log ext3 rw,nosuid,noexec,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/webdev/etc/mail ext3 rw,nosuid,noexec,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/loop0 /home/virtfs/webdev/tmp ext3 rw,nosuid,noexec,relatime,errors=continue,barrier=1,data=ordered 0 0
/dev/loop0 /home/virtfs/webdev/var/tmp ext3 rw,nosuid,noexec,relatime,errors=continue,barrier=1,data=ordered 0 0
devtmpfs /home/virtfs/webdev/dev devtmpfs rw,nosuid,noexec,relatime,size=487560k,nr_inodes=121890,mode=755 0 0
/dev/xvda1 /home/virtfs/webdev/usr/sbin ext3 ro,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/webdev/usr/local/cpanel/3rdparty/mailman ext3 ro,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/webdev/usr/local/cpanel/3rdparty/mailman/logs ext3 rw,noatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/webdev/usr/local/cpanel/3rdparty/mailman/locks ext3 rw,noatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/webdev/usr/local/cpanel/3rdparty/mailman/lists ext3 rw,noatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/webdev/usr/local/cpanel/3rdparty/mailman/qfiles ext3 rw,noatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/webdev/home/webdev ext3 rw,noatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/cdevsvr/lib ext3 ro,nosuid,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/cdevsvr/bin ext3 ro,nosuid,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/cdevsvr/sbin ext3 ro,nosuid,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/cdevsvr/opt ext3 ro,nosuid,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/cdevsvr/lib64 ext3 ro,nosuid,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/cdevsvr/usr ext3 ro,nosuid,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/cdevsvr/var ext3 ro,nosuid,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/cdevsvr/var/spool ext3 rw,nosuid,noexec,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/cdevsvr/var/log ext3 rw,nosuid,noexec,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/cdevsvr/etc/mail ext3 rw,nosuid,noexec,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/loop0 /home/virtfs/cdevsvr/tmp ext3 rw,nosuid,noexec,relatime,errors=continue,barrier=1,data=ordered 0 0
/dev/loop0 /home/virtfs/cdevsvr/var/tmp ext3 rw,nosuid,noexec,relatime,errors=continue,barrier=1,data=ordered 0 0
devtmpfs /home/virtfs/cdevsvr/dev devtmpfs rw,nosuid,noexec,relatime,size=487560k,nr_inodes=121890,mode=755 0 0
/dev/xvda1 /home/virtfs/cdevsvr/usr/sbin ext3 ro,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/cdevsvr/usr/local/cpanel/3rdparty/mailman ext3 ro,relatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/cdevsvr/usr/local/cpanel/3rdparty/mailman/logs ext3 rw,noatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/cdevsvr/usr/local/cpanel/3rdparty/mailman/locks ext3 rw,noatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/cdevsvr/usr/local/cpanel/3rdparty/mailman/lists ext3 rw,noatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/cdevsvr/usr/local/cpanel/3rdparty/mailman/qfiles ext3 rw,noatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/xvda1 /home/virtfs/cdevsvr/home/cdevsvr ext3 rw,noatime,errors=continue,barrier=0,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0

Open in new window

0
 
SandyCommented:
can you treat them by umount  ? if yes then go ahead and release this space.

TY/SA
0
 
SheppardDigitalAuthor Commented:
Hi Sandy,

If I unmount will it actually release the space, as I thought virtfs mounts were just links to existing files. So although du will show the space being used twice, in reality the space is only being used once on the server?
0
 
SandyCommented:
It shld release
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

  • 6
  • 4
  • 4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now