Solved

root volume "/" running out of space....

Posted on 2011-09-21
11
533 Views
Last Modified: 2013-12-28
Solaris newbie here. We have a Solaris 10 box running oracle databases. The disk space usage on the root volume "/" is continuing to climb into the danger zone. Last night it was 93% utlized and this morning it's up to 94%. How can I find out what is chewing up this space before there is a problem? Please help!


 df -k screenshot
0
Comment
Question by:IT_Telephonics
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
11 Comments
 
LVL 9

Expert Comment

by:amolg
ID: 36573603
use df -h from root directory and see which directory is using more space and do the same in subdirectories which is using more space.
0
 
LVL 5

Expert Comment

by:Alexey Komarov
ID: 36573648
Hi
Look at $ORACLE_BASE\admin
my be oracle logs and trace
0
 
LVL 78

Expert Comment

by:arnold
ID: 36573671
check /var/log /var/adm to make sure all the files that are there are on a logadm rotation schedule.
find /var/log -size +300000c
repeat for /var/adm

The problem is that you have a single partition which make it harder to isolate the cause.

what does the system do? web server, ftp server, mail server, database server, Virtualization?

0
Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

 
LVL 22

Accepted Solution

by:
blu earned 500 total points
ID: 36574301
The most likely locations for unbounded disk space usage is in /var and in $ORACLE_BASE. You can't use the df command very effectively to
find the usages, but the du command can come in handy for this. I use this command to find out where the file space goes:

du -sdk * | sort -rn

First, go to /var and run the above command and save the output. Then do the same at $ORACLE_BASE. Look at the output for each and see if there
is anything that is using more space than you expect.

Once you have done this, if you still do not know where the space is going, wait a while and repeat those two commands and see what is increasing.
If you still do not see where the space is going, trying doing it from the "/" directory.
0
 

Author Comment

by:IT_Telephonics
ID: 36574990
Thanks for the help everyone. Now I think I'm getting somewhere. Using the 'du -sdk * | sort -rn' command in the /var directory I can see that the audit dir is by far the largest.

41214449        audit
697144            tmp
87349              sadm
41106               webconsole
37768              opt

The following files are in that directory:
# pwd
/var/audit
# ls -l
total 82437200
-rw-------   1 root     root     6952053749 Aug 27 13:37 20110709000121.20110827173755.<server name>
-rw-------   1 root     root     14215319161 Sep  7 21:30 20110829105451.20110908013026.<server name>
-rw-------   1 root     root     13009525154 Sep 16 03:23 20110908013026.20110916072330.<server name>
-rw-------   1 root     root     8010284957 Sep 21 11:50 20110916072330.not_terminated.<server name>

We did have a problem recently where someone set our /etc/security/audit_control file to log everything under the sun. We have since tuned that down a lot. Can these logs be trimmed\purged\deleted?

Thanks

Thanks
0
 
LVL 22

Assisted Solution

by:blu
blu earned 500 total points
ID: 36575143
You can look at this link for some ideas on how to set this up:

http://otoh.org/xwiki/bin/view/Blog/2008-08-29-audit-logadm
0
 

Author Comment

by:IT_Telephonics
ID: 36575319
I did some reading to trying to come up to speed on how auditing works. Looks like I only have one "active" log file in that dir which is the one that includes "not_terminated" in its name. The others can be backedup and removed according to this read: http://download.oracle.com/docs/cd/E19109-01/tsolaris7/805-8057/6j7iuhh33/index.html


Anyone see any problems if I backup\delete the three "closed" audit files?

Thanks
0
 

Author Comment

by:IT_Telephonics
ID: 36575329
Just another note. I'm at 95% utilized now. I ran another 'du -sdk * | sort -rn' and have verified that the audit dir is what is chewing up my resources when comparing the two results together.
0
 
LVL 22

Expert Comment

by:blu
ID: 36575978
You can delete the oldest, assuming that you no longer need it. You can use the scripts from the link I gave above to keep it from happening again.

You should ask yourself whether or not you really need auditing and if so, what events you really need.
0
 
LVL 16

Expert Comment

by:Joseph Gan
ID: 36576641
YOu can move the old audit files to other directory, ie. /u01. Usually /var/audit should be on a seperated filesystem if you have spare disk space availible. Otherwise, you can create a link to /u01.
0
 

Author Comment

by:IT_Telephonics
ID: 36577494
Back down to 34%. Crisis averted!!
Thanks again.
0

Featured Post

Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction Regular patching is part of a system administrator's tasks. However, many patches require that the system be in single-user mode before they can be installed. A cluster patch in particular can take quite a while to apply if the machine…
FreeBSD on EC2 FreeBSD (https://www.freebsd.org) is a robust Unix-like operating system that has been around for many years. FreeBSD is available on Amazon EC2 through Amazon Machine Images (AMIs) provided by FreeBSD developer and security office…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

724 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question