Solved

Solaris directory ull

Posted on 2013-06-17
16
415 Views
Last Modified: 2013-06-19
Hi,

I have a Solaris Sparc system where df -h command shows the one partition almost full.

/dev/vx/dsk/dba_DG/dba_oraswVol
                        51G    49G   1.8G    97%    /opt/oracle

However, when I check with another command, du -sh, if just shows 17G used.

 du -sh oracle
  17G   oracle

Do you have any idea why there is such difference?

thanks!
0
Comment
Question by:joaotelles
  • 8
  • 8
16 Comments
 

Author Comment

by:joaotelles
Comment Utility
memory and CPU usage seems OK in prstat and top commands.

dpa@DP-NODE-1> top
load averages:  4.64,  4.34,  3.96;                    up 4916+15:03:59                                                                                        09:27:07
543 processes: 541 sleeping, 2 on cpu
CPU states: 86.4% idle,  4.0% user,  9.6% kernel,  0.0% iowait,  0.0% swap
Memory: 16G phys mem, 1996M free mem, 16G total swap, 5057M free swap
0
 
LVL 23

Expert Comment

by:nemws1
Comment Utility
Do you have other directories in /opt that might be taking up the space?  'df' shows the full disk, but you're calling 'du' on only one directory in /opt.

What does this tell you:

du -sh /opt
0
 
LVL 23

Expert Comment

by:nemws1
Comment Utility
Also, run this (as root):

/usr/sbin/quot /opt

(yes, there is no 'e' there)
0
 

Author Comment

by:joaotelles
Comment Utility
the problem is inside /opt/oracle.There is no other directory inside /opt/oracle that is taking the space.

See the df -h output below:

Filesystem             size   used  avail capacity  Mounted on
/dev/md/dsk/d0          19G    15G   3.7G    81%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   5.8G   1.5M   5.8G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
/platform/SUNW,Sun-Fire-T200/lib/libc_psr/libc_psr_hwcap1.so.1
                        19G    15G   3.7G    81%    /platform/sun4v/lib/libc_psr.so.1
/platform/SUNW,Sun-Fire-T200/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1
                        19G    15G   3.7G    81%    /platform/sun4v/lib/sparcv9/libc_psr.so.1
fd                       0K     0K     0K     0%    /dev/fd
swap                    16G   9.8G   5.8G    64%    /tmp
swap                   5.8G    48K   5.8G     1%    /var/run
/dev/vx/dsk/dm_DG/dm_log_Vol
                       5.0G   705M   4.0G    15%    /var/opt/smarttrust/dpa/dm/log
/dev/vx/dsk/dm_DG/dm_data_Vol
                       5.0G    18M   4.7G     1%    /var/opt/smarttrust/dpa/dm/data
/dev/vx/dsk/dba_DG/dba_ArchVol
                        45G   118M    42G     1%    /opt/arch
/dev/vx/dsk/dba_DG/dba_Vol1
                       111G    86G    25G    78%    /opt/mnt1
/dev/vx/dsk/dba_DG/dba_Vol4
                       146G   100G    45G    69%    /opt/mnt4
/dev/vx/dsk/dba_DG/dba_Vol3
                       245G   232G    12G    96%    /opt/mnt3
/dev/vx/dsk/dba_DG/dba_Vol2
                       167G   164G   3.4G    98%    /opt/mnt2
/dev/vx/dsk/dba_DG/dba_oraswVol
                        51G    49G   1.8G    97%    /opt/oracle
/dev/vx/dsk/dp6a_DG/dp6a_dataVol
                       172G   143G    28G    84%    /opt/rman


/usr/sbin/quot /opt
quot: /opt doesn't appear to be a filesystem.
ufs usage: quot [-nfcvha] [filesystem ...]


 /usr/sbin/quot /opt/oracle
quot: /opt/oracle not ufs filesystem


du -sh /opt
 733G   /opt


 du -sh /opt/oracle
  17G   /opt/oracle
0
 
LVL 23

Expert Comment

by:nemws1
Comment Utility
Ah - sorry.  I see much more clearly now what is going on.

I'd like to this the difference has something to do with sparse files, but then things would be turned around (du would report a lot of usage and 'df' would *not*).

Can you try a usage tree utility?

Like this:
http://xdiskusage.sourceforge.net/

Or maybe dutree (perl - I use this all the time):
http://www.prasoonagrawal.com/library/oreilly/perl3/cookbook/ch05_17.htm
0
 

Author Comment

by:joaotelles
Comment Utility
I am sending attached the ouput from the dutree as you asked.

Let me know what you find.
oracle.log
0
 
LVL 23

Expert Comment

by:nemws1
Comment Utility
What's crazy is we now have a 3rd number.  Your dump says 35 gigs of disk is used.  Closer to 49 but still way off.  Something very odd is going on here. :(

Can you run 'df -h' again right now?  I'm wondering if you have some process that is using/freeing up disk space as you're working on it right now.
0
 

Author Comment

by:joaotelles
Comment Utility
sure!
here is df -h command again.

df -h
Filesystem             size   used  avail capacity  Mounted on
/dev/md/dsk/d0          19G    15G   3.7G    81%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   5.7G   1.5M   5.7G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
/platform/SUNW,Sun-Fire-T200/lib/libc_psr/libc_psr_hwcap1.so.1
                        19G    15G   3.7G    81%    /platform/sun4v/lib/libc_psr.so.1
/platform/SUNW,Sun-Fire-T200/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1
                        19G    15G   3.7G    81%    /platform/sun4v/lib/sparcv9/libc_psr.so.1
fd                       0K     0K     0K     0%    /dev/fd
swap                    16G   9.8G   5.7G    64%    /tmp
swap                   5.7G    48K   5.7G     1%    /var/run
/dev/vx/dsk/dm_DG/dm_log_Vol
                       5.0G   705M   4.0G    15%    /var/opt/smarttrust/dpa/dm/log
/dev/vx/dsk/dm_DG/dm_data_Vol
                       5.0G    18M   4.7G     1%    /var/opt/smarttrust/dpa/dm/data
/dev/vx/dsk/dba_DG/dba_ArchVol
                        45G   712M    42G     2%    /opt/arch
/dev/vx/dsk/dba_DG/dba_Vol1
                       111G    86G    25G    78%    /opt/mnt1
/dev/vx/dsk/dba_DG/dba_Vol4
                       146G   100G    45G    69%    /opt/mnt4
/dev/vx/dsk/dba_DG/dba_Vol3
                       245G   232G    12G    96%    /opt/mnt3
/dev/vx/dsk/dba_DG/dba_Vol2
                       167G   164G   3.4G    98%    /opt/mnt2
/dev/vx/dsk/dba_DG/dba_oraswVol
                        51G    49G   1.7G    97%    /opt/oracle
/dev/vx/dsk/dp6a_DG/dp6a_dataVol
                       172G   143G    28G    84%    /opt/rman
0
Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 23

Expert Comment

by:nemws1
Comment Utility
Dangit. The same as before. Let me email my buddy that works more with Oracle than I do to see if he has any ideas.

Can you also run this and post the output:  mount | grep oracle
0
 

Author Comment

by:joaotelles
Comment Utility
here is the mount output:

mount | grep oracle
/opt/oracle on /dev/vx/dsk/dba_DG/dba_oraswVol read/write/setuid/devices/log/largefiles/ioerror=mwdisable/dev=4a0c355 on Sun May 26 19:30:21 2013
0
 

Author Comment

by:joaotelles
Comment Utility
searching on the internet, I saw a difference between how du and df commands work.

du is an example of a user-level program that is not aware of file system meta data, while df looks at the file system disk allocation maps and is aware of file system meta data. df obtains the true file system statistics, whereas du sees only a partial picture.

Do you think this explains the difference?
0
 
LVL 23

Expert Comment

by:nemws1
Comment Utility
That is all correct, but it can't explain a multi-gigabyte difference.

Are you running 'du' as the root user?
0
 

Author Comment

by:joaotelles
Comment Utility
The output from du is the same for both root and my user. I tried run with both.
0
 
LVL 23

Accepted Solution

by:
nemws1 earned 500 total points
Comment Utility
Hmm.. I was hoping 'du' running as root would return something closer to 50 gigs.

I see these are all vx filesystems, I assume Veritas VXFS.  Unfortunately, its been forever since I worked with VXes - could there be some issue with that?  Is it possible you have a pooled filesystem (like you can with ZFS) where 2 drives belong to the same pool and usage by one affects the other?  That's the only type of thing I can think of that's going on with this.
0
 

Author Comment

by:joaotelles
Comment Utility
the problem was a zombie file in the /opt/oracle directory. The file was already deleted but some processes in the machine were using it, and the space was not released.
0
 
LVL 23

Expert Comment

by:nemws1
Comment Utility
Ah!  Very difficult to track down.  Glad you got it figured out.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

I have been running these systems for a few years now and I am just very happy with them.   I just wanted to share the manual that I have created for upgrades and other things.  Oooh yes! FreeBSD makes me happy (as a server), no maintenance and I al…
Learn about cloud computing and its benefits for small business owners.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now