Linux Filesysyem full in /opt

root@[/] # df -PH /opt
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-optvol  4.1G  3.9G   17M 100% /opt
root@[/] #


root@[/] # tune2fs -l /dev/mapper/rootvg-optvol
tune2fs 1.41.12 (17-May-2010)
Filesystem volume name:   <none>
Last mounted on:          /opt
Filesystem UUID:          98fdee00-90e8-4004-aa9c-0ca1cee8e7fa
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              256000
Block count:              1024000
Reserved block count:     51200
Free blocks:              682506
Free inodes:              255508
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      249
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8000
Inode blocks per group:   500
Flex block group size:    16
Filesystem created:       Thu May 21 06:08:47 2015
Last mount time:          Sat Jul 22 00:48:54 2017
Last write time:          Sat Jul 22 00:48:54 2017
Mount count:              28
Maximum mount count:      -1
Last checked:             Thu May 21 06:08:47 2015
Check interval:           0 (<none>)
Lifetime writes:          10 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      b3290fba-ad7b-42ad-b624-74163eb83c47
Journal backup:           inode blocks

Select allOpen in new window

You had ..
lsof /opt look for a large file that actually has no entry in the /opt partition likely a result of a process that logs but an incorrect log rotation was setup that removed the filesystem reference without triggering the process to detach from the old file.


/opt is commonly not used, so not including that information is what prevents ado this.
Looks like Sysadmin has a home dir on /opt.

Likely Arnold's suggestion will turn up the problem.

Look for large deleted files, as some processes do this...

1) Open a temp/work file.

2) Unlink the file, which removes it from any directory listings.

3) Uses the file for a long time.

This means you end up with invisible files which can chew up all your disk space.

Try this command + see if any massive files show up.

lsof 2>/dev/null | egrep -e SIZE -e deleted

Select allOpen in new window

I see you have /opt/bmc/SCBmon mounted.  Having that mounted there could mask something in the /opt file system

Unmount /opt/bmc/SCBmon and re-run your du -sh * command and post.

@arnold,

Thanks for your suggestion. There is no big files at all in/opt

root@[devops] # lsof /opt
COMMAND     PID      USER   FD   TYPE DEVICE  SIZE/OFF NODE NAME
netprobe.  2271      itrs  cwd    DIR 253,11      4096  308 /opt/ITRS/Netprobe/HKLVAUEDM08
netprobe.  2271      itrs  txt    REG 253,11  20873029  299 /opt/ITRS/Netprobe/GA3.0.17-140205/netprobe.linux_64
netprobe.  2271      itrs    3w   REG 253,11    770819  309 /opt/ITRS/Netprobe/HKLVAUEDM08/netprobe.log
filebeat- 28725 npelastic  txt    REG 253,11   1179764  546 /opt/app/filebeat/config/filebeat-god
filebeat  28726 npelastic  txt    REG 253,11  46837773  565 /opt/app/filebeat/bin/filebeat-6.0.0-linux-x86_64/filebeat
filebeat  28726 npelastic    1u   REG 253,11    962615  635 /opt/app/filebeat/logs/filebeat
metricbea 31887 npelastic  txt    REG 253,11   1179764  547 /opt/app/metricbeat/config/metricbeat-god
metricbea 31888 npelastic  txt    REG 253,11 109164974  804 /opt/app/metricbeat/bin/metricbeat-6.0.0-linux-x86_64/metricbeat
metricbea 31888 npelastic    1u   REG 253,11   1338286  843 /opt/app/metricbeat/logs/metricbeat

root@[devops] # lsof /opt  | awk '{print $NF}' | xargs ls -rtlh
ls: cannot access NAME: No such file or directory
-rwxr-xr-x 1 itrs      itrs       20M Apr 29  2014 /opt/ITRS/Netprobe/GA3.0.17-140205/netprobe.linux_64
-rwxr-xr-x 1 npelastic npelastic 1.2M Oct 13 13:00 /opt/app/filebeat/config/filebeat-god
-rwxr-xr-x 1 npelastic npelastic 1.2M Oct 13 13:00 /opt/app/metricbeat/config/metricbeat-god
-rwxr-x--- 1 npelastic npelastic  45M Nov 11 02:30 /opt/app/filebeat/bin/filebeat-6.0.0-linux-x86_64/filebeat
-rwxr-x--- 1 npelastic npelastic 105M Nov 11 02:34 /opt/app/metricbeat/bin/metricbeat-6.0.0-linux-x86_64/metricbeat
-rw-r----- 1 itrs      itrs      753K Nov 25 01:04 /opt/ITRS/Netprobe/HKLVAUEDM08/netprobe.log
-rw------- 1 npelastic npelastic 1.3M Dec  8 14:04 /opt/app/metricbeat/logs/metricbeat
-rw------- 1 npelastic npelastic 941K Dec  8 14:05 /opt/app/filebeat/logs/filebeat

/opt/ITRS/Netprobe/HKLVAUEDM08:
total 768K
-rwxr--r-- 1 itrs itrs 1.1K Jul 22 00:49 ntp.sh
-rwxr--r-- 1 itrs itrs 2.1K Jul 22 00:49 DNS_Resolution_check.sh
-rw-r----- 1 itrs itrs 753K Nov 25 01:04 netprobe.log
root@[devops] #

Select allOpen in new window

@Erik,

I don't think it is a problem. Because the size matters here see below output for your info.

root@[devops] # df -PH /opt/bmc/SCBmon
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-scbmonlv  1.1G   35M  969M   4% /opt/bmc/SCBmon
root@[devops] # df -PH /opt
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-optvol  4.1G  4.0G  107M  98% /opt
root@[devops] #

Select allOpen in new window

please post the output of df -k
Was the system changed in terms of mount points as Eric pointed out earlier.
i.e. you had /opt with all the data and now you added the softwareag overlay without removing the data that was there.

The issue might be a delay in the mount of the softwareag overlay which leads to /opt increasing in space use and is being hidden by the overlay when it does come up.

The softwareag folders shows up as 18GB on a 4.1GB partition.

@arnold,

I also cheked home directory in /etc/passwd its as below...

root@[devops] # cat /etc/passwd | grep -i "/opt"
patrol:x:700:700:BMC Patrol user:/opt/bmc/patrol:/bin/ksh
scbmonlc:x:18562201:100:SCBmon TSS unix:/opt/tss/SCBmon:/bin/bash
root@[devops] # df -PH
Filesystem                                 Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-rootvol                 6.1G  2.3G  3.6G  39% /
tmpfs                                      8.4G     0  8.4G   0% /dev/shm
/dev/sda1                                   98M   80M   13M  87% /boot
/dev/mapper/rootvg-homevol                 4.1G  667M  3.2G  18% /home
/dev/mapper/rootvg-optvol                  4.1G  4.0G  107M  98% /opt
/dev/mapper/rootvg-tmpvol                  4.1G  243M  3.7G   7% /tmp
/dev/mapper/rootvg-oprvol                  317M  8.9M  291M   3% /u/operator
/dev/mapper/rootvg-usrctmvol               3.1G  4.7M  2.9G   1% /usr/controlm
/dev/mapper/rootvg-usrgemsvvol             1.1G  1.4M  969M   1% /usr/gems
/dev/mapper/rootvg-usropenvvol             2.1G  188M  1.8G  10% /usr/openv
/dev/mapper/rootvg-semavol                 1.1G  201M  770M  21% /usr/semasupp
/dev/mapper/rootvg-varvol                  4.1G  352M  3.6G  10% /var
/dev/mapper/HKLVAUEDM08_applvg1-wmstorage   11G  9.8G  109M  99% /wmstorage
/dev/mapper/HKLVAUEDM08_applvg1-isvol1      11G  5.1G  4.9G  52% /opt/softwareag/inst1
/dev/mapper/HKLVAUEDM08_applvg1-isvol2      11G  8.8G  1.2G  89% /opt/softwareag/inst2
/dev/mapper/HKLVAUEDM08_applvg1-isvol3      11G  2.7G  7.3G  27% /opt/softwareag/inst3
/dev/mapper/rootvg-scbmonlv                1.1G   35M  969M   4% /opt/bmc/SCBmon
/dev/mapper/HKLVAUEDM08_applvg1-tsavol      11G  1.9G  8.1G  19% /opt/softwareag/logs
root@[devops] # du -sh /opt/bmc/patrol
20K     /opt/bmc/patrol
root@[devops] # du -sh /opt/tss/SCBmon
92K     /opt/tss/SCBmon
root@[devops] # du -sh /opt/tss/SCBmon/*
60K     /opt/tss/SCBmon/bin
8.0K    /opt/tss/SCBmon/etc
root@[devops] # 

Select allOpen in new window

@Arnold,

Below point might be the scenario in my case, ,What i need to do in this case exactly now :( :(  i am clueless completely

please post the output of df -k
Was the system changed in terms of mount points as Eric pointed out earlier.
i.e. you had /opt with all the data and now you added the softwareag overlay without removing the data that was there.

The issue might be a delay in the mount of the softwareag overlay which leads to /opt increasing in space use and is being hidden by the overlay when it does come up.

@Arnold,

Please find the output of df -PH  as below..

root@[devops] # df -PH | column -t
Filesystem                                 Size  Used  Avail  Use%  Mounted                on
/dev/mapper/rootvg-rootvol                 6.1G  2.3G  3.6G   39%   /
tmpfs                                      8.4G  0     8.4G   0%    /dev/shm
/dev/sda1                                  98M   80M   13M    87%   /boot
/dev/mapper/rootvg-homevol                 4.1G  667M  3.2G   18%   /home
/dev/mapper/rootvg-optvol                  4.1G  4.0G  107M   98%   /opt
/dev/mapper/rootvg-tmpvol                  4.1G  243M  3.7G   7%    /tmp
/dev/mapper/rootvg-oprvol                  317M  8.9M  291M   3%    /u/operator
/dev/mapper/rootvg-usrctmvol               3.1G  4.7M  2.9G   1%    /usr/controlm
/dev/mapper/rootvg-usrgemsvvol             1.1G  1.4M  969M   1%    /usr/gems
/dev/mapper/rootvg-usropenvvol             2.1G  188M  1.8G   10%   /usr/openv
/dev/mapper/rootvg-semavol                 1.1G  201M  769M   21%   /usr/semasupp
/dev/mapper/rootvg-varvol                  4.1G  352M  3.6G   10%   /var
/dev/mapper/HKLVAUEDM08_applvg1-wmstorage  11G   9.8G  108M   99%   /wmstorage
/dev/mapper/HKLVAUEDM08_applvg1-isvol1     11G   5.1G  4.9G   52%   /opt/softwareag/inst1
/dev/mapper/HKLVAUEDM08_applvg1-isvol2     11G   8.8G  1.2G   89%   /opt/softwareag/inst2
/dev/mapper/HKLVAUEDM08_applvg1-isvol3     11G   2.7G  7.3G   27%   /opt/softwareag/inst3
/dev/mapper/rootvg-scbmonlv                1.1G  35M   969M   4%    /opt/bmc/SCBmon
/dev/mapper/HKLVAUEDM08_applvg1-tsavol     11G   1.9G  8.1G   19%   /opt/softwareag/logs
root@[devops] #

Select allOpen in new window

@Arnold,

Regarding your comment below...please find the output as below..

The softwareag folders shows up as 18GB on a 4.1GB partition

root@[opt] # pwd
/opt
root@[opt] # du -sh *
222M    app
28K     bmc
97M     ITRS
16K     lost+found
18G     softwareag
336K    sysadmin
3.0M    tss
root@[opt] #

Select allOpen in new window

@arnold...

See my /dev/mapper block devices as below..

root@[mapper] # df -PH | column -t
Filesystem                                 Size  Used  Avail  Use%  Mounted                on
/dev/mapper/rootvg-rootvol                 6.1G  2.3G  3.6G   39%   /
tmpfs                                      8.4G  0     8.4G   0%    /dev/shm
/dev/sda1                                  98M   80M   13M    87%   /boot
/dev/mapper/rootvg-homevol                 4.1G  667M  3.2G   18%   /home
/dev/mapper/rootvg-optvol                  4.1G  4.0G  107M   98%   /opt
/dev/mapper/rootvg-tmpvol                  4.1G  243M  3.7G   7%    /tmp
/dev/mapper/rootvg-oprvol                  317M  8.9M  291M   3%    /u/operator
/dev/mapper/rootvg-usrctmvol               3.1G  4.7M  2.9G   1%    /usr/controlm
/dev/mapper/rootvg-usrgemsvvol             1.1G  1.4M  969M   1%    /usr/gems
/dev/mapper/rootvg-usropenvvol             2.1G  188M  1.8G   10%   /usr/openv
/dev/mapper/rootvg-semavol                 1.1G  201M  769M   21%   /usr/semasupp
/dev/mapper/rootvg-varvol                  4.1G  352M  3.5G   10%   /var
/dev/mapper/HKLVAUEDM08_applvg1-wmstorage  11G   9.8G  99M    100%  /wmstorage
/dev/mapper/HKLVAUEDM08_applvg1-isvol1     11G   5.1G  4.9G   52%   /opt/softwareag/inst1
/dev/mapper/HKLVAUEDM08_applvg1-isvol2     11G   8.8G  1.2G   89%   /opt/softwareag/inst2
/dev/mapper/HKLVAUEDM08_applvg1-isvol3     11G   2.7G  7.3G   27%   /opt/softwareag/inst3
/dev/mapper/rootvg-scbmonlv                1.1G  35M   969M   4%    /opt/bmc/SCBmon
/dev/mapper/HKLVAUEDM08_applvg1-tsavol     11G   1.9G  8.1G   19%   /opt/softwareag/logs
root@[mapper] #



root@[opt] # cd /dev/mapper/
root@[mapper] # ls -rtlh
total 0
crw-rw---- 1 root root 10, 56 Jul 22 00:48 control
lrwxrwxrwx 1 root root      7 Jul 22 00:48 rootvg-rootvolsnap-cow -> ../dm-2
lrwxrwxrwx 1 root root      7 Jul 22 00:48 rootvg-rootvol-real -> ../dm-0
lrwxrwxrwx 1 root root      7 Jul 22 00:48 rootvg-rootvolsnap -> ../dm-3
lrwxrwxrwx 1 root root      7 Jul 22 00:48 rootvg-swapvol -> ../dm-4
lrwxrwxrwx 1 root root      7 Jul 22 00:48 HKLVAUEDM08_applvg1-isvol2 -> ../dm-7
lrwxrwxrwx 1 root root      7 Jul 22 00:48 HKLVAUEDM08_applvg1-tsavol -> ../dm-9
lrwxrwxrwx 1 root root      7 Jul 22 00:48 HKLVAUEDM08_applvg1-wmstorage -> ../dm-5
lrwxrwxrwx 1 root root      7 Jul 22 00:48 HKLVAUEDM08_applvg1-isvol3 -> ../dm-8
lrwxrwxrwx 1 root root      7 Jul 22 00:48 HKLVAUEDM08_applvg1-isvol1 -> ../dm-6
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-optvolsnap-cow -> ../dm-12
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-optvol-real -> ../dm-10
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-optvolsnap -> ../dm-13
lrwxrwxrwx 1 root root      7 Jul 22 00:48 rootvg-rootvol -> ../dm-1
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-homevol -> ../dm-19
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-tmpvol -> ../dm-18
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-oprvol -> ../dm-23
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-usrctmvol -> ../dm-24
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-usrgemsvvol -> ../dm-21
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-usropenvvol -> ../dm-20
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-semavol -> ../dm-22
lrwxrwxrwx 1 root root      8 Sep 13 20:30 rootvg-scbmonlv -> ../dm-25
lrwxrwxrwx 1 root root      8 Nov  7 03:07 rootvg-varvol-real -> ../dm-14
lrwxrwxrwx 1 root root      8 Nov  7 03:07 rootvg-varvolsnap-cow -> ../dm-16
lrwxrwxrwx 1 root root      8 Nov  7 03:07 rootvg-varvol -> ../dm-15
lrwxrwxrwx 1 root root      8 Nov  7 03:07 rootvg-varvolsnap -> ../dm-17
lrwxrwxrwx 1 root root      8 Dec  7 17:39 rootvg-optvol -> ../dm-11
root@[mapper] #

Select allOpen in new window

@Arnold,

Here are my blocks in /proc/partitions

root@[mapper] # cat /proc/partitions
major minor  #blocks  name

   8       16   52428800 sdb
   8       17   52428784 sdb1
   8        0   52428800 sda
   8        1     102400 sda1
   8        2   52325376 sda2
 253        0    6160384 dm-0
 253        1    6160384 dm-1
 253        2    3047424 dm-2
 253        3    6160384 dm-3
 253        4    6160384 dm-4
 253        5   10481664 dm-5
 253        6   10485760 dm-6
 253        7   10485760 dm-7
 253        8   10485760 dm-8
 253        9   10485760 dm-9
 253       10    4096000 dm-10
 253       11    4096000 dm-11
 253       12    4128768 dm-12
 253       13    4096000 dm-13
 253       14    4096000 dm-14
 253       15    4096000 dm-15
 253       16    4128768 dm-16
 253       17    4096000 dm-17
 253       18    4096000 dm-18
 253       19    4096000 dm-19
 253       20    2064384 dm-20
 253       21    1048576 dm-21
 253       22    1048576 dm-22
 253       23     327680 dm-23
 253       24    3080192 dm-24
 253       25    1048576 dm-25
root@[mapper] #

Select allOpen in new window

were changes made recently to add the extra partitions for softwareag/inst*?

du -ks /opt/softwareag

sh follows hard links which mount points are.
try
cd  /opt
ls | while read a; do
echo -e "$a has: "
du -ks "$a"
done 2>/dev/null

please post the output let see if that sheds light on the matter.

In short, your space is consumed by an overlay i.e. at some point, one of the /opt related mount points was not mounted likely/possibly the /opt/softwareag/logs and that could be where this space is consumed.

If you can, unmount /opt/softwareag/logs and see if /opt/softwareag/logs has data .
make sure the change does not impact production....
What is the source of these sofwareag mount overlays ?volume?

@armold,

Yes there were new partitions added it seems.

root@[mapper] # du -ks /opt/softwareag
17871156        /opt/softwareag
root@[mapper] # cd /opt
root@[opt] # ls | while read a; do
> echo -e "$a has: "
> du -ks "$a"
> done 2>/dev/null
app has:
226636  app
bmc has:
28      bmc
ITRS has:
99064   ITRS
lost+found has:
16      lost+found
softwareag has:
17871172        softwareag
sysadmin has:
336     sysadmin
tss has:
3032    tss
root@[opt] #

Select allOpen in new window

for your info please find fdisk output too.

root@[mapper] # df -PH | column -t | grep -i opt
Filesystem                                 Size  Used  Avail  Use%  Mounted                on
/dev/mapper/rootvg-optvol                  4.1G  4.0G  107M   98%   /opt
/dev/mapper/_applvg1-isvol1     11G   5.1G  4.9G   52%   /opt/softwareag/inst1
/dev/mapper/_applvg1-isvol2     11G   8.8G  1.2G   89%   /opt/softwareag/inst2
/dev/mapper/_applvg1-isvol3     11G   2.7G  7.3G   27%   /opt/softwareag/inst3
/dev/mapper/rootvg-scbmonlv                1.1G  35M   969M   4%    /opt/bmc/SCBmon
/dev/mapper/_applvg1-tsavol     11G   1.9G  8.1G   19%   /opt/softwareag/logs
root@[mapper] # fdisk -l /dev/mapper/rootvg-optvol
Disk /dev/mapper/rootvg-optvol: 4194 MB, 4194304000 bytes
255 heads, 63 sectors/track, 509 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

root@[mapper] # fdisk -l /dev/mapper/_applvg1-isvol1
Disk /dev/mapper/_applvg1-isvol1: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

root@[mapper] # fdisk -l /dev/mapper/_applvg1-isvol2
Disk /dev/mapper/_applvg1-isvol2: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

root@[mapper] # fdisk -l /dev/mapper/_applvg1-isvol3
Disk /dev/mapper/_applvg1-isvol3: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

root@[mapper] # fdisk -l /dev/mapper/rootvg-scbmonlv
Disk /dev/mapper/rootvg-scbmonlv: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

root@[mapper] # fdisk -l /dev/mapper/_applvg1-tsavol
Disk /dev/mapper/_applvg1-tsavol: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
root@[mapper] #

Select allOpen in new window

@Arnold,

Yes it is my production box, That's why i am scared to take any risk :( :(

The source for softwareag mount is as below VG....is there anyway i can remove somespace to reduce /opt space ?

root@[opt] # df -PH
Filesystem                                 Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-rootvol                 6.1G  2.3G  3.6G  39% /
tmpfs                                      8.4G     0  8.4G   0% /dev/shm
/dev/sda1                                   98M   80M   13M  87% /boot
/dev/mapper/rootvg-homevol                 4.1G  667M  3.2G  18% /home
/dev/mapper/rootvg-optvol                  4.1G  4.0G  107M  98% /opt
/dev/mapper/rootvg-tmpvol                  4.1G  243M  3.7G   7% /tmp
/dev/mapper/rootvg-oprvol                  317M  8.9M  291M   3% /u/operator
/dev/mapper/rootvg-usrctmvol               3.1G  4.7M  2.9G   1% /usr/controlm
/dev/mapper/rootvg-usrgemsvvol             1.1G  1.4M  969M   1% /usr/gems
/dev/mapper/rootvg-usropenvvol             2.1G  188M  1.8G  10% /usr/openv
/dev/mapper/rootvg-semavol                 1.1G  201M  769M  21% /usr/semasupp
/dev/mapper/rootvg-varvol                  4.1G  352M  3.5G  10% /var
/dev/mapper/HKLVAUEDM08_applvg1-wmstorage   11G  9.9G   17M 100% /wmstorage
/dev/mapper/HKLVAUEDM08_applvg1-isvol1      11G  5.1G  4.9G  52% /opt/softwareag/inst1
/dev/mapper/HKLVAUEDM08_applvg1-isvol2      11G  8.9G  1.1G  89% /opt/softwareag/inst2
/dev/mapper/HKLVAUEDM08_applvg1-isvol3      11G  2.7G  7.3G  27% /opt/softwareag/inst3
/dev/mapper/rootvg-scbmonlv                1.1G   35M  969M   4% /opt/bmc/SCBmon
/dev/mapper/HKLVAUEDM08_applvg1-tsavol      11G  1.9G  8.0G  20% /opt/softwareag/logs
root@[opt] # vgs
  VG                  #PV #LV #SN Attr   VSize  VFree
  HKLVAUEDM08_applvg1   1   5   0 wz--n- 50.00g    0
  rootvg                1  15   3 wz--n- 49.88g 3.50g
root@[opt] # lvs
  LV          VG                  Attr       LSize   Pool Origin  Data%  Meta%  Move Log Cpy%Sync Convert
  isvol1      HKLVAUEDM08_applvg1 -wi-ao----  10.00g
  isvol2      HKLVAUEDM08_applvg1 -wi-ao----  10.00g
  isvol3      HKLVAUEDM08_applvg1 -wi-ao----  10.00g
  tsavol      HKLVAUEDM08_applvg1 -wi-ao----  10.00g
  wmstorage   HKLVAUEDM08_applvg1 -wi-ao----  10.00g
  homevol     rootvg              -wi-ao----   3.91g
  oprvol      rootvg              -wi-ao---- 320.00m
  optvol      rootvg              owi-aos---   3.91g
  optvolsnap  rootvg              swi-a-s---   3.94g      optvol  95.39
  rootvol     rootvg              owi-aos---   5.88g
  rootvolsnap rootvg              swi-a-s---   2.91g      rootvol 69.49
  scbmonlv    rootvg              -wi-ao----   1.00g
  semavol     rootvg              -wi-ao----   1.00g
  swapvol     rootvg              -wi-ao----   5.88g
  tmpvol      rootvg              -wi-ao----   3.91g
  usrctmvol   rootvg              -wi-ao----   2.94g
  usrgemsvvol rootvg              -wi-ao----   1.00g
  usropenvvol rootvg              -wi-ao----   1.97g
  varvol      rootvg              owi-aos---   3.91g
  varvolsnap  rootvg              swi-a-s---   3.94g      varvol  88.22
root@[opt] #

Select allOpen in new window

the only possible location based on current usage, inst3 and logs within softwareag ..

 the only way is to refer  to Eric's point that the /opt filled up and inst3 or logs was added without actually freeing the space consumed there.

I would look at logs first.

What gets files into /opt/softwareag/logs ?
lsof /opt/softwareag/logs?

@Arnold,

How you identified that the /opt space consumed by inst3 and logs partition....please find below lsof ouput on logs

root@[opt] # lsof /opt/softwareag/logs
COMMAND     PID     USER   FD   TYPE DEVICE  SIZE/OFF   NODE NAME
server.sh  6644 webmadmn    1w   REG  253,9    749568 526876 /opt/softwareag/logs/inst1/server.out
server.sh  6644 webmadmn    2w   REG  253,9    749568 526876 /opt/softwareag/logs/inst1/server.out
sagis95    6677 webmadmn    1w   REG  253,9    749568 526876 /opt/softwareag/logs/inst1/server.out
sagis95    6677 webmadmn    2w   REG  253,9    749568 526876 /opt/softwareag/logs/inst1/server.out
wrapper-3  6731 webmadmn    1w   REG  253,9    749568 526876 /opt/softwareag/logs/inst1/server.out
wrapper-3  6731 webmadmn    2w   REG  253,9    749568 526876 /opt/softwareag/logs/inst1/server.out
java       6733 webmadmn  159w   REG  253,9    137415 524292 /opt/softwareag/logs/inst1/stats.log
java       6733 webmadmn  492w   REG  253,9   1611675 524318 /opt/softwareag/logs/inst1/server.log
java       6733 webmadmn  496w   REG  253,9         0 524317 /opt/softwareag/logs/inst1/FailedAudit_20171208_000000.log
java       6733 webmadmn  497w   REG  253,9         0 524296 /opt/softwareag/logs/inst1/WMTXOUT_20171208_000000.log
java       6733 webmadmn  512w   REG  253,9    843200 524299 /opt/softwareag/logs/inst1/WMSESSION_20171208_000000.log
java       6733 webmadmn  514w   REG  253,9         0 524300 /opt/softwareag/logs/inst1/WMTXIN_20171208_000000.log
java       6733 webmadmn  539w   REG  253,9         0 524313 /opt/softwareag/logs/inst1/WMERROR_20171208_000000.log
java       8004 webmadmn   24w   REG  253,9    180644     62 /opt/softwareag/logs/inst2/stats.log
java       8004 webmadmn  159w   REG  253,9   9478031     15 /opt/softwareag/logs/inst2/server.log
java       8004 webmadmn  172w   REG  253,9   1663101     19 /opt/softwareag/logs/inst2/WMSECURITY_20171208_000000.log
java       8004 webmadmn  770w   REG  253,9         0     63 /opt/softwareag/logs/inst2/WMTXOUT_20171208_000000.log
java       8004 webmadmn  773w   REG  253,9  20135616     64 /opt/softwareag/logs/inst2/WMSESSION_20171208_000000.log
java       8004 webmadmn  792u   REG  253,9         0     65 /opt/softwareag/logs/inst2/WMTXIN_20171208_000000.log
java       8004 webmadmn  793u   REG  253,9   1489440     67 /opt/softwareag/logs/inst2/WMSERVICE_20171208_000000.log
java       8004 webmadmn  795w   REG  253,9  15762912     68 /opt/softwareag/logs/inst2/WMERROR_20171208_000000.log
java       8004 webmadmn  798u   REG  253,9         0     69 /opt/softwareag/logs/inst2/FailedAudit_20171208_000000.log
server.sh 25743 webmadmn    1w   REG  253,9 113451204     12 /opt/softwareag/logs/inst2/server.out
server.sh 25743 webmadmn    2w   REG  253,9 113451204     12 /opt/softwareag/logs/inst2/server.out
sagis95   25776 webmadmn    1w   REG  253,9 113451204     12 /opt/softwareag/logs/inst2/server.out
sagis95   25776 webmadmn    2w   REG  253,9 113451204     12 /opt/softwareag/logs/inst2/server.out
wrapper-3 25830 webmadmn    1w   REG  253,9 113451204     12 /opt/softwareag/logs/inst2/server.out
wrapper-3 25830 webmadmn    2w   REG  253,9 113451204     12 /opt/softwareag/logs/inst2/server.out
root@[opt] #

Select allOpen in new window

@arnold,

Thanks for your valuable suggestions...

So I tried to delete some logs under  inst3 and logs mount point but still /opt showing same usage I.e. 98%

You mean to say if restart a application service in inst3 and logs mount point it may release the space in /opt right

when using LVM, you first use lvresize to adjust the lv. resizefs is the last thing one runs.
if adding
pvcreate to add a new resource (/dev/sdx)
vgextend to expand the VolumeGroup (/dev/sdx as a resource on which this volume group)
lvresize  to insrease the logical volume (lvextend) only reference the volumegroup and logical volume ...
resizefs (depending on the filesystem in use) to adjust the filesystem to match the logical volume.
 lvresize -L -50MB will reduce the logical volume mentioned by 50MB as long as there is 50MB of free space.

When making changes to the filesystem, always make sure you have a good backup.

This is great explanation Erik...thanks a lot inface it is my production i need to ask downtime to do it :( :( its bit process oriented....thanks a lot arnold as well for his greatest support