Linux Filesysyem full in /opt

Eventhough there are no big files there still /opt/ is showing 100% full it very confusing to me. Can anyone help me on this please


root@[opt] # lsof | grep -i deleted
root@[opt] #

Open in new window


root@[opt] # lvs
  LV          VG                  Attr       LSize   Pool Origin  Data%  Meta%  Move Log Cpy%Sync Convert
  isvol1      HKLVAUEDM08_applvg1 -wi-ao----  10.00g
  isvol2      HKLVAUEDM08_applvg1 -wi-ao----  10.00g
  isvol3      HKLVAUEDM08_applvg1 -wi-ao----  10.00g
  tsavol      HKLVAUEDM08_applvg1 -wi-ao----  10.00g
  wmstorage   HKLVAUEDM08_applvg1 -wi-ao----  10.00g
  homevol     rootvg              -wi-ao----   3.91g
  oprvol      rootvg              -wi-ao---- 320.00m
  optvol      rootvg              owi-aos---   3.91g
  optvolsnap  rootvg              swi-a-s---   3.94g      optvol  95.39
  rootvol     rootvg              owi-aos---   5.88g
  rootvolsnap rootvg              swi-a-s---   2.91g      rootvol 69.49
  scbmonlv    rootvg              -wi-ao----   1.00g
  semavol     rootvg              -wi-ao----   1.00g
  swapvol     rootvg              -wi-ao----   5.88g
  tmpvol      rootvg              -wi-ao----   3.91g
  usrctmvol   rootvg              -wi-ao----   2.94g
  usrgemsvvol rootvg              -wi-ao----   1.00g
  usropenvvol rootvg              -wi-ao----   1.97g
  varvol      rootvg              owi-aos---   3.91g
  varvolsnap  rootvg              swi-a-s---   3.94g      varvol  88.16

root@[opt] # df -PH | grep -i "/opt" | grep -i root
/dev/mapper/rootvg-optvol                  4.1G  3.9G   17M 100% /opt
/dev/mapper/rootvg-scbmonlv                1.1G   35M  969M   4% /opt/bmc/SCBmon
root@[opt] # pwd
/opt
root@[opt] # du -sh *
107M    app
28K     bmc
97M     ITRS
16K     lost+found
18G     softwareag
336K    sysadmin
3.0M    tss
root@[opt] #

Open in new window

LVL 1
BharathKumarRaju DasaraRajuDevops EngineerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

BharathKumarRaju DasaraRajuDevops EngineerAuthor Commented:
root@[/] # df -PH /opt
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-optvol  4.1G  3.9G   17M 100% /opt
root@[/] #


root@[/] # tune2fs -l /dev/mapper/rootvg-optvol
tune2fs 1.41.12 (17-May-2010)
Filesystem volume name:   <none>
Last mounted on:          /opt
Filesystem UUID:          98fdee00-90e8-4004-aa9c-0ca1cee8e7fa
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              256000
Block count:              1024000
Reserved block count:     51200
Free blocks:              682506
Free inodes:              255508
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      249
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8000
Inode blocks per group:   500
Flex block group size:    16
Filesystem created:       Thu May 21 06:08:47 2015
Last mount time:          Sat Jul 22 00:48:54 2017
Last write time:          Sat Jul 22 00:48:54 2017
Mount count:              28
Maximum mount count:      -1
Last checked:             Thu May 21 06:08:47 2015
Check interval:           0 (<none>)
Lifetime writes:          10 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      b3290fba-ad7b-42ad-b624-74163eb83c47
Journal backup:           inode blocks

Open in new window

0
arnoldCommented:
You had ..
lsof /opt look for a large file that actually has no entry in the /opt partition likely a result of a process that logs but an incorrect log rotation was setup that removed the filesystem reference without triggering the process to detach from the old file.


/opt is commonly not used, so not including that information is what prevents ado this.
Looks like Sysadmin has a home dir on /opt.
1
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
Likely Arnold's suggestion will turn up the problem.

Look for large deleted files, as some processes do this...

1) Open a temp/work file.

2) Unlink the file, which removes it from any directory listings.

3) Uses the file for a long time.

This means you end up with invisible files which can chew up all your disk space.

Try this command + see if any massive files show up.

lsof 2>/dev/null | egrep -e SIZE -e deleted

Open in new window

0
Acronis True Image 2019 just released!

Create a reliable backup. Make sure you always have dependable copies of your data so you can restore your entire system or individual files.

Erik SchminkeAIX/Linux Systems AdministratorCommented:
I see you have /opt/bmc/SCBmon mounted.  Having that mounted there could mask something in the /opt file system

Unmount /opt/bmc/SCBmon and re-run your du -sh * command and post.
1
BharathKumarRaju DasaraRajuDevops EngineerAuthor Commented:
@arnold,

Thanks for your suggestion. There is no big files at all in/opt

root@[devops] # lsof /opt
COMMAND     PID      USER   FD   TYPE DEVICE  SIZE/OFF NODE NAME
netprobe.  2271      itrs  cwd    DIR 253,11      4096  308 /opt/ITRS/Netprobe/HKLVAUEDM08
netprobe.  2271      itrs  txt    REG 253,11  20873029  299 /opt/ITRS/Netprobe/GA3.0.17-140205/netprobe.linux_64
netprobe.  2271      itrs    3w   REG 253,11    770819  309 /opt/ITRS/Netprobe/HKLVAUEDM08/netprobe.log
filebeat- 28725 npelastic  txt    REG 253,11   1179764  546 /opt/app/filebeat/config/filebeat-god
filebeat  28726 npelastic  txt    REG 253,11  46837773  565 /opt/app/filebeat/bin/filebeat-6.0.0-linux-x86_64/filebeat
filebeat  28726 npelastic    1u   REG 253,11    962615  635 /opt/app/filebeat/logs/filebeat
metricbea 31887 npelastic  txt    REG 253,11   1179764  547 /opt/app/metricbeat/config/metricbeat-god
metricbea 31888 npelastic  txt    REG 253,11 109164974  804 /opt/app/metricbeat/bin/metricbeat-6.0.0-linux-x86_64/metricbeat
metricbea 31888 npelastic    1u   REG 253,11   1338286  843 /opt/app/metricbeat/logs/metricbeat

root@[devops] # lsof /opt  | awk '{print $NF}' | xargs ls -rtlh
ls: cannot access NAME: No such file or directory
-rwxr-xr-x 1 itrs      itrs       20M Apr 29  2014 /opt/ITRS/Netprobe/GA3.0.17-140205/netprobe.linux_64
-rwxr-xr-x 1 npelastic npelastic 1.2M Oct 13 13:00 /opt/app/filebeat/config/filebeat-god
-rwxr-xr-x 1 npelastic npelastic 1.2M Oct 13 13:00 /opt/app/metricbeat/config/metricbeat-god
-rwxr-x--- 1 npelastic npelastic  45M Nov 11 02:30 /opt/app/filebeat/bin/filebeat-6.0.0-linux-x86_64/filebeat
-rwxr-x--- 1 npelastic npelastic 105M Nov 11 02:34 /opt/app/metricbeat/bin/metricbeat-6.0.0-linux-x86_64/metricbeat
-rw-r----- 1 itrs      itrs      753K Nov 25 01:04 /opt/ITRS/Netprobe/HKLVAUEDM08/netprobe.log
-rw------- 1 npelastic npelastic 1.3M Dec  8 14:04 /opt/app/metricbeat/logs/metricbeat
-rw------- 1 npelastic npelastic 941K Dec  8 14:05 /opt/app/filebeat/logs/filebeat

/opt/ITRS/Netprobe/HKLVAUEDM08:
total 768K
-rwxr--r-- 1 itrs itrs 1.1K Jul 22 00:49 ntp.sh
-rwxr--r-- 1 itrs itrs 2.1K Jul 22 00:49 DNS_Resolution_check.sh
-rw-r----- 1 itrs itrs 753K Nov 25 01:04 netprobe.log
root@[devops] #

Open in new window

0
BharathKumarRaju DasaraRajuDevops EngineerAuthor Commented:
@Erik,

I don't think it is a problem. Because the size matters here see below output for your info.

root@[devops] # df -PH /opt/bmc/SCBmon
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-scbmonlv  1.1G   35M  969M   4% /opt/bmc/SCBmon
root@[devops] # df -PH /opt
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-optvol  4.1G  4.0G  107M  98% /opt
root@[devops] #

Open in new window

0
arnoldCommented:
please post the output of df -k
Was the system changed in terms of mount points as Eric pointed out earlier.
i.e. you had /opt with all the data and now you added the softwareag overlay without removing the data that was there.

The issue might be a delay in the mount of the softwareag overlay which leads to /opt increasing in space use and is being hidden by the overlay when it does come up.
1
arnoldCommented:
The softwareag folders shows up as 18GB on a 4.1GB partition.
1
BharathKumarRaju DasaraRajuDevops EngineerAuthor Commented:
@arnold,

I also cheked home directory in /etc/passwd its as below...

root@[devops] # cat /etc/passwd | grep -i "/opt"
patrol:x:700:700:BMC Patrol user:/opt/bmc/patrol:/bin/ksh
scbmonlc:x:18562201:100:SCBmon TSS unix:/opt/tss/SCBmon:/bin/bash
root@[devops] # df -PH
Filesystem                                 Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-rootvol                 6.1G  2.3G  3.6G  39% /
tmpfs                                      8.4G     0  8.4G   0% /dev/shm
/dev/sda1                                   98M   80M   13M  87% /boot
/dev/mapper/rootvg-homevol                 4.1G  667M  3.2G  18% /home
/dev/mapper/rootvg-optvol                  4.1G  4.0G  107M  98% /opt
/dev/mapper/rootvg-tmpvol                  4.1G  243M  3.7G   7% /tmp
/dev/mapper/rootvg-oprvol                  317M  8.9M  291M   3% /u/operator
/dev/mapper/rootvg-usrctmvol               3.1G  4.7M  2.9G   1% /usr/controlm
/dev/mapper/rootvg-usrgemsvvol             1.1G  1.4M  969M   1% /usr/gems
/dev/mapper/rootvg-usropenvvol             2.1G  188M  1.8G  10% /usr/openv
/dev/mapper/rootvg-semavol                 1.1G  201M  770M  21% /usr/semasupp
/dev/mapper/rootvg-varvol                  4.1G  352M  3.6G  10% /var
/dev/mapper/HKLVAUEDM08_applvg1-wmstorage   11G  9.8G  109M  99% /wmstorage
/dev/mapper/HKLVAUEDM08_applvg1-isvol1      11G  5.1G  4.9G  52% /opt/softwareag/inst1
/dev/mapper/HKLVAUEDM08_applvg1-isvol2      11G  8.8G  1.2G  89% /opt/softwareag/inst2
/dev/mapper/HKLVAUEDM08_applvg1-isvol3      11G  2.7G  7.3G  27% /opt/softwareag/inst3
/dev/mapper/rootvg-scbmonlv                1.1G   35M  969M   4% /opt/bmc/SCBmon
/dev/mapper/HKLVAUEDM08_applvg1-tsavol      11G  1.9G  8.1G  19% /opt/softwareag/logs
root@[devops] # du -sh /opt/bmc/patrol
20K     /opt/bmc/patrol
root@[devops] # du -sh /opt/tss/SCBmon
92K     /opt/tss/SCBmon
root@[devops] # du -sh /opt/tss/SCBmon/*
60K     /opt/tss/SCBmon/bin
8.0K    /opt/tss/SCBmon/etc
root@[devops] # 

Open in new window

0
BharathKumarRaju DasaraRajuDevops EngineerAuthor Commented:
@Arnold,

Below point might be the scenario in my case, ,What i need to do in this case exactly now :( :(  i am clueless completely

please post the output of df -k
Was the system changed in terms of mount points as Eric pointed out earlier.
i.e. you had /opt with all the data and now you added the softwareag overlay without removing the data that was there.

The issue might be a delay in the mount of the softwareag overlay which leads to /opt increasing in space use and is being hidden by the overlay when it does come up.
0
BharathKumarRaju DasaraRajuDevops EngineerAuthor Commented:
@Arnold,

Please find the output of df -PH  as below..

root@[devops] # df -PH | column -t
Filesystem                                 Size  Used  Avail  Use%  Mounted                on
/dev/mapper/rootvg-rootvol                 6.1G  2.3G  3.6G   39%   /
tmpfs                                      8.4G  0     8.4G   0%    /dev/shm
/dev/sda1                                  98M   80M   13M    87%   /boot
/dev/mapper/rootvg-homevol                 4.1G  667M  3.2G   18%   /home
/dev/mapper/rootvg-optvol                  4.1G  4.0G  107M   98%   /opt
/dev/mapper/rootvg-tmpvol                  4.1G  243M  3.7G   7%    /tmp
/dev/mapper/rootvg-oprvol                  317M  8.9M  291M   3%    /u/operator
/dev/mapper/rootvg-usrctmvol               3.1G  4.7M  2.9G   1%    /usr/controlm
/dev/mapper/rootvg-usrgemsvvol             1.1G  1.4M  969M   1%    /usr/gems
/dev/mapper/rootvg-usropenvvol             2.1G  188M  1.8G   10%   /usr/openv
/dev/mapper/rootvg-semavol                 1.1G  201M  769M   21%   /usr/semasupp
/dev/mapper/rootvg-varvol                  4.1G  352M  3.6G   10%   /var
/dev/mapper/HKLVAUEDM08_applvg1-wmstorage  11G   9.8G  108M   99%   /wmstorage
/dev/mapper/HKLVAUEDM08_applvg1-isvol1     11G   5.1G  4.9G   52%   /opt/softwareag/inst1
/dev/mapper/HKLVAUEDM08_applvg1-isvol2     11G   8.8G  1.2G   89%   /opt/softwareag/inst2
/dev/mapper/HKLVAUEDM08_applvg1-isvol3     11G   2.7G  7.3G   27%   /opt/softwareag/inst3
/dev/mapper/rootvg-scbmonlv                1.1G  35M   969M   4%    /opt/bmc/SCBmon
/dev/mapper/HKLVAUEDM08_applvg1-tsavol     11G   1.9G  8.1G   19%   /opt/softwareag/logs
root@[devops] #

Open in new window

0
BharathKumarRaju DasaraRajuDevops EngineerAuthor Commented:
@Arnold,

Regarding your comment below...please find the output as below..

The softwareag folders shows up as 18GB on a 4.1GB partition


root@[opt] # pwd
/opt
root@[opt] # du -sh *
222M    app
28K     bmc
97M     ITRS
16K     lost+found
18G     softwareag
336K    sysadmin
3.0M    tss
root@[opt] #

Open in new window

0
BharathKumarRaju DasaraRajuDevops EngineerAuthor Commented:
@arnold...

See my /dev/mapper block devices as below..


root@[mapper] # df -PH | column -t
Filesystem                                 Size  Used  Avail  Use%  Mounted                on
/dev/mapper/rootvg-rootvol                 6.1G  2.3G  3.6G   39%   /
tmpfs                                      8.4G  0     8.4G   0%    /dev/shm
/dev/sda1                                  98M   80M   13M    87%   /boot
/dev/mapper/rootvg-homevol                 4.1G  667M  3.2G   18%   /home
/dev/mapper/rootvg-optvol                  4.1G  4.0G  107M   98%   /opt
/dev/mapper/rootvg-tmpvol                  4.1G  243M  3.7G   7%    /tmp
/dev/mapper/rootvg-oprvol                  317M  8.9M  291M   3%    /u/operator
/dev/mapper/rootvg-usrctmvol               3.1G  4.7M  2.9G   1%    /usr/controlm
/dev/mapper/rootvg-usrgemsvvol             1.1G  1.4M  969M   1%    /usr/gems
/dev/mapper/rootvg-usropenvvol             2.1G  188M  1.8G   10%   /usr/openv
/dev/mapper/rootvg-semavol                 1.1G  201M  769M   21%   /usr/semasupp
/dev/mapper/rootvg-varvol                  4.1G  352M  3.5G   10%   /var
/dev/mapper/HKLVAUEDM08_applvg1-wmstorage  11G   9.8G  99M    100%  /wmstorage
/dev/mapper/HKLVAUEDM08_applvg1-isvol1     11G   5.1G  4.9G   52%   /opt/softwareag/inst1
/dev/mapper/HKLVAUEDM08_applvg1-isvol2     11G   8.8G  1.2G   89%   /opt/softwareag/inst2
/dev/mapper/HKLVAUEDM08_applvg1-isvol3     11G   2.7G  7.3G   27%   /opt/softwareag/inst3
/dev/mapper/rootvg-scbmonlv                1.1G  35M   969M   4%    /opt/bmc/SCBmon
/dev/mapper/HKLVAUEDM08_applvg1-tsavol     11G   1.9G  8.1G   19%   /opt/softwareag/logs
root@[mapper] #



root@[opt] # cd /dev/mapper/
root@[mapper] # ls -rtlh
total 0
crw-rw---- 1 root root 10, 56 Jul 22 00:48 control
lrwxrwxrwx 1 root root      7 Jul 22 00:48 rootvg-rootvolsnap-cow -> ../dm-2
lrwxrwxrwx 1 root root      7 Jul 22 00:48 rootvg-rootvol-real -> ../dm-0
lrwxrwxrwx 1 root root      7 Jul 22 00:48 rootvg-rootvolsnap -> ../dm-3
lrwxrwxrwx 1 root root      7 Jul 22 00:48 rootvg-swapvol -> ../dm-4
lrwxrwxrwx 1 root root      7 Jul 22 00:48 HKLVAUEDM08_applvg1-isvol2 -> ../dm-7
lrwxrwxrwx 1 root root      7 Jul 22 00:48 HKLVAUEDM08_applvg1-tsavol -> ../dm-9
lrwxrwxrwx 1 root root      7 Jul 22 00:48 HKLVAUEDM08_applvg1-wmstorage -> ../dm-5
lrwxrwxrwx 1 root root      7 Jul 22 00:48 HKLVAUEDM08_applvg1-isvol3 -> ../dm-8
lrwxrwxrwx 1 root root      7 Jul 22 00:48 HKLVAUEDM08_applvg1-isvol1 -> ../dm-6
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-optvolsnap-cow -> ../dm-12
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-optvol-real -> ../dm-10
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-optvolsnap -> ../dm-13
lrwxrwxrwx 1 root root      7 Jul 22 00:48 rootvg-rootvol -> ../dm-1
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-homevol -> ../dm-19
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-tmpvol -> ../dm-18
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-oprvol -> ../dm-23
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-usrctmvol -> ../dm-24
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-usrgemsvvol -> ../dm-21
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-usropenvvol -> ../dm-20
lrwxrwxrwx 1 root root      8 Jul 22 00:48 rootvg-semavol -> ../dm-22
lrwxrwxrwx 1 root root      8 Sep 13 20:30 rootvg-scbmonlv -> ../dm-25
lrwxrwxrwx 1 root root      8 Nov  7 03:07 rootvg-varvol-real -> ../dm-14
lrwxrwxrwx 1 root root      8 Nov  7 03:07 rootvg-varvolsnap-cow -> ../dm-16
lrwxrwxrwx 1 root root      8 Nov  7 03:07 rootvg-varvol -> ../dm-15
lrwxrwxrwx 1 root root      8 Nov  7 03:07 rootvg-varvolsnap -> ../dm-17
lrwxrwxrwx 1 root root      8 Dec  7 17:39 rootvg-optvol -> ../dm-11
root@[mapper] #

Open in new window

0
BharathKumarRaju DasaraRajuDevops EngineerAuthor Commented:
@Arnold,

Here are my blocks in /proc/partitions

root@[mapper] # cat /proc/partitions
major minor  #blocks  name

   8       16   52428800 sdb
   8       17   52428784 sdb1
   8        0   52428800 sda
   8        1     102400 sda1
   8        2   52325376 sda2
 253        0    6160384 dm-0
 253        1    6160384 dm-1
 253        2    3047424 dm-2
 253        3    6160384 dm-3
 253        4    6160384 dm-4
 253        5   10481664 dm-5
 253        6   10485760 dm-6
 253        7   10485760 dm-7
 253        8   10485760 dm-8
 253        9   10485760 dm-9
 253       10    4096000 dm-10
 253       11    4096000 dm-11
 253       12    4128768 dm-12
 253       13    4096000 dm-13
 253       14    4096000 dm-14
 253       15    4096000 dm-15
 253       16    4128768 dm-16
 253       17    4096000 dm-17
 253       18    4096000 dm-18
 253       19    4096000 dm-19
 253       20    2064384 dm-20
 253       21    1048576 dm-21
 253       22    1048576 dm-22
 253       23     327680 dm-23
 253       24    3080192 dm-24
 253       25    1048576 dm-25
root@[mapper] #

Open in new window

0
arnoldCommented:
were changes made recently to add the extra partitions for softwareag/inst*?

du -ks /opt/softwareag

sh follows hard links which mount points are.
try
cd  /opt
ls | while read a; do
echo -e "$a has: "
du -ks "$a"
done 2>/dev/null

please post the output let see if that sheds light on the matter.

In short, your space is consumed by an overlay i.e. at some point, one of the /opt related mount points was not mounted likely/possibly the /opt/softwareag/logs and that could be where this space is consumed.

If you can, unmount /opt/softwareag/logs and see if /opt/softwareag/logs has data .
make sure the change does not impact production....
What is the source of these sofwareag mount overlays ?volume?
0
BharathKumarRaju DasaraRajuDevops EngineerAuthor Commented:
@armold,

Yes there were new partitions added it seems.

root@[mapper] # du -ks /opt/softwareag
17871156        /opt/softwareag
root@[mapper] # cd /opt
root@[opt] # ls | while read a; do
> echo -e "$a has: "
> du -ks "$a"
> done 2>/dev/null
app has:
226636  app
bmc has:
28      bmc
ITRS has:
99064   ITRS
lost+found has:
16      lost+found
softwareag has:
17871172        softwareag
sysadmin has:
336     sysadmin
tss has:
3032    tss
root@[opt] #

Open in new window



for your info please find fdisk output too.

root@[mapper] # df -PH | column -t | grep -i opt
Filesystem                                 Size  Used  Avail  Use%  Mounted                on
/dev/mapper/rootvg-optvol                  4.1G  4.0G  107M   98%   /opt
/dev/mapper/_applvg1-isvol1     11G   5.1G  4.9G   52%   /opt/softwareag/inst1
/dev/mapper/_applvg1-isvol2     11G   8.8G  1.2G   89%   /opt/softwareag/inst2
/dev/mapper/_applvg1-isvol3     11G   2.7G  7.3G   27%   /opt/softwareag/inst3
/dev/mapper/rootvg-scbmonlv                1.1G  35M   969M   4%    /opt/bmc/SCBmon
/dev/mapper/_applvg1-tsavol     11G   1.9G  8.1G   19%   /opt/softwareag/logs
root@[mapper] # fdisk -l /dev/mapper/rootvg-optvol
Disk /dev/mapper/rootvg-optvol: 4194 MB, 4194304000 bytes
255 heads, 63 sectors/track, 509 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

root@[mapper] # fdisk -l /dev/mapper/_applvg1-isvol1
Disk /dev/mapper/_applvg1-isvol1: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

root@[mapper] # fdisk -l /dev/mapper/_applvg1-isvol2
Disk /dev/mapper/_applvg1-isvol2: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

root@[mapper] # fdisk -l /dev/mapper/_applvg1-isvol3
Disk /dev/mapper/_applvg1-isvol3: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

root@[mapper] # fdisk -l /dev/mapper/rootvg-scbmonlv
Disk /dev/mapper/rootvg-scbmonlv: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

root@[mapper] # fdisk -l /dev/mapper/_applvg1-tsavol
Disk /dev/mapper/_applvg1-tsavol: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
root@[mapper] #

Open in new window

0
BharathKumarRaju DasaraRajuDevops EngineerAuthor Commented:
@Arnold,

Yes it is my production box, That's why i am scared to take any risk :( :(

The source for softwareag mount is as below VG....is there anyway i can remove somespace to reduce /opt space ?

root@[opt] # df -PH
Filesystem                                 Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-rootvol                 6.1G  2.3G  3.6G  39% /
tmpfs                                      8.4G     0  8.4G   0% /dev/shm
/dev/sda1                                   98M   80M   13M  87% /boot
/dev/mapper/rootvg-homevol                 4.1G  667M  3.2G  18% /home
/dev/mapper/rootvg-optvol                  4.1G  4.0G  107M  98% /opt
/dev/mapper/rootvg-tmpvol                  4.1G  243M  3.7G   7% /tmp
/dev/mapper/rootvg-oprvol                  317M  8.9M  291M   3% /u/operator
/dev/mapper/rootvg-usrctmvol               3.1G  4.7M  2.9G   1% /usr/controlm
/dev/mapper/rootvg-usrgemsvvol             1.1G  1.4M  969M   1% /usr/gems
/dev/mapper/rootvg-usropenvvol             2.1G  188M  1.8G  10% /usr/openv
/dev/mapper/rootvg-semavol                 1.1G  201M  769M  21% /usr/semasupp
/dev/mapper/rootvg-varvol                  4.1G  352M  3.5G  10% /var
/dev/mapper/HKLVAUEDM08_applvg1-wmstorage   11G  9.9G   17M 100% /wmstorage
/dev/mapper/HKLVAUEDM08_applvg1-isvol1      11G  5.1G  4.9G  52% /opt/softwareag/inst1
/dev/mapper/HKLVAUEDM08_applvg1-isvol2      11G  8.9G  1.1G  89% /opt/softwareag/inst2
/dev/mapper/HKLVAUEDM08_applvg1-isvol3      11G  2.7G  7.3G  27% /opt/softwareag/inst3
/dev/mapper/rootvg-scbmonlv                1.1G   35M  969M   4% /opt/bmc/SCBmon
/dev/mapper/HKLVAUEDM08_applvg1-tsavol      11G  1.9G  8.0G  20% /opt/softwareag/logs
root@[opt] # vgs
  VG                  #PV #LV #SN Attr   VSize  VFree
  HKLVAUEDM08_applvg1   1   5   0 wz--n- 50.00g    0
  rootvg                1  15   3 wz--n- 49.88g 3.50g
root@[opt] # lvs
  LV          VG                  Attr       LSize   Pool Origin  Data%  Meta%  Move Log Cpy%Sync Convert
  isvol1      HKLVAUEDM08_applvg1 -wi-ao----  10.00g
  isvol2      HKLVAUEDM08_applvg1 -wi-ao----  10.00g
  isvol3      HKLVAUEDM08_applvg1 -wi-ao----  10.00g
  tsavol      HKLVAUEDM08_applvg1 -wi-ao----  10.00g
  wmstorage   HKLVAUEDM08_applvg1 -wi-ao----  10.00g
  homevol     rootvg              -wi-ao----   3.91g
  oprvol      rootvg              -wi-ao---- 320.00m
  optvol      rootvg              owi-aos---   3.91g
  optvolsnap  rootvg              swi-a-s---   3.94g      optvol  95.39
  rootvol     rootvg              owi-aos---   5.88g
  rootvolsnap rootvg              swi-a-s---   2.91g      rootvol 69.49
  scbmonlv    rootvg              -wi-ao----   1.00g
  semavol     rootvg              -wi-ao----   1.00g
  swapvol     rootvg              -wi-ao----   5.88g
  tmpvol      rootvg              -wi-ao----   3.91g
  usrctmvol   rootvg              -wi-ao----   2.94g
  usrgemsvvol rootvg              -wi-ao----   1.00g
  usropenvvol rootvg              -wi-ao----   1.97g
  varvol      rootvg              owi-aos---   3.91g
  varvolsnap  rootvg              swi-a-s---   3.94g      varvol  88.22
root@[opt] #

Open in new window

0
arnoldCommented:
the only possible location based on current usage, inst3 and logs within softwareag ..

 the only way is to refer  to Eric's point that the /opt filled up and inst3 or logs was added without actually freeing the space consumed there.

I would look at logs first.

What gets files into /opt/softwareag/logs ?
lsof /opt/softwareag/logs?
0
BharathKumarRaju DasaraRajuDevops EngineerAuthor Commented:
@Arnold,

How you identified that the /opt space consumed by inst3 and logs partition....please find below lsof ouput on logs



root@[opt] # lsof /opt/softwareag/logs
COMMAND     PID     USER   FD   TYPE DEVICE  SIZE/OFF   NODE NAME
server.sh  6644 webmadmn    1w   REG  253,9    749568 526876 /opt/softwareag/logs/inst1/server.out
server.sh  6644 webmadmn    2w   REG  253,9    749568 526876 /opt/softwareag/logs/inst1/server.out
sagis95    6677 webmadmn    1w   REG  253,9    749568 526876 /opt/softwareag/logs/inst1/server.out
sagis95    6677 webmadmn    2w   REG  253,9    749568 526876 /opt/softwareag/logs/inst1/server.out
wrapper-3  6731 webmadmn    1w   REG  253,9    749568 526876 /opt/softwareag/logs/inst1/server.out
wrapper-3  6731 webmadmn    2w   REG  253,9    749568 526876 /opt/softwareag/logs/inst1/server.out
java       6733 webmadmn  159w   REG  253,9    137415 524292 /opt/softwareag/logs/inst1/stats.log
java       6733 webmadmn  492w   REG  253,9   1611675 524318 /opt/softwareag/logs/inst1/server.log
java       6733 webmadmn  496w   REG  253,9         0 524317 /opt/softwareag/logs/inst1/FailedAudit_20171208_000000.log
java       6733 webmadmn  497w   REG  253,9         0 524296 /opt/softwareag/logs/inst1/WMTXOUT_20171208_000000.log
java       6733 webmadmn  512w   REG  253,9    843200 524299 /opt/softwareag/logs/inst1/WMSESSION_20171208_000000.log
java       6733 webmadmn  514w   REG  253,9         0 524300 /opt/softwareag/logs/inst1/WMTXIN_20171208_000000.log
java       6733 webmadmn  539w   REG  253,9         0 524313 /opt/softwareag/logs/inst1/WMERROR_20171208_000000.log
java       8004 webmadmn   24w   REG  253,9    180644     62 /opt/softwareag/logs/inst2/stats.log
java       8004 webmadmn  159w   REG  253,9   9478031     15 /opt/softwareag/logs/inst2/server.log
java       8004 webmadmn  172w   REG  253,9   1663101     19 /opt/softwareag/logs/inst2/WMSECURITY_20171208_000000.log
java       8004 webmadmn  770w   REG  253,9         0     63 /opt/softwareag/logs/inst2/WMTXOUT_20171208_000000.log
java       8004 webmadmn  773w   REG  253,9  20135616     64 /opt/softwareag/logs/inst2/WMSESSION_20171208_000000.log
java       8004 webmadmn  792u   REG  253,9         0     65 /opt/softwareag/logs/inst2/WMTXIN_20171208_000000.log
java       8004 webmadmn  793u   REG  253,9   1489440     67 /opt/softwareag/logs/inst2/WMSERVICE_20171208_000000.log
java       8004 webmadmn  795w   REG  253,9  15762912     68 /opt/softwareag/logs/inst2/WMERROR_20171208_000000.log
java       8004 webmadmn  798u   REG  253,9         0     69 /opt/softwareag/logs/inst2/FailedAudit_20171208_000000.log
server.sh 25743 webmadmn    1w   REG  253,9 113451204     12 /opt/softwareag/logs/inst2/server.out
server.sh 25743 webmadmn    2w   REG  253,9 113451204     12 /opt/softwareag/logs/inst2/server.out
sagis95   25776 webmadmn    1w   REG  253,9 113451204     12 /opt/softwareag/logs/inst2/server.out
sagis95   25776 webmadmn    2w   REG  253,9 113451204     12 /opt/softwareag/logs/inst2/server.out
wrapper-3 25830 webmadmn    1w   REG  253,9 113451204     12 /opt/softwareag/logs/inst2/server.out
wrapper-3 25830 webmadmn    2w   REG  253,9 113451204     12 /opt/softwareag/logs/inst2/server.out
root@[opt] #

Open in new window

0
arnoldCommented:
Total space of opt is 4.1Gb
Logs consumes 1.9GB
Inst3 has 2.7GB

If partitions are added sequentially, inst1, inst2 may have been added before inst3, logs such that at one point data in those locations was taken from the /opt filesystem space

Of course it can also be under bmc or the other /opt mounts, but the utilization of that space does not seem to grow such that if there are sone data remnants there they are likely small.

Relying on Eric's observation, looking under the /opt related mount points is when one would find this missing space as you do not seem to have a large open file that is deleted as far as the filesystem is concerned. A restart of services running and relying on /opt partition for logs etc would release the file handle thus releasing the space.
1
BharathKumarRaju DasaraRajuDevops EngineerAuthor Commented:
@arnold,

Thanks for your valuable suggestions...

So I tried to delete some logs under  inst3 and logs mount point but still /opt showing same usage I.e. 98%

You mean to say if restart a application service in inst3 and logs mount point it may release the space in /opt right
0
Erik SchminkeAIX/Linux Systems AdministratorCommented:
You need to unmount the other file systems and re-examine space usage.  Here's why:

I setup a disk allocation/file system layout similar to what you show:
[root@localhost ~]# pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created.
[root@localhost ~]# vgcreate xxdemo /dev/sdb
  Volume group "xxdemo" successfully created
[root@localhost ~]# vgcreate xxdemo /dev/sdb
  Volume group "xxdemo" successfully created
[root@localhost ~]# lvcreate -n xxopt -L 2G xxdemo
  Logical volume "xxopt" created.
[root@localhost ~]# lvcreate -n xxlog -L 512M xxdemo
  Logical volume "xxlog" created.
[root@localhost ~]# lvcreate -n xxsbcm -L 512M xxdemo
  Logical volume "xxsbcm" created.
[root@localhost ~]# lvcreate -n xxswi1 -L 512M xxdemo
  Logical volume "xxswi1" created.
[root@localhost ~]# lvcreate -n xxswi2 -L 512M xxdemo
  Logical volume "xxswi2" created.
[root@localhost ~]# lvcreate -n xxswi3 -L 512M xxdemo
  Logical volume "xxswi3" created.
[root@localhost ~]# lvs
  LV     VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root   centos -wi-ao---- <55.50g                                                    
  swap   centos -wi-ao----   8.00g                                                    
  xxlog  xxdemo -wi-a----- 512.00m                                                    
  xxopt  xxdemo -wi-a-----   2.00g                                                    
  xxswi1 xxdemo -wi-a----- 512.00m                                                    
  xxswi2 xxdemo -wi-a----- 512.00m                                                    
  xxswi3 xxdemo -wi-a----- 512.00m                                                    
[root@localhost ~]# mkfs.ext4 /dev/mapper/xxdemo-xxopt 
[root@localhost ~]# mkfs.ext4 /dev/mapper/xxdemo-xxlog 
[root@localhost ~]# mkfs.ext4 /dev/mapper/xxdemo-xxsbcm 
[root@localhost ~]# mkfs.ext4 /dev/mapper/xxdemo-xxswi1 
[root@localhost ~]# mkfs.ext4 /dev/mapper/xxdemo-xxswi2
[root@localhost ~]# mkfs.ext4 /dev/mapper/xxdemo-xxswi3

Open in new window


I mounted this file system at /opt:
[root@localhost ~]# mount /dev/mapper/xxdemo-xxopt /opt
[root@localhost ~]#

Open in new window


I created a 128M file and the directories to later serve as mount points for other filesystems inside /opt:
[root@localhost ~]# dd if=/dev/zero of=/opt/file1 bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB) copied, 0.260778 s, 515 MB/s
[root@localhost ~]# mkdir -p /opt/bmc/SCBmon
[root@localhost ~]# mkdir -p /opt/softwareag/inst1
[root@localhost ~]# mkdir -p /opt/softwareag/inst2
[root@localhost ~]# mkdir -p /opt/softwareag/inst3
[root@localhost ~]# mkdir -p /opt/softwareag/logs
[root@localhost ~]#

Open in new window


My /opt usage is now only ~129M, shown by both the du and df commands:
[root@localhost ~]# df -h /opt
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/xxdemo-xxopt  2.0G  135M  1.7G   8% /opt
[root@localhost ~]# du -sh /opt
129M	/opt
[root@localhost ~]#

Open in new window


Now I create a very large file within /opt inside one of the directories that will later be a mount point:
[root@localhost ~]# dd if=/dev/zero of=/opt/softwareag/inst1/mount-masked-file bs=1M count=1280
1280+0 records in
1280+0 records out
1342177280 bytes (1.3 GB) copied, 2.4624 s, 545 MB/s
[root@localhost ~]# 

Open in new window


My df and du commands still agree, there is now approx 1.4G of space used.
[root@localhost ~]# df -h /opt
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/xxdemo-xxopt  2.0G  1.4G  420M  78% /opt
[root@localhost ~]# du -sh /opt
1.4G	/opt
[root@localhost ~]# 

Open in new window


Here is a listing of all files inside /opt:
[root@localhost ~]# find /opt -exec ls -ld {} \;
drwxr-xr-x. 5 root root 4096 Dec  8 07:28 /opt
drwxr-xr-x. 6 root root 4096 Dec  8 07:28 /opt/softwareag
drwxr-xr-x. 2 root root 4096 Dec  8 07:28 /opt/softwareag/inst2
drwxr-xr-x. 2 root root 4096 Dec  8 07:28 /opt/softwareag/logs
drwxr-xr-x. 2 root root 4096 Dec  8 07:50 /opt/softwareag/inst1
-rw-r--r--. 1 root root 1342177280 Dec  8 07:51 /opt/softwareag/inst1/mount-masked-file
drwxr-xr-x. 2 root root 4096 Dec  8 07:28 /opt/softwareag/inst3
drwx------. 2 root root 16384 Dec  8 07:23 /opt/lost+found
-rw-r--r--. 1 root root 134217728 Dec  8 07:27 /opt/file1
drwxr-xr-x. 3 root root 4096 Dec  8 07:28 /opt/bmc
drwxr-xr-x. 2 root root 4096 Dec  8 07:28 /opt/bmc/SCBmon
[root@localhost ~]# 

Open in new window


Now I'm going to mount the additional file systems within /opt, like you have in your environment:
[root@localhost ~]# mount /dev/mapper/xxdemo-xxsbcm /opt/bmc/SCBmon/
[root@localhost ~]# mount /dev/mapper/xxdemo-xxlog /opt/softwareag/logs/
[root@localhost ~]# mount /dev/mapper/xxdemo-xxswi1 /opt/softwareag/inst1/
[root@localhost ~]# mount /dev/mapper/xxdemo-xxswi2 /opt/softwareag/inst2/
[root@localhost ~]# mount /dev/mapper/xxdemo-xxswi3 /opt/softwareag/inst3/
[root@localhost ~]# df -h
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/centos-root     56G   19G   38G  34% /
devtmpfs                   3.9G     0  3.9G   0% /dev
tmpfs                      3.9G     0  3.9G   0% /dev/shm
tmpfs                      3.9G  8.8M  3.9G   1% /run
tmpfs                      3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                  509M  367M  143M  73% /boot
tmpfs                      783M   28K  783M   1% /run/user/1042
/dev/mapper/xxdemo-xxopt   2.0G  1.4G  420M  78% /opt
/dev/mapper/xxdemo-xxsbcm  488M  780K  452M   1% /opt/bmc/SCBmon
/dev/mapper/xxdemo-xxlog   488M  780K  452M   1% /opt/softwareag/logs
/dev/mapper/xxdemo-xxswi1  488M  780K  452M   1% /opt/softwareag/inst1
/dev/mapper/xxdemo-xxswi2  488M  780K  452M   1% /opt/softwareag/inst2
/dev/mapper/xxdemo-xxswi3  488M  780K  452M   1% /opt/softwareag/inst3
[root@localhost ~]#

Open in new window



Notice, that the space usage in /opt has not changed, but also notice what the du command now says about /opt:
[root@localhost ~]# df -h /opt
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/xxdemo-xxopt   2.0G  1.4G  420M  78% /opt
[root@localhost ~]# du -sh /opt
129M	/opt
[root@localhost ~]#

Open in new window


Look at how the file listing has changed now.  Notice that the large file is not listed:
[root@localhost ~]# find /opt -exec ls -ld {} \;
drwxr-xr-x. 5 root root 4096 Dec  8 07:28 /opt
drwxr-xr-x. 6 root root 4096 Dec  8 07:28 /opt/softwareag
drwxr-xr-x. 3 root root 4096 Dec  8 07:24 /opt/softwareag/inst2
drwx------. 2 root root 16384 Dec  8 07:24 /opt/softwareag/inst2/lost+found
drwxr-xr-x. 3 root root 4096 Dec  8 07:24 /opt/softwareag/logs
drwx------. 2 root root 16384 Dec  8 07:24 /opt/softwareag/logs/lost+found
drwxr-xr-x. 3 root root 4096 Dec  8 07:24 /opt/softwareag/inst1
drwx------. 2 root root 16384 Dec  8 07:24 /opt/softwareag/inst1/lost+found
drwxr-xr-x. 3 root root 4096 Dec  8 07:24 /opt/softwareag/inst3
drwx------. 2 root root 16384 Dec  8 07:24 /opt/softwareag/inst3/lost+found
drwx------. 2 root root 16384 Dec  8 07:23 /opt/lost+found
-rw-r--r--. 1 root root 134217728 Dec  8 07:27 /opt/file1
drwxr-xr-x. 3 root root 4096 Dec  8 07:28 /opt/bmc
drwxr-xr-x. 3 root root 4096 Dec  8 07:26 /opt/bmc/SCBmon
drwx------. 2 root root 16384 Dec  8 07:26 /opt/bmc/SCBmon/lost+found
[root@localhost ~]# 

Open in new window


The large file that was inside the one of the directories that got a new file system mounted, is now masked from the du command.  Even though another file system is on top of it, it is still there.

As I said at the outset.. you need to unmount the other file systems.  You will probably need to take an outage to do it, because if there are any open files in those other file systems, it won't let you unmount them until they are closed.
1

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Erik SchminkeAIX/Linux Systems AdministratorCommented:
Also, yes.. you can reduce the size of overallocated filesystems, but they must be done offline (unmounted); on-line shrinking is not supported.

Also... be very careful.  Doing this incorrectly, could destroy data.  I've done it.

Here I attempted to reduce the size of the file system.  I was prevented from doing so, because the file system is mounted:
[root@localhost ~]# resize2fs /dev/mapper/xxdemo-xxsbcm 256M
resize2fs 1.42.9 (28-Dec-2013)
Filesystem at /dev/mapper/xxdemo-xxsbcm is mounted on /opt/bmc/SCBmon; on-line resizing required
resize2fs: On-line shrinking not supported
[root@localhost ~]#

Open in new window


Here's how to do it.

First, I'm going to create some random data on the file system I'm going to reduce to demonstrate I haven't destroyed anything:
[root@localhost ~]# dd if=/dev/urandom of=/opt/bmc/SCBmon/important-file bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB) copied, 1.0323 s, 130 MB/s
[root@localhost ~]# sha256sum /opt/bmc/SCBmon/important-file
1a465af14aca84c130d5d22157d50ef59826caa4342462d4351c8996e1f67c8d  /opt/bmc/SCBmon/important-file
[root@localhost ~]# 

Open in new window


Notice, the sha256 checksum of my "important-file"

Now, unmount the file system you want to reduce:
[root@localhost ~]# umount /opt/bmc/SCBmon/
[root@localhost ~]#

Open in new window


Here is the current state of my demo volume groups and logical volumes:
[root@localhost ~]# vgs
  VG     #PV #LV #SN Attr   VSize   VFree 
  centos   1   2   0 wz--n- <63.50g     0 
  xxdemo   1   6   0 wz--n-  <8.00g <3.50g
[root@localhost ~]# lvs
  LV     VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root   centos -wi-ao---- <55.50g                                                    
  swap   centos -wi-ao----   8.00g                                                    
  xxlog  xxdemo -wi-ao---- 512.00m                                                    
  xxopt  xxdemo -wi-ao----   2.00g                                                    
  xxsbcm xxdemo -wi-a----- 512.00m                                                    
  xxswi1 xxdemo -wi-ao---- 512.00m                                                    
  xxswi2 xxdemo -wi-ao---- 512.00m                                                    
  xxswi3 xxdemo -wi-ao---- 512.00m                                                    
[root@localhost ~]#

Open in new window


Reduce the LVM logical volume:
[root@localhost ~]# lvreduce --resizefs --size 256M /dev/mapper/xxdemo-xxsbcm 
fsck from util-linux 2.23.2
/dev/mapper/xxdemo-xxsbcm: clean, 12/32768 files, 39121/131072 blocks
resize2fs 1.42.9 (28-Dec-2013)
Resizing the filesystem on /dev/mapper/xxdemo-xxsbcm to 65536 (4k) blocks.
The filesystem on /dev/mapper/xxdemo-xxsbcm is now 65536 blocks long.

  Size of logical volume xxdemo/xxsbcm changed from 512.00 MiB (128 extents) to 256.00 MiB (64 extents).
  Logical volume xxdemo/xxsbcm successfully resized.
[root@localhost ~]#

Open in new window


Mount the file system and check integrity of the important-file
[root@localhost ~]# mount /dev/mapper/xxdemo-xxsbcm /opt/bmc/SCBmon
[root@localhost ~]# sha256sum /opt/bmc/SCBmon/important-file 
1a465af14aca84c130d5d22157d50ef59826caa4342462d4351c8996e1f67c8d  /opt/bmc/SCBmon/important-file
[root@localhost ~]# 

Open in new window


My important-file is intact.  The check sums are the same.

I now have an extra 256M on xxdemo to play with:
[root@localhost ~]# vgs
  VG     #PV #LV #SN Attr   VSize   VFree 
  centos   1   2   0 wz--n- <63.50g     0 
  xxdemo   1   6   0 wz--n-  <8.00g <3.75g
[root@localhost ~]# lvs
  LV     VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root   centos -wi-ao---- <55.50g                                                    
  swap   centos -wi-ao----   8.00g                                                    
  xxlog  xxdemo -wi-ao---- 512.00m                                                    
  xxopt  xxdemo -wi-ao----   2.00g                                                    
  xxsbcm xxdemo -wi-a----- 256.00m                                                    
  xxswi1 xxdemo -wi-ao---- 512.00m                                                    
  xxswi2 xxdemo -wi-ao---- 512.00m                                                    
  xxswi3 xxdemo -wi-ao---- 512.00m                                                    
[root@localhost ~]# 

Open in new window


With that extra space, I'll grow my logs directory:
[root@localhost ~]# lvextend --size 768M /dev/mapper/xxdemo-xxlog 
  Size of logical volume xxdemo/xxlog changed from 512.00 MiB (128 extents) to 768.00 MiB (192 extents).
  Logical volume xxdemo/xxlog successfully resized.
[root@localhost ~]# resize2fs /dev/mapper/xxdemo-xxlog 
resize2fs 1.42.9 (28-Dec-2013)
Filesystem at /dev/mapper/xxdemo-xxlog is mounted on /opt/softwareag/logs; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/mapper/xxdemo-xxlog is now 196608 blocks long.

[root@localhost ~]# df -h /opt/softwareag/logs
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/xxdemo-xxlog  740M  1.1M  693M   1% /opt/softwareag/logs
[root@localhost ~]# 

Open in new window


Take a look at my volume groups and logical volumes again:
[root@localhost ~]# vgs
  VG     #PV #LV #SN Attr   VSize   VFree 
  centos   1   2   0 wz--n- <63.50g     0 
  xxdemo   1   6   0 wz--n-  <8.00g <3.50g
[root@localhost ~]# lvs
  LV     VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root   centos -wi-ao---- <55.50g                                                    
  swap   centos -wi-ao----   8.00g                                                    
  xxlog  xxdemo -wi-ao---- 768.00m                                                    
  xxopt  xxdemo -wi-ao----   2.00g                                                    
  xxsbcm xxdemo -wi-a----- 256.00m                                                    
  xxswi1 xxdemo -wi-ao---- 512.00m                                                    
  xxswi2 xxdemo -wi-ao---- 512.00m                                                    
  xxswi3 xxdemo -wi-ao---- 512.00m                                                    
[root@localhost ~]#

Open in new window


Here's how NOT to do it...

If you specify a size that is smaller than the current size of the file system and use the --resizefs option.. the command will fail:
[root@localhost ~]# lvreduce --resizefs --size 16M /dev/mapper/xxdemo-xxsbcm 
fsck from util-linux 2.23.2
/dev/mapper/xxdemo-xxsbcm: 12/16384 files (8.3% non-contiguous), 38028/65536 blocks
resize2fs 1.42.9 (28-Dec-2013)
resize2fs: New size smaller than minimum (45747)

fsadm: Resize ext4 failed.
  /usr/sbin/fsadm failed: 1
  Filesystem resize failed.
[root@localhost ~]#

Open in new window


If you specify a size that is smaller than the current size of the file system and don't use the --resizefs option.. the command will ask you if you're sure:
[root@localhost ~]# lvreduce --size 16M /dev/mapper/xxdemo-xxsbcm 
  WARNING: Reducing active logical volume to 16.00 MiB.
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce xxdemo/xxsbcm? [y/n]: y
  Size of logical volume xxdemo/xxsbcm changed from 256.00 MiB (64 extents) to 16.00 MiB (4 extents).
  Logical volume xxdemo/xxsbcm successfully resized.
[root@localhost ~]#

Open in new window


Try to remount your file system..
[root@localhost ~]# mount /dev/mapper/xxdemo-xxsbcm /opt/bmc/SCBmon/
mount: wrong fs type, bad option, bad superblock on /dev/mapper/xxdemo-xxsbcm,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
[root@localhost ~]#

Open in new window


Now you've got problems...  be very careful what you type when you start reducing file systems.

In case anyone is interested... my full command history/shell output as I was putting this together is attached in the plain text file...  it's >1200 lines... God bless you if you can follow it :)
command_history.txt
1
arnoldCommented:
when using LVM, you first use lvresize to adjust the lv. resizefs is the last thing one runs.
if adding
pvcreate to add a new resource (/dev/sdx)
vgextend to expand the VolumeGroup (/dev/sdx as a resource on which this volume group)
lvresize  to insrease the logical volume (lvextend) only reference the volumegroup and logical volume ...
resizefs (depending on the filesystem in use) to adjust the filesystem to match the logical volume.
 lvresize -L -50MB will reduce the logical volume mentioned by 50MB as long as there is 50MB of free space.

When making changes to the filesystem, always make sure you have a good backup.
0
BharathKumarRaju DasaraRajuDevops EngineerAuthor Commented:
This is great explanation Erik...thanks a lot inface it is my production i need to ask downtime to do it :( :( its bit process oriented....thanks a lot arnold as well for his greatest support
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.