Solved

Disk Cleanup - with Active files

Posted on 2011-09-11
8
553 Views
Last Modified: 2013-11-17
There is a procedure that has stirred some debate amongst our team.

We agree here:  If a file is removed while allocated, the space is not freed.
   Example of large file statically allocated:
unix178:# fuser BIG.log
BIG.log:   454734
#
unix178:# ls -l BIG.log
-rw-r--r--    1 ivmgr    ivmgr    1255292755 Sep 11 10:15 BIG.log
#
unix178:# df -Pm .
Filesystem    MB blocks      Used Available Capacity Mounted on
/dev/hd9var     1536.00   1372.28    163.72      90% /var

Open in new window


If BIG.log is removed,  the inode is still allocated, and thus, no disk space is freed, until the process ends, or closes it's files.

Here's the question:  
    We have two methods discussed, to free the disk space.  
    Group 1 does not like touching "active" files with a redirect(>),  
    Group 2 says "it appears to work fine".

Here's the steps for the two methods.  
Method 1:
     mv BIG.log  BIG.log-YYMMDD.HHMMSS
     touch BIG.log             #   (optionally chmod/chown if required to make match original)
     Recycle application  
     mv BIG.log-YYMMDD.HHMMSS   /archive
### the Filesystem space is truly freed.

Method 2:
    cat BIG.log > /archive/BIG.log-YYMMDD.HHMMSS
    > BIG.log          
### The Filesystem space does appear to be freed.

Assuming the risk of loosing a couple of log records using method 2 (that may have been created between the end of the cat command, and the redirect) is acceptable

    Pro for Method 1:  * absolutely no records are lost
                              * No Risk of orphaned inodes (if steps followed properly)
    Con for method 1:  * Causes application outage
                               * More commands, risk of a new admin orphaning an inode.

Is there any reason to avoid method 2?

Is there a risk of "confusing" a program that has a file allocated, by issuing a redirect to it's file?

Thanks,
Tom



0
Comment
Question by:Tomunique
  • 4
  • 2
  • 2
8 Comments
 
LVL 68

Accepted Solution

by:
woolmilkporc earned 500 total points
ID: 36521274
Hi,

if you can afford recycling an application but can't afford losing log data methosd 1 is the one to choose.

But your command sequence seems a bit strange. Why not:

- stop application
cp -p BIG.log  /archive/BIG.log-YYMMDD.HHMMSS
>BIG.log
- start application

I used "cp -p" and ">" instead of "mv" and "touch" to preserve the permissions and timestamps (source and destination).
I prepended /archive to the target of the "cp" step to avoild the final "mv".

Or are you using "mv" to the current dir to make the whole thing as fast as possible? If speed is really a concern you should keep your "mv" version! But splitting "recycle" into "start" and "stop" still seems more practical to me.


The second method has to be chosen if restarting an application is not an option, but losing log data can be tolerated.

Instead of "cat" you can also use "cp -p" here, to preserve permissions and timestamps on the target.

cp -p BIG.log > /archive/BIG.log-YYMMDD.HHMMSS
>BIG.log

And No, the writing program will not be confused by your emptying of the target file.
I also don't see a risk of orphaning inodes, as long as you don't use "rm" anywhere.

wmp
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 36521278
... the ">" in the last "cp" step is of course wrong (copy-and-paste error)! Sorry!
0
 
LVL 40

Expert Comment

by:noci
ID: 36521298
It depends....

- Do logs get used regularly?
- Is your business regulated? [Banking, government...].
In that case you DON't want to loose records.


- If you're hobyist,
- never (realy never) look at logs..
- the software cannot suffer any down time
Who cares.

If you cannot suffer downtime then you obviousely have a bid UPS standing by, with longer term solutions to keep running too...

There is an other solution, use some tool to read from pipelines and create syslog records,
send the syslog records to something like metalog, or syslog-ng  those loggers can change files at intervals, so you have best of all worlds...
0
Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

 
LVL 68

Expert Comment

by:woolmilkporc
ID: 36521483
... or you can use logrotate, which is available for AIX here:

http://www.perzl.org/aix/index.php?n=Main.Logrotate

Install it using the rpm tool which comes with AIX.

There are some dependencies. All required packages are available at the above site http://www.perzl.org/aix/index.php?n=Main.HomePage
as well.

Please note that logrotate can be set up to use either of the methods you mentioned, it's just a matter of configuration.

wmp
0
 
LVL 6

Author Comment

by:Tomunique
ID: 36521503
We're not regulated, but certainly not a hobby... (fortune 500).  The methodology used would have to be selected on a case by case basis, those that can't loose any records, clearly fall to method 1.

It's more concept than specifics.  We have annoying offending apps, but some will be resolved with Version upgrades.  My concern was around method 2, in what appears to be working, could present itself as a nightmare if not used properly (orphaned inodes is our biggest problem).

WMP:
   The reason we do the mv/recycle, is there's two teams involved in the process.  Admin team, and application team.  Historically, we assist managing the logs, as app team isn't unix literate.  But we need them to recycle the app.  So, we prep the environment, ask them to bounce the app.  when we see the logs are free via fuser, we move the off to archive.

If using redirect (>) to truncate a file that another process is writing to is generally acceptable, then I'd say that would make the most sense, and the admin team can put a step in the disk management monitors to do this without the assistance of the app teams.




0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 36521611
Yes, redirect will never do any harm. I (and my logrotate) use it all the time without any issues.

By the way, logrotate is able to recycle applications (i.e. issue arbitrary commands during the rotation process).
Maybe this feature could get you rid of the need to involve your application team, should you decide to use method 1 anyway?

If switching to a particular userid is required for this, "su -" is an option, since logrotate runs with root credentials:

/path/to/BIG.log {
rotate 5
size=100M
olddir /archive
create 544 appl_user appl_group
postrotate
   su - appl_user -c "/command/to/recycle/application"
endscript
}

Or, with a split "recycle":

/path/to/BIG.log {
rotate 5
size=100M
olddir /archive
create 544 appl_user appl_group
prerotate
   su - appl_user -c "/command/to/stop/application"
endscript
postrotate
   su - appl_user -c "/command/to/start/application"
endscript
}


Method 2:

/path/to/BIG.log {
rotate 5
size=100M
olddir /archive
copytruncate
}

Of course, when you say that there are disk monitors in place and that there is some scripting skill - no need for logrotate.

wmp
0
 
LVL 40

Expert Comment

by:noci
ID: 36523372
And finaly be sure that a specification gets accepted to build rotation into any newly developped & bought application.
0
 
LVL 6

Author Closing Comment

by:Tomunique
ID: 36523463
Thanks guys...  appreciate the input.
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
linux  centos   virtualization network quetion 6 81
unix scripting question 1 97
Convert OverPunch 25 64
check unix curl command return value 7 123
Installing FreeBSD… FreeBSD is a darling of an operating system. The stability and usability make it a clear choice for servers and desktops (for the cunning). Savvy?  The Ports collection makes available every popular FOSS application and packag…
I promised to write further about my project, and here I am.  First, I needed to setup the Primary Server.  You can read how in this article: Setup FreeBSD Server with full HDD encryption (http://www.experts-exchange.com/OS/Unix/BSD/FreeBSD/A_3660-S…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

803 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question