Log File Cleanup in Unix Bash Environment

Log file cleanup issue in a Unix bash shell environment.  My development environment consist of about 60-80 J2EE developers at any given time.  Most developer are unit testing and integration their code through the day.  Each developer's execution of their application generates multiple log files.  I need to absolutely insure that prior to each developer's execution run all of their previous set of log files are deleted/removed.  One thing in common the log files for a specific developer all will have the same userid embedded in a set of logs.  I need an bash example of deleting all the common logs before the developer is allowed to invoke their next test run of the application.
LVL 1
sdrussAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

dfkeCommented:
Hi,

can you be more specific about which files need to be removed and their exact locations?

Cheers
0
sdrussAuthor Commented:
Have many log files in (2) different sub-directories for several different users.  File names are mostly randomly created. For example, I use mktemp to generate a unique file suffix:

export BaseNameScript=$(basename ${1})

log=$(mktemp "BaseNameScript.$(userid).XXXX)

Example file names (could be several hundred similarly named):

CreateDb4hSchema.aa7H
GrantDb4cAccess.ae4i
LoadDb5qData.aGAi

And I want to delete all files that have the same userId embedded in the body of the files
0
skullnobrainsCommented:
find /PATH/TO/LOGS -name '*.ID.*' -delete
0
Angular Fundamentals

Learn the fundamentals of Angular 2, a JavaScript framework for developing dynamic single page applications.

sdrussAuthor Commented:
No, not looking to delete anything based on the name of the file.  I'm interested with the contents of the file.

So, would something like this work:

           userAcct="gx456_234";
           find -type f -exec grep -q "$userAcct" {} \; -delete
0
sdrussAuthor Commented:
Need help quickly.  Not much progress on this question.  Please help!

Maybe I overly complicated my question in my description.  Unix/Linux experts this does not seem terrifically complicated for someone who works in this environment.  I want to delete bunch of files in directory that match a pattern; this primarily has nothing to do with the filename or file extension.

Maybe something like this, but I need help finalizing this command:

                  find . -type f [[ NEED HELP with search string maybe with grep]] -exec rm -i {} \;

Ex:Want to remove files that contain "Owner=gf_123_456"
0
skullnobrainsCommented:
to delete based on file names, use

find /PATH/TO/LOGS -name "*.Owner=gf_123_456.*" -delete 

Open in new window


if you need to match the files contents rather than their names, try something like the following

grep -r "Owner=gf_123_456" /path/to/logfiles | cut -d: -n1 | uniq | xargs rm

Open in new window


btw your find + grep command should work as well.
the one i provided might perform better depending on the situation

if the logs are long and you expect the string in the beginning of the files, it might be better to run something like "head FILENAME | grep -q -m1 STRING" on each file in the directory

this one will search in the first 20 lines of each file
for file in `find -type f /path/to/logfiles`
do head -n 20 $file | grep -q -m1 STRING && rm $file
done

Open in new window


this ones scans 10 files in parallel but spawns a new shell for each file
find -type f /path/to/logfiles | xargs -n1 -P 10 -I% sh -c 'head -n 20 % | grep -q -m1 STRING && rm %'

Open in new window


a variation of this is probably your best bet
find -type f /path/to/logfiles -printf 'head -n 20 %p | grep -q -m1 STRING && rm %p\n' | sh -s

Open in new window

0
sdrussAuthor Commented:
skullnobrains: This didn't work for me.  I went to work today and tried to embed this in my existing Solaris bash script:

      for file in $(find -type f /path/to/logfiles)
      do
             head -n 20 $file | grep -q -m1 "Owner=gf_123_456"  && rm $file
     done

Get this error:
      find:  illegal option --t
      find:  [ -H | -L ]  path-list  predicate -list
0
skullnobrainsCommented:
my bad

find /path/to/log/files -type f

the first find was ok, all the others have the same idiotic copy-paste mistake

if possible, you should probably change the code that creates those files so the ids appear in their names and a basic glob can identify the files rather than open each of them
0
sdrussAuthor Commented:
skullnobrains:  This is meant to be part of a bigger bash shell script.  Should, grep -q -l work okay?  Want to be quiet and only need the names of files to delete.  BTW, how does the "&&" work.  Thanks

So:

for file in $(find -type f /path/to/logfiles)
      do
             head -n 20 $file | grep -q -l "Owner=gf_123_456"  && rm $file
     done
0
skullnobrainsCommented:
Should, grep -q -l work okay?

if your implementation has -l, that is much better than my cut + uniq mess

but if the logfiles are big, the problem of reading through all of them stands, and you cannot combine head with grep -l because grep will print something like /dev/stdin rather than the actual file name.

but yeah, something trivial like "grep -l -r ID /path/to/logs | xargs rm" or " rm $(grep -l -r ID /path/to/logs)" would work

how does the "&&" work.

&& means "and". the shell will only evaluate as many instructions it needs to determine the "truth" of the current statement.

so when specifying "A && B" , B only needs to be evaluated if A evaluates to true : if A if false, the shell already knows "A && B" is false without needing to evaluate B. overall this is equivalent to "if A ; then B ; fi".

some people like using these constructs, others hate them. i consider them easier to read in many cases when you get used to them and they usually require less typing
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
sdrussAuthor Commented:
Really, really good help.  Absolutely appreciate.  Look forward to working with you again!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Bash

From novice to tech pro — start learning today.