ksh: How to merge several files into one file and customize layout of the merged file?

Hello all,
I have a directory called /apps/logs/sct/{$YEAR}/{$MONTH}. I have the following ksh script which check for year and month directory and create them if they don't exist.

#!/bin/ksh

YEAR=`date '+%Y'`
MONTH=`date '+%y-%B'`
DAY=`date '+%d'`
BASE_DIR=/apps/logs/sct
cd ${BASE_DIR}

   if [  -e  ${YEAR} ]
   then
      if [ ! -e ${MONTH} ]
      then
       mkdir ${YEAR}/${MONTH}
      fi
   else
     mkdir -p ${YEAR}/${MONTH}
   fi

/apps/logs/sct/2005/05-July contains several files with the following reports:


Subject: run_cron monthly on ecs51h2

==== START run_cron monthly on ecs51h2 === Fri Jul  1 05:05:00 EDT 2005
  Summary File /var/sct/logs/monthly.ecs51h2.20050701.sum.01
  For details refer to Log File:
    /var/sct/logs/monthly.ecs51h2.20050701.log.01


audit_filesys Warning : 1 files are owned by root and world-writable

audit_filesys Warning : 1 directories are world-writable and not sticky

audit_filesys Warning : 5581 files are not owned by a recognized owner or group

audit_filesys Warning : 2 suspected extra password and group files

pwck Warning : 6 Password file inconsistencies

verifypw ERROR : 3 Password entries are not coinciding with SILAS.

List_of_SGID_and_SUID_Programs Display : 1 List of SUID Programs

=== END run_cron monthly on ecs51h2 === Fri Jul  1 05:06:33 EDT 2005
===   1 Error(s) and 5 Warning(s) ===
 
Subject: run_cron monthly on nicdev2

==== START run_cron monthly on nicdev2 === Fri Jul  1 12:43:17 EDT 2005
  Summary File /var/sct/logs/monthly.nicdev2.20050701.sum.01
  For details refer to Log File:
    /var/sct/logs/monthly.nicdev2.20050701.log.01


audit_filesys Warning : 1 files are owned by root and world-writable

audit_filesys Warning : 7 directories are world-writable and not sticky

audit_filesys Warning : 97 files are not owned by a recognized owner or group

audit_filesys Warning : 3 suspected extra password and group files

pwck Warning : 17 Password file inconsistencies

su_chk Warning : 3 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display : 2 List of SGID Programs

List_of_SGID_and_SUID_Programs Display : 4 List of SUID Programs

=== END run_cron monthly on nicdev2 === Fri Jul  1 12:45:12 EDT 2005
===   0 Error(s) and 6 Warning(s) ===

Now, I want to merge these two files (ecs51h2,nicdev2) into one file and create one report. I also was wondering whether there is any way I can make changes to the format of each files before it's sent to the merged file. As you see each file has a subject which I want to add some border around each one by using +. In a simple way, I want to format the merge file some how that can be easy to read.
Please be clear with details when you are commenting on my question.
Thanks in advance
Leyla
leylakhavaninAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

TintinCommented:
Firstly, your create directory script can be simplified to:

#!/bin/ksh
BASE_DIR=/apps/logs/sct
DAY=`date +%d`
DATE=`date +`%Y/%y-%B'`
mkdir -p $BASE_DIR/$DATE 2>/dev/null

You can create a merged file, where each file is separated by  dashes (-), by doing:

cd $BASE_DIR/$DATE

rm -f combined.report

for file in *
do
  cat $file >>combined.report
  echo ------------------------------------------------------------------------ >>combined.report
  rm -f $file
done



Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
ahoffmannCommented:
> Now, I want to merge these two files (ecs51h2,nicdev2)
cat ecs51h2 > mergedfile
cat nicdev2 >> mergedfile

> any way I can make changes to the format of each files before it's sent to the merged file.
pipe file through awk, perl, sed, tr or whatever you feel comfortable

> As you see each file has a subject which I want to add some border around each one by using +
awk '($1=="Subject:"){print "+++++++++++++++++++";print;print "+++++++++++++++++++"} file-containing-subject

> In a simple way, I want to format the merge file some how that can be easy to read.
easy to read by what/whom? humans, machines?
what is the definition of "easy"?

BTW your code:
>   if [  -e  ${YEAR} ]
checks for existance, hence if a file exists named $YEAR your following mkdir command fails, you better check liek:
if [  -d  ${YEAR} ]
leylakhavaninAuthor Commented:
Tintin,
I haven't able to run this yet due to some permission error so I have to go and find out why I am getting this error. In the mean time, would you explain to me why you are using "  rm -f combined.report " and " rm -f $file " in the for loop wouldn't those remove the file?
 
CompTIA Cloud+

The CompTIA Cloud+ Basic training course will teach you about cloud concepts and models, data storage, networking, and network infrastructure.

leylakhavaninAuthor Commented:
ahoffmann,
Thanks for your comment, but I do not want to merge only two file. I just gave an example of two of the 70 files that needs to be merged. I also do not know how to work with  awk, perl, sed, tr
>easy to read by what/whom? humans, machines?what is the definition of "easy"?
By humans. Since the combined report contains Heather, Subject,ERROR and Warning for 70 different server one after another I just want to make it easier to whomever wants to read it. In a better way, an easy way to make it distinguishable between each server names and their report.
Hope that made it clear.


ahoffmannCommented:
> .. but I do not want to merge only two file. I just gave an example of two of the 70 files
and what is the problem using my method to merge 70 files?

awk '($1=="Subject:"){print "+++++++++++++++++++";print;print "+++++++++++++++++++"} file-containing-subject >> merged-file
leylakhavaninAuthor Commented:
Ahoffmann,
I never said that ther eis some thing wrong with your method. As I mentioned before I don't know how do awk and I guess that's where the confusion came from. The following is what I am running now:

#!/bin/ksh

YEAR= `date '+%Y'`
MONTH=`date '+%y-%B'`
DAY=`date '+%d'`
BASE_DIR=/apps/logs/sct

#Check to make sure the Year and Month direcotries exist if not
#create the cureent year/month directory
cd ${BASE_DIR}

   if [  -d  ${YEAR} ]
   then
      if [ ! -e ${MONTH} ]
      then
       mkdir ${YEAR}/${MONTH}
      fi
   else
     mkdir -p ${YEAR}/${MONTH}
   fi

#Now move the files into the directory.
cp *.---.---.com /apps/logs/sct/${YEAR}/${MONTH}

#Combined all the file into one file called mergedfile
cat ecs45h1 >mergedfile
cat ecs49n >>mergedfile

awk '($1=="Subject:")'{print "+++++++++++++++++++";print;print "+++++++++++++++++++"} file-containing-subject >> merged-file

These are the error that I am getting?!?!?!?!? Any reason why?

mkdir: Failed to make directory "/05-July"; File exists
cp: /apps/logs/sct//05-July not found
cat: cannot open ecs45h1
cat: cannot open ecs49n
awk: syntax error near line 1
awk: illegal statement near line 1


Tintin,

when I used these line of code, it worked fine

#Change the directory and create a combined report
cd /apps/logs/sct/${YEAR}/${MONTH}

for file in *
do
        cat $file >>combined.report
        echo ------------------------------------------------------------------>>combined.report
done

ahoffmannCommented:
> mkdir: Failed to make directory "/05-July"; File exists
that's 'cause you have wrong syntax in seting your variables YEAR and MONTH, please remove any white space arround =

> cp: /apps/logs/sct//05-July not found
a follow up of the previous one

> cat: cannot open ecs45h1
well, that's something to check for you

> awk: syntax error near line 1
oops, missed the closing quote, sorry.
use:
awk '($1=="Subject:")'{print "+++++++++++++++++++";print;print "+++++++++++++++++++"}' file-containing-subject >> merged-file

and dont forget to fill in *your* valid filenames for file-containing-subject and merged-file

> when I used these line of code, it worked fine
probably you need to have such a loop arround your list of file in your previous code snippet
leylakhavaninAuthor Commented:
Ahoffman,
here is what I run:
rm merged.report
for file in *
do
   awk '($1=="Subject:")'{print "+++++++++++++++++++";print;print "+++++++++++++++++++"}*.pd9.ford.com>>merged.report
done
This is the error that I get: around 40 of them!

awk: syntax error near line 1
awk: illegal statement near line 1

awk: syntax error near line 1
awk: illegal statement near line 1

awk: syntax error near line 1
awk: illegal statement near line 1

Do you know why?
ahoffmannCommented:
awk '($1=="Subject:")'{print "+++++++++++++++++++";print;print "+++++++++++++++++++"}*.pd9.ford.com>>merged.report

should be

 awk '($1=="Subject:")'{print "+++++++++++++++++++";print;print "+++++++++++++++++++"}' $file >>merged.report
leylakhavaninAuthor Commented:
Still syntax error?!?!
for file in *
do
awk '($1=="Subject:")' '{print"+++++++++++++++++++";print;print"+++++++++++++++++++"}'$file>>merged.report
done

ERRORS:
awk: syntax error near line 1
awk: bailing out near line 1

Here is another line with different syntax error?!?!
for file in *
do
awk '($1=="Subject:")'{print"+++++++++++++++++++";print;print"+++++++++++++++++++"}'$file>>merged.report
done

ERRORS:
./test.sh[59]: syntax error at line 61 : `'' unmatched
ahoffmannCommented:
awk '($1=="Subject:"){print "+++++++++++++++++++";print;print "+++++++++++++++++++"}' $file >>merged.report
leylakhavaninAuthor Commented:
Same thing!!!!
awk: syntax error near line 1
awk: bailing out near line 1
ahoffmannCommented:
unbelievable, works for me
leylakhavaninAuthor Commented:
Why do I have to run this twice to be able to create combined.report file? If the file does not exist yet, it will give me this error:
./test.sh
combined.report: No such file or directory
Is there anyway to fix this?
Thanks,
Leyla
******************************************************************************************
#!/bin/ksh

YEAR=`date '+%Y'`
MONTH=`date '+%y-%B'`
DAY=`date '+%d'`
BASE_DIR=/apps/logs/sct

#############################################################################
#Check to make sure that current Year and Month direcotries exist           #
#and if not create one                                                      #
#############################################################################
cd ${BASE_DIR}

   if [  -d  ${YEAR} ]
   then
      if [ -d ${YEAR}/${MONTH} ]
      then
        :
      else
       mkdir ${YEAR}/${MONTH}
      fi
   else
     mkdir -p ${YEAR}/${MONTH}
   fi

###########################################
#Now move the files into the directory    #
###########################################
cp *.pd9.ford.com /apps/logs/sct/${YEAR}/${MONTH}

##########################################################
#Change the directory and create a combined report in    #
# /apps/logs/sct/${YEAR}/${MONTH}                        #
##########################################################

cd /apps/logs/sct/${YEAR}/${MONTH}

rm combined.report

for file in *
do
        pr -f $file >>combined.report
        echo ----------------------------------------------------------------------------->>combined.report
done
ahoffmannCommented:
> combined.report: No such file or directory
rm -f combined.report
leylakhavaninAuthor Commented:
That works fine, thanks.
Now I am getting a little bit problem with the format of the report :( Here is how the report will look like when it's merged:

Jul 11 14:12 2005  2005-Jul-1-batch.---.---.com Page 1


==== START run_cron monthly on batch === Fri Jul  1 05:05:05 EDT 2005
  Summary File /var/sct/logs/monthly.20050701.sum.01
  For details refer to Log File:
    /var/sct/logs/monthly.20050701.log.01


su_chk Warning :       11 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :       10 List of SUID Programs

=== END run_cron monthly on batch === Fri Jul  1 05:12:58 EDT 2005
===   0 Error(s) and 1 Warning(s) ===

^L-----------------------------------------------------------------------------


Jul 11 14:12 2005  2005-Jul-1-batch1.---.---.com Page 1


==== START run_cron monthly on batch1 === Fri Jul  1 05:05:01 EDT 2005
  Summary File /var/sct/logs/monthly.20050701.sum.01
  For details refer to Log File:
    /var/sct/logs/monthly.20050701.log.01


audit_filesys Warning :       13 files are not owned by a recognized owner or group

su_chk Warning :        5 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :        1 List of SGID Programs

List_of_SGID_and_SUID_Programs Display :        9 List of SUID Programs

=== END run_cron monthly on batch1 === Fri Jul  1 05:08:13 EDT 2005
===   0 Error(s) and 2 Warning(s) ===

^L-----------------------------------------------------------------------------


Jul 11 14:12 2005  2005-Jul-1-batch2.---.---.com Page 1


==== START run_cron monthly on batch2 === Fri Jul  1 05:05:02 EDT 2005
  Summary File /var/sct/logs/monthly.20050701.sum.01
  For details refer to Log File:
    /var/sct/logs/monthly.20050701.log.01


List_of_SGID_and_SUID_Programs Display :        1 List of SGID Programs

List_of_SGID_and_SUID_Programs Display :        6 List of SUID Programs

=== END run_cron monthly on batch2 === Fri Jul  1 05:09:29 EDT 2005
===   0 Error(s) and 0 Warning(s) ===

^L-----------------------------------------------------------------------------


Jul 11 14:12 2005  2005-Jul-1-batch3.---.---.com Page 1


==== START run_cron monthly on batch3 === Fri Jul  1 05:05:04 EDT 2005
  Summary File /var/sct/logs/monthly.20050701.sum.01
  For details refer to Log File:
    /var/sct/logs/monthly.20050701.log.01


su_chk Warning :        1 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :        1 List of SGID Programs

List_of_SGID_and_SUID_Programs Display :        4 List of SUID Programs

=== END run_cron monthly on batch3 === Fri Jul  1 05:09:23 EDT 2005
===   0 Error(s) and 1 Warning(s) ===

^L-----------------------------------------------------------------------------


I run the following script to create this report:
#!/bin/ksh

YEAR=`date '+%Y'`
MONTH=`date '+%y-%B'`
DAY=`date '+%d'`
BASE_DIR=/apps/logs/sct

#############################################################################
#Check to make sure that current Year and Month direcotries exist           #
#and if not create one                                                      #
#############################################################################
cd ${BASE_DIR}

   if [  -d  ${YEAR} ]
   then
      if [ -d ${YEAR}/${MONTH} ]
      then
        :
      else
       mkdir ${YEAR}/${MONTH}
      fi
   else
     mkdir -p ${YEAR}/${MONTH}
   fi

###########################################
#Now move the files into the directory    #
###########################################
cp *.pd9.ford.com /apps/logs/sct/${YEAR}/${MONTH}

##########################################################
#Change the directory and create a combined report in    #
# /apps/logs/sct/${YEAR}/${MONTH}                        #
##########################################################

cd /apps/logs/sct/${YEAR}/${MONTH}

rm -f combined.report

for file in *
do
        pr -f $file >>combined.report
        echo ----------------------------------------------------------------------------->>combined.report
done


I want to change the format of the reports as following:



batch.---.---.com



su_chk Warning :       11 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :       10 List of SUID Programs

-----------------------------------------------------------------------------

batch1.---.---.com

 

audit_filesys Warning :       13 files are not owned by a recognized owner or group

su_chk Warning :        5 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :        1 List of SGID Programs

List_of_SGID_and_SUID_Programs Display :        9 List of SUID Programs

-----------------------------------------------------------------------------

batch2.---.---.com



List_of_SGID_and_SUID_Programs Display :        1 List of SGID Programs

List_of_SGID_and_SUID_Programs Display :        6 List of SUID Programs

-----------------------------------------------------------------------------
batch3.---.---.com



su_chk Warning :        1 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :        1 List of SGID Programs

List_of_SGID_and_SUID_Programs Display :        4 List of SUID Programs

-----------------------------------------------------------------------------


batch, batch1, batch2, batch3 are 3 servers that we have in our system. The report will include around 60 servers with the following name format 2005-Jul-1-<server name>.---.---.com which will copy and merge them into combined.report file.
Is there any way that I can create my report as it is shown above with the only the server name as title?

ahoffmannCommented:
# something like:
awk '(NF>3){if($(NF-1)=="Page"){f=$(NF-2);print f;next}}/attempt/{print;next}/rograms/{print;next}/----------/{print;next}END{print f}' $file >>combined.report
# to be improved in many ways
leylakhavaninAuthor Commented:
awk '(NF>3){if($(NF-1)=="Page"){f=$(NF-2);print f;next}}/attempt/{print;next}/rograms/{print;next}/----------/{print;next}END{print f}' $file >>combined.report
Unfortunately, it doesn't give the format that I am looking for. The following line is what I used your suggestions and came up with:

for file in `ls -1t *.pd9.com`
do
        cat $file >> combined.report
        echo ${file} | awk -F"-" ' { print $4} '| awk -F"." ' { print $1} ' >> combined.report
        echo ----------------------------------------------------------------------------->>combined.report
done

But not exactly right yet. It's getting very close though. Here is what I get when I run it:

==== START run_cron monthly on batch === Fri Jul  1 05:05:05 EDT 2005
  Summary File /var/sct/logs/monthly.20050701.sum.01
  For details refer to Log File:
    /var/sct/logs/monthly.20050701.log.01


su_chk Warning :       11 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :       10 List of SUID Programs

=== END run_cron monthly on batch === Fri Jul  1 05:12:58 EDT 2005
===   0 Error(s) and 1 Warning(s) ===

batch
-----------------------------------------------------------------------------
==== START run_cron monthly on batch1 === Fri Jul  1 05:05:01 EDT 2005
  Summary File /var/sct/logs/monthly.20050701.sum.01
  For details refer to Log File:
    /var/sct/logs/monthly.20050701.log.01


audit_filesys Warning :       13 files are not owned by a recognized owner or group

su_chk Warning :        5 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :        1 List of SGID Programs

List_of_SGID_and_SUID_Programs Display :        9 List of SUID Programs

=== END run_cron monthly on batch1 === Fri Jul  1 05:08:13 EDT 2005
===   0 Error(s) and 2 Warning(s) ===

batch1
-----------------------------------------------------------------------------

Any help?
ahoffmannCommented:
awk '/^===/{next;}(NF>3){if($(NF-1)=="Page"){f=$(NF-2);print f;next}}/attempt/{print;next}/rograms/{print;next}/----------/{print;next}END{print f}' $file >>combined.report
leylakhavaninAuthor Commented:
Not doing any thing!
leylakhavaninAuthor Commented:
This for loop is what I could have worked on to be able to create the format that I am looking for. But my egrep doesn't work properly :( Anyone knows why?

for file in `ls -1t *.---.com`
do
        server=`echo ${file} | awk -F"-" ' { print $4} '| awk -F"." ' { print $1} ' `
        echo " " >> combined.report
        echo " Server *${server}* report:" >> combined.report
        egrep -v "==|From:|Sent:|To:|Subject:|File:|refer" ${file}  >> combined.report
        echo ----------------------------------------------------------------------------->>combined.report
done

This is what it returns:

-------------------------------------------------------------------------------
Server *batch* report:
  Summary File /var/sct/logs/monthly.20050701.sum.01
    /var/sct/logs/monthly.20050701.log.01


su_chk Warning :       11 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :       10 List of SUID Programs


-----------------------------------------------------------------------------
 
 Server *batch1* report:
  Summary File /var/sct/logs/monthly.20050701.sum.01
    /var/sct/logs/monthly.20050701.log.01


audit_filesys Warning :       13 files are not owned by a recognized owner or group

su_chk Warning :        5 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :        1 List of SGID Programs

List_of_SGID_and_SUID_Programs Display :        9 List of SUID Programs

And for some reason my egrep does not work on the following lines:
Summary File /var/sct/logs/monthly.20050701.sum.01
    /var/sct/logs/monthly.20050701.log.01
ahoffmannCommented:
egrep works on these lines, means they're returned
leylakhavaninAuthor Commented:
Is there any thing else that I can use instead of my egrep to remove the following lines in each report before I send them in combined.report file?
==== START run_cron monthly on batch1 === Fri Jul  1 05:05:01 EDT 2005
  Summary File /var/sct/logs/monthly.20050701.sum.01
  For details refer to Log File:
    /var/sct/logs/monthly.20050701.log.01





=== END run_cron monthly on batch1 === Fri Jul  1 05:08:13 EDT 2005
===   0 Error(s) and 2 Warning(s) ===
ahoffmannCommented:
awk '/^===/{next}($0~/monthly/){next}/refer to Log File/{next}'

but I'd use egrep ..
leylakhavaninAuthor Commented:
>awk '/^===/{next}($0~/monthly/){next}/refer to Log File/{next}'
Hangs on when I add this into my codes!!

leylakhavaninAuthor Commented:
Any one knows what is wrong with my egrep?

egrep -v "==|Summary:|For" ${file}  >> combined.report
leylakhavaninAuthor Commented:
Cheers, my egrep finally started to work and thanks Tintin for your help on showing me how to merge my files.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
System Programming

From novice to tech pro — start learning today.