ksh: How to merge several files into one file and customize layout of the merged file?

> Now, I want to merge these two files (ecs51h2,nicdev2)
cat ecs51h2 > mergedfile
cat nicdev2 >> mergedfile

> any way I can make changes to the format of each files before it's sent to the merged file.
pipe file through awk, perl, sed, tr or whatever you feel comfortable

> As you see each file has a subject which I want to add some border around each one by using +
awk '($1=="Subject:"){print "+++++++++++++++++++";print;print "+++++++++++++++++++"} file-containing-subject

> In a simple way, I want to format the merge file some how that can be easy to read.
easy to read by what/whom? humans, machines?
what is the definition of "easy"?

BTW your code:
>   if [  -e  ${YEAR} ]
checks for existance, hence if a file exists named $YEAR your following mkdir command fails, you better check liek:
if [  -d  ${YEAR} ]

Tintin,
I haven't able to run this yet due to some permission error so I have to go and find out why I am getting this error. In the mean time, would you explain to me why you are using "  rm -f combined.report " and " rm -f $file " in the for loop wouldn't those remove the file?
 

ahoffmann,
Thanks for your comment, but I do not want to merge only two file. I just gave an example of two of the 70 files that needs to be merged. I also do not know how to work with  awk, perl, sed, tr
>easy to read by what/whom? humans, machines?what is the definition of "easy"?
By humans. Since the combined report contains Heather, Subject,ERROR and Warning for 70 different server one after another I just want to make it easier to whomever wants to read it. In a better way, an easy way to make it distinguishable between each server names and their report.
Hope that made it clear.

> .. but I do not want to merge only two file. I just gave an example of two of the 70 files
and what is the problem using my method to merge 70 files?

awk '($1=="Subject:"){print "+++++++++++++++++++";print;print "+++++++++++++++++++"} file-containing-subject >> merged-file

Ahoffmann,
I never said that ther eis some thing wrong with your method. As I mentioned before I don't know how do awk and I guess that's where the confusion came from. The following is what I am running now:

#!/bin/ksh

YEAR= `date '+%Y'`
MONTH=`date '+%y-%B'`
DAY=`date '+%d'`
BASE_DIR=/apps/logs/sct

#Check to make sure the Year and Month direcotries exist if not
#create the cureent year/month directory
cd ${BASE_DIR}

   if [  -d  ${YEAR} ]
   then
      if [ ! -e ${MONTH} ]
      then
       mkdir ${YEAR}/${MONTH}
      fi
   else
     mkdir -p ${YEAR}/${MONTH}
   fi

#Now move the files into the directory.
cp *.---.---.com /apps/logs/sct/${YEAR}/${MONTH}

#Combined all the file into one file called mergedfile
cat ecs45h1 >mergedfile
cat ecs49n >>mergedfile

awk '($1=="Subject:")'{print "+++++++++++++++++++";print;print "+++++++++++++++++++"} file-containing-subject >> merged-file

These are the error that I am getting?!?!?!?!? Any reason why?

mkdir: Failed to make directory "/05-July"; File exists
cp: /apps/logs/sct//05-July not found
cat: cannot open ecs45h1
cat: cannot open ecs49n
awk: syntax error near line 1
awk: illegal statement near line 1


Tintin,

when I used these line of code, it worked fine

#Change the directory and create a combined report
cd /apps/logs/sct/${YEAR}/${MONTH}

for file in *
do
        cat $file >>combined.report
        echo ------------------------------------------------------------------>>combined.report
done

> mkdir: Failed to make directory "/05-July"; File exists
that's 'cause you have wrong syntax in seting your variables YEAR and MONTH, please remove any white space arround =

> cp: /apps/logs/sct//05-July not found
a follow up of the previous one

> cat: cannot open ecs45h1
well, that's something to check for you

> awk: syntax error near line 1
oops, missed the closing quote, sorry.
use:
awk '($1=="Subject:")'{print "+++++++++++++++++++";print;print "+++++++++++++++++++"}' file-containing-subject >> merged-file

and dont forget to fill in *your* valid filenames for file-containing-subject and merged-file

> when I used these line of code, it worked fine
probably you need to have such a loop arround your list of file in your previous code snippet

Ahoffman,
here is what I run:
rm merged.report
for file in *
do
   awk '($1=="Subject:")'{print "+++++++++++++++++++";print;print "+++++++++++++++++++"}*.pd9.ford.com>>merged.report
done
This is the error that I get: around 40 of them!

awk: syntax error near line 1
awk: illegal statement near line 1

awk: syntax error near line 1
awk: illegal statement near line 1

awk: syntax error near line 1
awk: illegal statement near line 1

Do you know why?

awk '($1=="Subject:")'{print "+++++++++++++++++++";print;print "+++++++++++++++++++"}*.pd9.ford.com>>merged.report

should be

 awk '($1=="Subject:")'{print "+++++++++++++++++++";print;print "+++++++++++++++++++"}' $file >>merged.report

Still syntax error?!?!
for file in *
do
awk '($1=="Subject:")' '{print"+++++++++++++++++++";print;print"+++++++++++++++++++"}'$file>>merged.report
done

ERRORS:
awk: syntax error near line 1
awk: bailing out near line 1

Here is another line with different syntax error?!?!
for file in *
do
awk '($1=="Subject:")'{print"+++++++++++++++++++";print;print"+++++++++++++++++++"}'$file>>merged.report
done

ERRORS:
./test.sh[59]: syntax error at line 61 : `'' unmatched

awk '($1=="Subject:"){print "+++++++++++++++++++";print;print "+++++++++++++++++++"}' $file >>merged.report

Same thing!!!!
awk: syntax error near line 1
awk: bailing out near line 1

unbelievable, works for me

Why do I have to run this twice to be able to create combined.report file? If the file does not exist yet, it will give me this error:
./test.sh
combined.report: No such file or directory
Is there anyway to fix this?
Thanks,
Leyla
******************************************************************************************
#!/bin/ksh

YEAR=`date '+%Y'`
MONTH=`date '+%y-%B'`
DAY=`date '+%d'`
BASE_DIR=/apps/logs/sct

#############################################################################
#Check to make sure that current Year and Month direcotries exist           #
#and if not create one                                                      #
#############################################################################
cd ${BASE_DIR}

   if [  -d  ${YEAR} ]
   then
      if [ -d ${YEAR}/${MONTH} ]
      then
        :
      else
       mkdir ${YEAR}/${MONTH}
      fi
   else
     mkdir -p ${YEAR}/${MONTH}
   fi

###########################################
#Now move the files into the directory    #
###########################################
cp *.pd9.ford.com /apps/logs/sct/${YEAR}/${MONTH}

##########################################################
#Change the directory and create a combined report in    #
# /apps/logs/sct/${YEAR}/${MONTH}                        #
##########################################################

cd /apps/logs/sct/${YEAR}/${MONTH}

rm combined.report

for file in *
do
        pr -f $file >>combined.report
        echo ----------------------------------------------------------------------------->>combined.report
done

That works fine, thanks.
Now I am getting a little bit problem with the format of the report :( Here is how the report will look like when it's merged:

Jul 11 14:12 2005  2005-Jul-1-batch.---.---.com Page 1


==== START run_cron monthly on batch === Fri Jul  1 05:05:05 EDT 2005
  Summary File /var/sct/logs/monthly.20050701.sum.01
  For details refer to Log File:
    /var/sct/logs/monthly.20050701.log.01


su_chk Warning :       11 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :       10 List of SUID Programs

=== END run_cron monthly on batch === Fri Jul  1 05:12:58 EDT 2005
===   0 Error(s) and 1 Warning(s) ===

^L-----------------------------------------------------------------------------


Jul 11 14:12 2005  2005-Jul-1-batch1.---.---.com Page 1


==== START run_cron monthly on batch1 === Fri Jul  1 05:05:01 EDT 2005
  Summary File /var/sct/logs/monthly.20050701.sum.01
  For details refer to Log File:
    /var/sct/logs/monthly.20050701.log.01


audit_filesys Warning :       13 files are not owned by a recognized owner or group

su_chk Warning :        5 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :        1 List of SGID Programs

List_of_SGID_and_SUID_Programs Display :        9 List of SUID Programs

=== END run_cron monthly on batch1 === Fri Jul  1 05:08:13 EDT 2005
===   0 Error(s) and 2 Warning(s) ===

^L-----------------------------------------------------------------------------


Jul 11 14:12 2005  2005-Jul-1-batch2.---.---.com Page 1


==== START run_cron monthly on batch2 === Fri Jul  1 05:05:02 EDT 2005
  Summary File /var/sct/logs/monthly.20050701.sum.01
  For details refer to Log File:
    /var/sct/logs/monthly.20050701.log.01


List_of_SGID_and_SUID_Programs Display :        1 List of SGID Programs

List_of_SGID_and_SUID_Programs Display :        6 List of SUID Programs

=== END run_cron monthly on batch2 === Fri Jul  1 05:09:29 EDT 2005
===   0 Error(s) and 0 Warning(s) ===

^L-----------------------------------------------------------------------------


Jul 11 14:12 2005  2005-Jul-1-batch3.---.---.com Page 1


==== START run_cron monthly on batch3 === Fri Jul  1 05:05:04 EDT 2005
  Summary File /var/sct/logs/monthly.20050701.sum.01
  For details refer to Log File:
    /var/sct/logs/monthly.20050701.log.01


su_chk Warning :        1 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :        1 List of SGID Programs

List_of_SGID_and_SUID_Programs Display :        4 List of SUID Programs

=== END run_cron monthly on batch3 === Fri Jul  1 05:09:23 EDT 2005
===   0 Error(s) and 1 Warning(s) ===

^L-----------------------------------------------------------------------------


I run the following script to create this report:
#!/bin/ksh

YEAR=`date '+%Y'`
MONTH=`date '+%y-%B'`
DAY=`date '+%d'`
BASE_DIR=/apps/logs/sct

#############################################################################
#Check to make sure that current Year and Month direcotries exist           #
#and if not create one                                                      #
#############################################################################
cd ${BASE_DIR}

   if [  -d  ${YEAR} ]
   then
      if [ -d ${YEAR}/${MONTH} ]
      then
        :
      else
       mkdir ${YEAR}/${MONTH}
      fi
   else
     mkdir -p ${YEAR}/${MONTH}
   fi

###########################################
#Now move the files into the directory    #
###########################################
cp *.pd9.ford.com /apps/logs/sct/${YEAR}/${MONTH}

##########################################################
#Change the directory and create a combined report in    #
# /apps/logs/sct/${YEAR}/${MONTH}                        #
##########################################################

cd /apps/logs/sct/${YEAR}/${MONTH}

rm -f combined.report

for file in *
do
        pr -f $file >>combined.report
        echo ----------------------------------------------------------------------------->>combined.report
done


I want to change the format of the reports as following:



batch.---.---.com



su_chk Warning :       11 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :       10 List of SUID Programs

-----------------------------------------------------------------------------

batch1.---.---.com

 

audit_filesys Warning :       13 files are not owned by a recognized owner or group

su_chk Warning :        5 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :        1 List of SGID Programs

List_of_SGID_and_SUID_Programs Display :        9 List of SUID Programs

-----------------------------------------------------------------------------

batch2.---.---.com



List_of_SGID_and_SUID_Programs Display :        1 List of SGID Programs

List_of_SGID_and_SUID_Programs Display :        6 List of SUID Programs

-----------------------------------------------------------------------------
batch3.---.---.com



su_chk Warning :        1 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :        1 List of SGID Programs

List_of_SGID_and_SUID_Programs Display :        4 List of SUID Programs

-----------------------------------------------------------------------------


batch, batch1, batch2, batch3 are 3 servers that we have in our system. The report will include around 60 servers with the following name format 2005-Jul-1-<server name>.---.---.com which will copy and merge them into combined.report file.
Is there any way that I can create my report as it is shown above with the only the server name as title?

# something like:
awk '(NF>3){if($(NF-1)=="Page"){f=$(NF-2);print f;next}}/attempt/{print;next}/rograms/{print;next}/----------/{print;next}END{print f}' $file >>combined.report
# to be improved in many ways

awk '(NF>3){if($(NF-1)=="Page"){f=$(NF-2);print f;next}}/attempt/{print;next}/rograms/{print;next}/----------/{print;next}END{print f}' $file >>combined.report
Unfortunately, it doesn't give the format that I am looking for. The following line is what I used your suggestions and came up with:

for file in `ls -1t *.pd9.com`
do
        cat $file >> combined.report
        echo ${file} | awk -F"-" ' { print $4} '| awk -F"." ' { print $1} ' >> combined.report
        echo ----------------------------------------------------------------------------->>combined.report
done

But not exactly right yet. It's getting very close though. Here is what I get when I run it:

==== START run_cron monthly on batch === Fri Jul  1 05:05:05 EDT 2005
  Summary File /var/sct/logs/monthly.20050701.sum.01
  For details refer to Log File:
    /var/sct/logs/monthly.20050701.log.01


su_chk Warning :       11 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :       10 List of SUID Programs

=== END run_cron monthly on batch === Fri Jul  1 05:12:58 EDT 2005
===   0 Error(s) and 1 Warning(s) ===

batch
-----------------------------------------------------------------------------
==== START run_cron monthly on batch1 === Fri Jul  1 05:05:01 EDT 2005
  Summary File /var/sct/logs/monthly.20050701.sum.01
  For details refer to Log File:
    /var/sct/logs/monthly.20050701.log.01


audit_filesys Warning :       13 files are not owned by a recognized owner or group

su_chk Warning :        5 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :        1 List of SGID Programs

List_of_SGID_and_SUID_Programs Display :        9 List of SUID Programs

=== END run_cron monthly on batch1 === Fri Jul  1 05:08:13 EDT 2005
===   0 Error(s) and 2 Warning(s) ===

batch1
-----------------------------------------------------------------------------

Any help?

awk '/^===/{next;}(NF>3){if($(NF-1)=="Page"){f=$(NF-2);print f;next}}/attempt/{print;next}/rograms/{print;next}/----------/{print;next}END{print f}' $file >>combined.report

Not doing any thing!

This for loop is what I could have worked on to be able to create the format that I am looking for. But my egrep doesn't work properly :( Anyone knows why?

for file in `ls -1t *.---.com`
do
        server=`echo ${file} | awk -F"-" ' { print $4} '| awk -F"." ' { print $1} ' `
        echo " " >> combined.report
        echo " Server *${server}* report:" >> combined.report
        egrep -v "==|From:|Sent:|To:|Subject:|File:|refer" ${file}  >> combined.report
        echo ----------------------------------------------------------------------------->>combined.report
done

This is what it returns:

-------------------------------------------------------------------------------
Server *batch* report:
  Summary File /var/sct/logs/monthly.20050701.sum.01
    /var/sct/logs/monthly.20050701.log.01


su_chk Warning :       11 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :       10 List of SUID Programs


-----------------------------------------------------------------------------
 
 Server *batch1* report:
  Summary File /var/sct/logs/monthly.20050701.sum.01
    /var/sct/logs/monthly.20050701.log.01


audit_filesys Warning :       13 files are not owned by a recognized owner or group

su_chk Warning :        5 failed su login attempt(s) to root found.

List_of_SGID_and_SUID_Programs Display :        1 List of SGID Programs

List_of_SGID_and_SUID_Programs Display :        9 List of SUID Programs

And for some reason my egrep does not work on the following lines:
Summary File /var/sct/logs/monthly.20050701.sum.01
    /var/sct/logs/monthly.20050701.log.01

egrep works on these lines, means they're returned

Is there any thing else that I can use instead of my egrep to remove the following lines in each report before I send them in combined.report file?
==== START run_cron monthly on batch1 === Fri Jul  1 05:05:01 EDT 2005
  Summary File /var/sct/logs/monthly.20050701.sum.01
  For details refer to Log File:
    /var/sct/logs/monthly.20050701.log.01





=== END run_cron monthly on batch1 === Fri Jul  1 05:08:13 EDT 2005
===   0 Error(s) and 2 Warning(s) ===

>awk '/^===/{next}($0~/monthly/){next}/refer to Log File/{next}'
Hangs on when I add this into my codes!!

Any one knows what is wrong with my egrep?

egrep -v "==|Summary:|For" ${file}  >> combined.report

Cheers, my egrep finally started to work and thanks Tintin for your help on showing me how to merge my files.