Add diff functionality to overall script

Currently I have a fully automated RHEL auditing script that works fine (see attached).

However, additional scope aspects have been added to the original project and need to be incorporated into the script.

Once the script has completed, it needs to take a 'diff' comparison (if available) of a previous audit run of that server and the one that just completed, then create another file (i.e. RHEL-hostname.datestamp.audit.diff) in the same directory. Once diff comparison is completed, zip up 'diff' result file and embed/encrypt the zip file (i.e. zip -P passwd filename.zip), then e-mail the zip file to a selected list of users.

After a number of trail and error attempts outside of the large script, I determined it would be easier to do each server diff individually (since the data is readily available) and give a separate report on each hostname rather than try to group them all together and compile a huge diff listing with all the servers embedded in the report document.

rhel-audit.txt
LVL 29
Michael WorshamStaff Infrastructure ArchitectAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Maciej SsysadminCommented:
Without the whole script - just two functions modified. (Remember to replace destination@email in send_report() with some valid email address).

You have a lot of $(date +%m-%d-%y) and $(hostname) executions. It would be better to run them once, save the output in some variable, and then use these variables.
diff_comparison(){
        write_header "Audit Diff Comparison"
        # finding file from previous run
        OUTPUT_DIR=$(dirname ${OUTPUT})

        TIMESTAMP_LAST=$(find ${OUTPUT_DIR} -type f -name "RHEL-$(hostname).audit*" | sed 's/.*audit\.\([0-9][0-9]\)-\([0-9][0-9]\)-\([0-9][0-9]\)\.txt/20\3\1\2/' | sort | tail -n 1)
        DATE_PREV=$(date -d "${TIMESTAMP_LAST} -1 week" +%m-%d-%y)
        OUTPUT_PREV=${OUTPUT_DIR}/RHEL-$(hostname).audit.${DATE_PREV}.txt
        diff -uN ${OUTPUT_PREV} ${OUTPUT} > ${DIFF_OUTPUT}
}

send_report(){
        write_header "E-Mailing Recipients DIFF output"
        echo "zip -P ${PASSWD} -er ${DIFF_REPORT} ${DIFF_OUTPUT}"
        mutt -x -a ${DIFF_REPORT} -s "Diff report from $(hostname) $(date +%m-%d-%y)" destination@email < /dev/null
}

Open in new window

Michael WorshamStaff Infrastructure ArchitectAuthor Commented:
Something buggy...

date: invalid date "/var/tmp/RHEL-reaper.audit.04-20-10.diff -1 week"

And echo of TIMESTAMP_LAST is showing: /var/tmp/RHEL-reaper.audit.04-20-10.diff

Maciej SsysadminCommented:
My mistake. Corrected version:
diff_comparison(){
        write_header "Audit Diff Comparison"
        # finding file from previous run
        OUTPUT_DIR=$(dirname ${OUTPUT})

        TIMESTAMP_LAST=$(find ${OUTPUT_DIR} -type f -name "RHEL-$(hostname).audit.*.txt" | sed 's/.*audit\.\([0-9][0-9]\)-\([0-9][0-9]\)-\([0-9][0-9]\)\.txt/20\3\1\2/' | sort | tail -n 1)
        DATE_PREV=$(date -d "${TIMESTAMP_LAST} -1 week" +%m-%d-%y)
        OUTPUT_PREV=${OUTPUT_DIR}/RHEL-$(hostname).audit.${DATE_PREV}.txt
        diff -uN ${OUTPUT_PREV} ${OUTPUT} > ${DIFF_OUTPUT}
}

Open in new window

IT Pros Agree: AI and Machine Learning Key

We’d all like to think our company’s data is well protected, but when you ask IT professionals they admit the data probably is not as safe as it could be.

Michael WorshamStaff Infrastructure ArchitectAuthor Commented:
I just discovered something rather important. The diff/mailing script would have to be run on the DCS server (i.e. where the files eventually end up) so the original rhel_audit script running on the client servers wouldn't be able to create the diff files (as they would not be available on the client server anymore) and handle the e-mail portion as the client servers don't have any mailing software installed on them (nor are they allowed to).

So rather than wasting the code written, I would like to break out the diff/mailing portion to be self-run on the DCS server.

Scope is similar, except:
      - Parse the /var/audit/linux-audit directory on the DCS server, based on all the 'RHEL-[multiple FQDN hostnames].audit.datestamp.txt' files found
        - Do the file comparison of all the files discovered, create separate .diff files using the present and 1 week ago audit files found, based on hostname
      - List all the .diff files and zip/encrypt them up as an individual zip file (multiple diff files)
      - E-mail the zip to the recipients

NOTE: If this seems a bit much from the original question/scope, I can open another question and will just reference this one.

Maciej SsysadminCommented:
Well.. isn't it (almost) exactly as in your previous question (http:/Q_25976227.html)? That script is able to run diff on last and previous files (for each hostname). There is no creating encrypted zip and sending email, but that's not a big problem.
Let's try to make all necessary modifications and additions (zip, mail).
#!/bin/sh

AUDIT_DIR=/var/audit/linux-audit
DEST_EMAIL=destination@email

for FILENAME in `find ${AUDIT_DIR} -type f -name "RHEL*.txt" | sed 's/\(RHEL.*\.audit\.\).*/\1/' | sort -u`; do
        SERVER=`echo ${FILENAME} | sed 's/.*RHEL-\(.*\)\.\([0-9]\{2\}-\)\{2\}[0-9]\{2\}\.audit\..*/\1/'`
        # this is just for finding the last file
        DATE2=0
        for FILE in $(find ${AUDIT_DIR} -type f -name "$(basename ${FILENAME})*.txt"); do
                DATE=`echo ${FILE} | sed 's/.*audit\.\([0-9][0-9]\)-\([0-9][0-9]\)-\([0-9][0-9]\)\.txt/20\3\1\2/'`
                test "${DATE}" -gt "${DATE2}" && DATE2=${DATE}
        done

        DATE_PREV=`date -d "${DATE2} -1 week" +%m-%d-%y`

        FILE_LAST=${FILENAME}$(date -d "${DATE2}" +%m-%d-%y).txt
        FILE_PREV=${FILENAME}${DATE_PREV}.txt

        DIFF_OUTPUT=$(echo ${FILE_LAST} | sed 's/\.txt$//').diff
        DIFF_REPORT=${DIFF_OUTPUT}.report.zip

        diff -uN ${FILE_LAST} ${FILE_PREV} > ${DIFF_OUTPUT}

        zip -P ${PASSWD} -er ${DIFF_REPORT} ${DIFF_OUTPUT}
        mutt -x -a ${DIFF_REPORT} -s "Diff report from ${SERVER} $(date +%m-%d-%y)" ${DEST_EMAIL} < /dev/null
done

Open in new window

Maciej SsysadminCommented:
Ah.. broken link. I meant http:/Q_25976227.html
Michael WorshamStaff Infrastructure ArchitectAuthor Commented:
Almost there.

Rather than have individual zip files to e-mail, it it possible to accumulate all of the diff files from the run and add them to one zip file, then send just that one zip out instead?

I have attached the script below with your code (I got a real pain of a document librarian that wants 'pretty' scripts...)

#!/bin/bash
#
# Name: RedHat Enterprise Linux Audit Diff Script (rhel_audit_diff.sh)
#
# Once the rhel_audit script has completed, the DCS server will trigger this
# script via cron, taking a 'diff' comparison of a previous audit run files that
# has been uploaded by the RHEL client servers. The rhel_audit_diff script will
# then produce a new diff file, using the discovered files from the same directory
#
# Then once original audit and previous audit comparison is completed, this script
# will then zip up 'diff' result as a bundle with an encrypted password, then
# e-mail the zip file to a selected list of users.
# -------------------------------------------------------------------------------

## Hard variables ##
RUN_DATE_TIME=$(date +%s);
RUN_DATE=$(date --date="@${RUN_DATE_TIME}" +%D);

## Files ##
#AUDIT_DIR=/var/audit/linux-audit
AUDIT_DIR=/var/tmp
#DIFF_OUTPUT="$AUDIT_DIR/RHEL-$(hostname).audit.$(date +'%m-%d-%y').diff"
#DIFF_REPORT="$AUDIT_DIR/RHEL-$(hostname).audit.$(date +'%m-%d-%y').diff.report.zip"

## User Configurable Options ##
PASSWD="password"
RECIPIENTS="myname@you.wish.com,you@youwish.com"
SUBJECT="RHEL Audit Diff Report -- ${RUN_DATE}"

## Let's begin... ##

chk_root(){
        local meid=$(id -u)
        if [ $meid -ne 0 ];
        then
                echo "You must be root user to run this tool"
                exit 999
        fi
}

write_diff_header(){
        echo " " >> $DIFF_OUTPUT
        echo "---------------------------------------------------" >> $DIFF_OUTPUT
        echo "$@" >> $DIFF_OUTPUT
        echo "---------------------------------------------------"  >> $DIFF_OUTPUT
}

dump_diff_hostdate(){
        write_diff_header "Hostname and Timestamp" >$DIFF_OUTPUT
        echo "* Hostname: $(hostname)" >>$DIFF_OUTPUT
        echo "* Run date and time: $(date)" >>$DIFF_OUTPUT
}

diff_comparison_report(){
        write_diff_header "Audit Diff Comparison & Report"

        for FILENAME in `find ${AUDIT_DIR} -type f -name "RHEL*.txt" | sed 's/\(RHEL.*\.audit\.\).*/\1/' | sort -u`; do 
                SERVER=`echo ${FILENAME} | sed 's/.*RHEL-\(.*\)\.\([0-9]\{2\}-\)\{2\}[0-9]\{2\}\.audit\..*/\1/'` 
                # this is just for finding the last file 
                DATE2=0 
                for FILE in $(find ${AUDIT_DIR} -type f -name "$(basename ${FILENAME})*.txt"); do 
                        DATE=`echo ${FILE} | sed 's/.*audit\.\([0-9][0-9]\)-\([0-9][0-9]\)-\([0-9][0-9]\)\.txt/20\3\1\2/'` 
                        test "${DATE}" -gt "${DATE2}" && DATE2=${DATE} 
                done 
 
                DATE_PREV=`date -d "${DATE2} -1 week" +%m-%d-%y` 
 
                FILE_LAST=${FILENAME}$(date -d "${DATE2}" +%m-%d-%y).txt 
                FILE_PREV=${FILENAME}${DATE_PREV}.txt 
 
                DIFF_OUTPUT=$(echo ${FILE_LAST} | sed 's/\.txt$//').diff 
                DIFF_REPORT=${DIFF_OUTPUT}.report.zip 
 
                diff -uN ${FILE_LAST} ${FILE_PREV} > ${DIFF_OUTPUT} 
 
                zip -P ${PASSWD} -er ${DIFF_REPORT} ${DIFF_OUTPUT} 
#               mutt -x -a ${DIFF_REPORT} -s "Diff report from ${SERVER} $(date +%m-%d-%y)" ${RECIPIENTS} < /dev/null 
done

}

## Trigger functions ##
chk_root
dump_diff_hostdate
diff_comparison_report

Open in new window

Maciej SsysadminCommented:
Of course it is possible :)
Remove (comment) zip and mutt lines from diff_comparison_report() function.
As you like/have to use functions, create new one, and run it as the last one.
email_diffs() {
        DATE=$(date +%m-%d-%y)
        DIFF_REPORT=${AUDIT_DIR}/RHEL-audit-${DATE}.report.zip
        zip -P ${PASSWD} -er ${DIFF_REPORT} ${AUDIT_DIR}/*${DATE}.diff
        mutt -x -a ${DIFF_REPORT} -s "${SUBJECT}" ${RECIPIENTS} < /dev/null
}

Open in new window

Michael WorshamStaff Infrastructure ArchitectAuthor Commented:
Quick question/fix...

I want to use the same script for our Windows auditing logs as well, but I just discovered their incoming file format looks like this:

WIN-SITE-SERVERNAME.audit.MM-DD-YYYY.txt

What do I need to add/alter under the file/sed portion so I can do the same diff formatting/zip as well.
Maciej SsysadminCommented:

diff_comparison_report(){
        write_diff_header "Audit Diff Comparison & Report"

        for FILENAME in `find ${AUDIT_DIR} -type f -name "WIN*.txt" | sed 's/\(WIN.*\.audit\.\).*/\1/' | sort -u`; do 
                SERVER=`echo ${FILENAME} | sed 's/.*WIN-\(.*\)\.\([0-9]\{2\}-\)\{2\}[0-9]\{4\}\.audit\..*/\1/'` 
                # this is just for finding the last file 
                DATE2=0 
                for FILE in $(find ${AUDIT_DIR} -type f -name "$(basename ${FILENAME})*.txt"); do 
                        DATE=`echo ${FILE} | sed 's/.*audit\.\([0-9][0-9]\)-\([0-9][0-9]\)-\([0-9]\{4\}\)\.txt/\3\1\2/'` 
                        test "${DATE}" -gt "${DATE2}" && DATE2=${DATE} 
                done 
 
                DATE_PREV=`date -d "${DATE2} -1 week" +%m-%d-%Y` 
 
                FILE_LAST=${FILENAME}$(date -d "${DATE2}" +%m-%d-%Y).txt 
                FILE_PREV=${FILENAME}${DATE_PREV}.txt 
 
                DIFF_OUTPUT=$(echo ${FILE_LAST} | sed 's/\.txt$//').diff 
                DIFF_REPORT=${DIFF_OUTPUT}.report.zip 
 
                diff -uN ${FILE_LAST} ${FILE_PREV} > ${DIFF_OUTPUT} 
done

email_diffs() {
        DATE_LIN=$(date +%m-%d-%y)
        DATE_WIN=$(date +%m-%d-%Y)
        DIFF_REPORT_LIN=${AUDIT_DIR}/RHEL-audit-${DATE_LIN}.report.zip
        DIFF_REPORT_WIN=${AUDIT_DIR}/WIN-audit-${DATE_WIN}.report.zip
        zip -P ${PASSWD} -er ${DIFF_REPORT_LIN} ${AUDIT_DIR}/RHEL*${DATE_LIN}.diff
        zip -P ${PASSWD} -er ${DIFF_REPORT_WIN} ${AUDIT_DIR}/WIN*${DATE_WIN}.diff
        mutt -x -a ${DIFF_REPORT_LIN} -a ${DIFF_REPORT_WIN} -s "${SUBJECT}" ${RECIPIENTS} < /dev/null
}

Open in new window

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Maciej SsysadminCommented:
I forgot to put ending } in diff_comparision_report() (and indent 'done' line in this function).
Michael WorshamStaff Infrastructure ArchitectAuthor Commented:
You, my friend, are a god! Thank you very much for the help.
Michael WorshamStaff Infrastructure ArchitectAuthor Commented:
For anyone wants to use it, here is the final script in it's completion.


#!/bin/bash
#
# Name: RHEL & Windows Audit Diff Reporting Script (audit_diff_report)
#
# Once the rhel/win audit script has completed, the DCS server will trigger this
# script via cron, taking a 'diff' comparison of a previous audit run files that
# has been uploaded by the RHEL client servers. The audit_diff_report script will
# then produce a new diff file, using the discovered files from the same directory
#
# Then once original audit and previous audit comparison is completed, this script
# will then zip up 'diff' result as a bundle with an encrypted password, then
# e-mail the zip file to a selected list of users.
# -------------------------------------------------------------------------------

## Hard variables ##
RUN_DATE_TIME=$(date +%s);
RUN_DATE=$(date --date="@${RUN_DATE_TIME}" +%D);

## Files ##
LIN_AUDIT_DIR=/var/audit/linux-audit
WIN_AUDIT_DIR=/var/audit/win-audit

## User Configurable Options ##
PASSWD="password"
RECIPIENTS="your-email-address"
SUBJECT="RHEL & Windows Audit Diff Reports -- ${RUN_DATE}"

## Let's begin... ##

chk_root(){
        local meid=$(id -u)
        if [ $meid -ne 0 ];
        then
                echo "You must be root user to run this tool"
                exit 999
        fi
}

lin_diff_comparison_report(){

        for FILENAME in `find ${LIN_AUDIT_DIR} -type f -name "RHEL*.txt" | sed 's/\(RHEL.*\.audit\.\).*/\1/' | sort -u`; do
                SERVER=`echo ${FILENAME} | sed 's/.*RHEL-\(.*\)\.\([0-9]\{2\}-\)\{2\}[0-9]\{2\}\.audit\..*/\1/'`
                # this is just for finding the last file
                DATE2=0
                for FILE in $(find ${LIN_AUDIT_DIR} -type f -name "$(basename ${FILENAME})*.txt"); do
                        DATE=`echo ${FILE} | sed 's/.*audit\.\([0-9][0-9]\)-\([0-9][0-9]\)-\([0-9][0-9]\)\.txt/20\3\1\2/'`
                        test "${DATE}" -gt "${DATE2}" && DATE2=${DATE}
                done

                DATE_PREV=`date -d "${DATE2} -1 week" +%m-%d-%y`

                FILE_LAST=${FILENAME}$(date -d "${DATE2}" +%m-%d-%y).txt
                FILE_PREV=${FILENAME}${DATE_PREV}.txt

                LIN_DIFF_OUTPUT=$(echo ${FILE_LAST} | sed 's/\.txt$//').diff
                LIN_DIFF_REPORT=${LIN_DIFF_OUTPUT}.report.zip

                diff -uN ${FILE_LAST} ${FILE_PREV} > ${LIN_DIFF_OUTPUT}
        done
}

win_diff_comparison_report(){

        for FILENAME in `find ${WIN_AUDIT_DIR} -type f -name "WIN*.txt" | sed 's/\(WIN.*\.audit\.\).*/\1/' | sort -u`; do
                SERVER=`echo ${FILENAME} | sed 's/.*WIN-\(.*\)\.\([0-9]\{2\}-\)\{2\}[0-9]\{4\}\.audit\..*/\1/'`
                # this is just for finding the last file
                DATE2=0
                for FILE in $(find ${WIN_AUDIT_DIR} -type f -name "$(basename ${FILENAME})*.txt"); do
                        DATE=`echo ${FILE} | sed 's/.*audit\.\([0-9][0-9]\)-\([0-9][0-9]\)-\([0-9]\{4\}\)\.txt/\3\1\2/'`
                        test "${DATE}" -gt "${DATE2}" && DATE2=${DATE}
                done

                DATE_PREV=`date -d "${DATE2} -1 week" +%m-%d-%Y`

                FILE_LAST=${FILENAME}$(date -d "${DATE2}" +%m-%d-%Y).txt
                FILE_PREV=${FILENAME}${DATE_PREV}.txt

                WIN_DIFF_OUTPUT=$(echo ${FILE_LAST} | sed 's/\.txt$//').diff
                WIN_DIFF_REPORT=${WIN_DIFF_OUTPUT}.report.zip

                diff -uN ${FILE_LAST} ${FILE_PREV} > ${WIN_DIFF_OUTPUT}
        done
}

email_diffs() {
        DATE_LIN=$(date +%m-%d-%y)
        DATE_WIN=$(date +%m-%d-%Y)
        DIFF_REPORT_LIN=${LIN_AUDIT_DIR}/RHEL-audit-${DATE_LIN}.report.zip
        DIFF_REPORT_WIN=${WIN_AUDIT_DIR}/WIN-audit-${DATE_WIN}.report.zip
        zip -P ${PASSWD} -er ${DIFF_REPORT_LIN} ${LIN_AUDIT_DIR}/RHEL*${DATE_LIN}.diff
        zip -P ${PASSWD} -er ${DIFF_REPORT_WIN} ${WIN_AUDIT_DIR}/WIN*${DATE_WIN}.diff
        mutt -x -a ${DIFF_REPORT_LIN} -a ${DIFF_REPORT_WIN} -s "${SUBJECT}" ${RECIPIENTS} < /dev/null
}


## Trigger functions ##
chk_root
lin_diff_comparison_report
win_diff_comparison_report
email_diffs

Open in new window

It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Shell Scripting

From novice to tech pro — start learning today.