bash remove string for logs

If the format will always be as you show, then this should do it:

#!/usr/bin/bash

log_file="/tmp/log.txt"

dt=`head -1 ${log_file} | cut -f3-4 -d' '`

echo "#Date: ${dt}" > /tmp/log.txt.$$

cut -f3- -d' ' ${log_file} >> ${log_file}.$$

#
# Uncomment next line to replace original file with modified file
#
# mv ${log_file}.$$ ${log_file}

Select allOpen in new window

Assumes spaces separating the IP and the date/time will always be 2 fields.

Obviously, you need to modify the variable to set the name of the file.

Hi John,

If the IP address change like to different length like 192.168.128.193 what do I need to modify? The Date time format is not fix too.

I have tested the script, the string is removed but the date is not append at the top

Hi Julio Jose,

You can do that using following awk:

/bin/awk '{
        if ( 1 == NR)
        {
                printf( "#Date: %s %s\n", $2, $3);
        }
        for ( Loc=2; Loc<=NF; Loc++)
        {
                printf( "%s ", $Loc);
        }
        printf( "\n");
}' /tmp/log.txt

Select allOpen in new window

Also above code take less time(most of the times) which has been tested using:

START_NANO_SECONDS=''`/bin/date "+%N"`''
#REQUIRED_COMMANDS.....
END_NANO_SECONDS=''`/bin/date "+%N"`''
echo "TIME TAKEN BY CURRENT  SCRIPT: "''`/usr/bin/expr $END_NANO_SECONDS - $START_NANO_SECONDS`''

Select allOpen in new window

This could have been done using /bin/sed alone, reason for adding /bin/awk due to your requirement:
>> to append date time from first line

2.
Same thing using /bin/sed where total number of /bin/sed being 3 times.

/bin/sed -n 1p  /tmp/log.txt | /bin/sed "s/ *[0-9]*\.[0-9]*\.[0-9]*\.[0-9]* *\([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]\).*/#Date: \1/;"
/bin/sed "s/ *[0-9]*\.[0-9]*\.[0-9]*\.[0-9]* *//;" /tmp/log.txt

Select allOpen in new window

3. Using /bin/sed (1 time)
and /bin/awk(1 time)

/bin/awk '{
        if ( 1 == NR)
        {
                printf( "#Date: %s %s\n", $2, $3);
        }
}' /tmp/log.txt
/bin/sed "s/ *[0-9]*\.[0-9]*\.[0-9]*\.[0-9]* *//;" /tmp/log.txt

Select allOpen in new window

Also always use full path while using any Linux oriented commands.
Example:

unalias ls >/dev/null 2>&1
unset -f ls
ls ()
{
      echo Always use full path for ls command
      echo Julio Jose, Just FYI.
      Ret=12
      return $Ret
}
ls -latr /root
echo $? will be twelve
#Un defining that ls function
unset -f ls

Select allOpen in new window

Mentioned Linux oriented commands
>> Applicable to all operating systems:
SunOS
AIX
HP-UX
Linux ( Redhat Enterprise Linux/CentOS/OpenSUSE/UNIX/....)
Windows_Cygwin_OR_mingw

@Julio Jose,
Mentioning the related script for the same.

#!/bin/ksh
#+-----------+----------------+------------------------------------------------------------------+
#|dd/mmm/yyyy|Name            |Svn History                                                       |
#+-----------+----------------+------------------------------------------------------------------+
#|31/Mar/2017|Murugesan Dinesh|Remove string from server log file.                               |
#+-----------+----------------+------------------------------------------------------------------+
#Function Disp_Log_Msg
#        If no parameter passed
#                Script reading default Log file /tmp/Server.log
#        else
#                Script taking all parameter "$@" as Log file.
#        Disp_Log_Msg function display the log message excluding the first column
#        Removing first column due to presence of IP Address.
unset -f Disp_Log_Msg
Disp_Log_Msg ()
{
        if test 0 -eq $#
        then
                INPUT_LOG_FILE="/tmp/Server.log"
        else
                INPUT_LOG_FILE="$@"
        fi
        if test -f "$INPUT_LOG_FILE"
        then
                OUTPUT_DIR=''`/bin/dirname $INPUT_LOG_FILE`''
                while [ 1 ]
                do
                        OUTPUT_LOG_FILE="$OUTPUT_DIR/Server_"''`/bin/date "+%d_%b_%Y_%S_%N"`''".log"
                        if test -f "$OUTPUT_LOG_FILE"
                        then
                                /bin/sleep 1;
                        else
                                break
                        fi
                done
               # INPUT_LOG_FILE_NUM_OF_LN
                export INPUT_LOG_FILE_NUM_OF_LN=''`/bin/awk 'END { print NR}' "$INPUT_LOG_FILE"`''
                /bin/awk '{
                        INPUT_LOG_FILE_NUM_OF_LN=ENVIRON["INPUT_LOG_FILE_NUM_OF_LN"];
                }
                {
                        if ( 1 == NR)
                        {
                                printf( "#Date: %s %s\n", $2, $3);
                        }
                        for ( Loc=2; Loc<NF; Loc++)
                        {
                                printf( "%s ", $Loc);
                        }
                        if ( NR != INPUT_LOG_FILE_NUM_OF_LN)
                        {
                                printf( "%s\n", $Loc);
                        }
                        else
                        {
                                printf( "%s", $Loc);
                        }
                }' "$INPUT_LOG_FILE" > "$OUTPUT_LOG_FILE"
                if test -f "$OUTPUT_LOG_FILE"
                then
                        echo "Input  log file: $INPUT_LOG_FILE"
                        echo "Output log file: $OUTPUT_LOG_FILE"
                fi
        else
                echo "$INPUT_LOG_FILE No such file"
                Ret=1
                return $Ret
        fi
}
Disp_Log_Msg $@

Select allOpen in new window

Hi Murugesan,

I want to test the script

I have define INPUT_LOG_FILE full path to the login

then set  OUTPUT_DIR=''`$INPUT_LOG_FILE.$$`''

./log2.sh[25]: <the log path>.4158: not found [No such file or directory]

Use the following:

.......
then
        set  OUTPUT_DIR="$INPUT_LOG_FILE.$$"
.....

Select allOpen in new window

Reason for using different output log file:
Possibility of $$ can become same and might overwrite the old output log file.
Hence handled that error inside that script itself.
However it can be handled manually also(for you to move the output file where ever you wish to backup).

Also instead of posting only error,
post the script you have used, not sure how you handled that inside the script.

#!/bin/bash
INPUT_LOG_FILE="/tmp/Server.log"
export  OUTPUT_DIR="$INPUT_LOG_FILE.$$"
echo OUTPUT_DIR $OUTPUT_DIR
echo INPUT_LOG_FILE $INPUT_LOG_FILE
if test ! -d "$OUTPUT_DIR"
then
        echo "Need to create the output directory /bin/mkdir $OUTPUT_DIR"
        echo "I feel this is not the right way."
fi

Select allOpen in new window

sample output here:

MurugesanDinesh@TEL /home/murugesandins [ 0 ]
$ ./Julio_jose.sh
OUTPUT_DIR /tmp/Server.log.7728
INPUT_LOG_FILE /tmp/Server.log
Need to create the output directory /bin/mkdir /tmp/Server.log.7728
I feel this is not the right way.
MurugesanDinesh@TEL /home/murugesandins [ 0 ]
$

Select allOpen in new window

Comment if any objection.https://www.experts-exchange.com/questions/29012555/bash-remove-string-for-logs.html#

Hi, I been away for sometime, I added the "set"

I got the log generate under root instead of the same directory, what is the problem?

log2.sh: line 25: /xx/xx/xx/xx/xx.log.1431: No such file or directory
Input  log file: /xx/xx/xx/xx/xx.log
Output log file: /Server_05_May_2017_16_063950669.log

I will see your current query tomorrow.
Sending this from home.

can you post the code
log2.sh

#!/bin/ksh
#+-----------+----------------+------------------------------------------------------------------+
#|dd/mmm/yyyy|Name            |Svn History                                                       |
#+-----------+----------------+------------------------------------------------------------------+
#|31/Mar/2017|Murugesan Dinesh|Remove string from server log file.                               |
#+-----------+----------------+------------------------------------------------------------------+
#Function Disp_Log_Msg
#        If no parameter passed
#                Script reading default Log file /tmp/Server.log
#        else
#                Script taking all parameter "$@" as Log file.
#        Disp_Log_Msg function display the log message excluding the first column
#        Removing first column due to presence of IP Address.
unset -f Disp_Log_Msg
Disp_Log_Msg ()
{
        if test 0 -eq $#
        then
                INPUT_LOG_FILE="/xx/xx/xx/xx/xx.log"
        else
                INPUT_LOG_FILE="$@"
        fi
        if test -f "$INPUT_LOG_FILE"
        then
                set OUTPUT_DIR=''`$INPUT_LOG_FILE.$$`''
                while [ 1 ]
                do
                        OUTPUT_LOG_FILE="$OUTPUT_DIR/Server_"''`/bin/date "+%d_%b_%Y_%S_%N"`''".log"
                        if test -f "$OUTPUT_LOG_FILE"
                        then
                                /bin/sleep 1;
                        else
                                break
                        fi
                done
               # INPUT_LOG_FILE_NUM_OF_LN
                export INPUT_LOG_FILE_NUM_OF_LN=''`/bin/awk 'END { print NR}' "$INPUT_LOG_FILE"`''
                /bin/awk '{
                        INPUT_LOG_FILE_NUM_OF_LN=ENVIRON["INPUT_LOG_FILE_NUM_OF_LN"];
                }
                {
                        if ( 1 == NR)
                        {
                                printf( "#Date: %s %s\n", $2, $3);
                        }
                        for ( Loc=2; Loc<NF; Loc++)
                        {
                                printf( "%s ", $Loc);
                        }
                        if ( NR != INPUT_LOG_FILE_NUM_OF_LN)
                        {
                                printf( "%s\n", $Loc);
                        }
                        else
                        {
                                printf( "%s", $Loc);
                        }
                }' "$INPUT_LOG_FILE" > "$OUTPUT_LOG_FILE"
                if test -f "$OUTPUT_LOG_FILE"
                then
                        echo "Input  log file: $INPUT_LOG_FILE"
                        echo "Output log file: $OUTPUT_LOG_FILE"
                fi
        else
                echo "$INPUT_LOG_FILE No such file"
                Ret=1
                return $Ret
        fi
}
Disp_Log_Msg $@

Select allOpen in new window

Hi Julio Jose,

I copied and test the code you have pasted here:
Sample output at cygwin:

$ /bin/cp -ip /bin/bash /bin/ksh
$ ./log2.sh
/xx/xx/xx/xx/xx.log No such file
$ echo $?
1
$ /bin/date
Tue May 16 10:02:29 IST 2017

Select allOpen in new window

a)
Give the exact command you are using to execute log2.sh

b)
I have also verified the type of the file being used. Give the output for following command

$ /usr/bin/file ./log2.sh
./log2.sh: Korn shell script, ASCII text executable

Select allOpen in new window

c)
Give the output for following commands:

/usr/bin/od -bc  ./log2.sh | /bin/egrep "\\\r"
echo $?

Select allOpen in new window

This is the output

#sh log2.sh
# /usr/bin/od -bc ./log2.sh | /bin/egrep "\\\r"
# /usr/bin/file ./log2.sh
./log2.sh: Korn shell script text executable

Here goes updated script:

#!/bin/ksh
#+-----------+----------------+---------------------------------------------------------------------+
#|dd/mmm/yyyy|Name            | Svn History                                                         |
#+-----------+----------------+---------------------------------------------------------------------+
#|24/May/2017|Murugesan Dinesh| Changed INPUT_LOG_FILE to have /tmp/Server.log                      |
#|                              Changed OUTPUT_DIR to point to source directory (/tmp) of input file|
#|                              Also displaying output in current terminal.                         |
#|                              Removed following variable in current update.                       |
#|                              INPUT_LOG_FILE_NUM_OF_LN                                            |
#|31/Mar/2017|Murugesan Dinesh| Remove string from server log file.                                 |
#+-----------+----------------+---------------------------------------------------------------------+
#Function Disp_Log_Msg
#        If no parameter passed
#                Script reading default Log file /tmp/Server.log
#        else
#                Script taking all parameter "$@" as Log file.
#        Disp_Log_Msg function display the log message excluding the first column
#        Removing first column due to presence of IP Address.
unset -f Disp_Log_Msg
Disp_Log_Msg ()
{
        if test 0 -eq $#
        then
                INPUT_LOG_FILE="/tmp/Server.log"
        else
                INPUT_LOG_FILE="$@"
        fi
        if test -f "$INPUT_LOG_FILE"
        then
                OUTPUT_DIR=''`/bin/dirname $INPUT_LOG_FILE`''
		if test "." = "$OUTPUT_DIR"
		then
			OUTPUT_DIR="$PWD"
		fi
                while [ 1 ]
                do
                        OUTPUT_LOG_FILE="$OUTPUT_DIR/Server_"''`/bin/date "+%d_%b_%Y_%S_%N"`''".log"
                        if test -f "$OUTPUT_LOG_FILE"
                        then
                                /bin/sleep 1;
                        else
                                break
                        fi
                done
		echo "Current output:"
		echo "------------------------:"
                /bin/awk '{
                        if ( 1 == NR)
                        {
                                printf( "#Date: %s %s\n", $2, $3);
                        }
                        for ( Loc=2; Loc<NF; Loc++)
                        {
                                printf( "%s ", $Loc);
                        }
                        printf( "%s\n", $Loc);
                }' "$INPUT_LOG_FILE" 2>&1 | /usr/bin/tee -a "$OUTPUT_LOG_FILE"
		echo "------------------------:"
                if test -f "$OUTPUT_LOG_FILE"
                then
                        echo "Input  log file: $INPUT_LOG_FILE"
                        echo "Output log file: $OUTPUT_LOG_FILE"
                fi
        else
                echo "$INPUT_LOG_FILE No such file"
                Ret=1
                return $Ret
        fi
}
Disp_Log_Msg $@

Select allOpen in new window

Related sample output here:

$ ./29012555.sh
Current output:
------------------------:
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
------------------------:
Input  log file: /tmp/Server.log
Output log file: /tmp/Server_24_May_2017_49_677778500.log
$ /bin/cat /tmp/Server_24_May_2017_49_677778500.log
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
$ ./29012555.sh  /root/Server.log
Current output:
------------------------:
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
------------------------:
Input  log file: /root/Server.log
Output log file: /root/Server_24_May_2017_19_936798800.log
$ /bin/cat /root/Server_24_May_2017_19_936798800.log
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
$

Select allOpen in new window

Test the updated code and let us know if getting expected output for the same.