Link to home
Start Free TrialLog in
Avatar of Julio Jose
Julio JoseFlag for Malaysia

asked on

bash remove string for logs

Hi, I have the log file, each of the line first few string "<space>10.99.9.99<space>" need to remove and the IP address could be different for each log files, I need the script to count the character to remove for each line

<space>10.99.9.99<space>2017-03-27 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Open in new window


After process remove the string the script need to append date time from first line of the log to the top the log file

example :

#Date: 2017-03-27 23:44:49
2017-03-27 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Open in new window

Avatar of johnsone
johnsone
Flag of United States of America image

If the format will always be as you show, then this should do it:
#!/usr/bin/bash

log_file="/tmp/log.txt"

dt=`head -1 ${log_file} | cut -f3-4 -d' '`

echo "#Date: ${dt}" > /tmp/log.txt.$$

cut -f3- -d' ' ${log_file} >> ${log_file}.$$

#
# Uncomment next line to replace original file with modified file
#
# mv ${log_file}.$$ ${log_file}

Open in new window

Assumes spaces separating the IP and the date/time will always be 2 fields.

Obviously, you need to modify the variable to set the name of the file.
Avatar of Julio Jose

ASKER

Hi John,

If the IP address change like to different length like 192.168.128.193 what do I need to modify? The Date time format is not fix too.

I have tested the script, the string is removed but the date is not append at the top
SOLUTION
Avatar of johnsone
johnsone
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Hi Julio Jose,

You can do that using following awk:
/bin/awk '{
        if ( 1 == NR)
        {
                printf( "#Date: %s %s\n", $2, $3);
        }
        for ( Loc=2; Loc<=NF; Loc++)
        {
                printf( "%s ", $Loc);
        }
        printf( "\n");
}' /tmp/log.txt

Open in new window


Also above code take less time(most of the times) which has been tested using:
START_NANO_SECONDS=''`/bin/date "+%N"`''
#REQUIRED_COMMANDS.....
END_NANO_SECONDS=''`/bin/date "+%N"`''
echo "TIME TAKEN BY CURRENT  SCRIPT: "''`/usr/bin/expr $END_NANO_SECONDS - $START_NANO_SECONDS`''

Open in new window


This could have been done using /bin/sed alone, reason for adding /bin/awk due to your requirement:
>> to append date time from first line
2.
Same thing using /bin/sed where total number of /bin/sed being 3 times.
/bin/sed -n 1p  /tmp/log.txt | /bin/sed "s/ *[0-9]*\.[0-9]*\.[0-9]*\.[0-9]* *\([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]\).*/#Date: \1/;"
/bin/sed "s/ *[0-9]*\.[0-9]*\.[0-9]*\.[0-9]* *//;" /tmp/log.txt

Open in new window

3. Using /bin/sed (1 time)
and /bin/awk(1 time)
/bin/awk '{
        if ( 1 == NR)
        {
                printf( "#Date: %s %s\n", $2, $3);
        }
}' /tmp/log.txt
/bin/sed "s/ *[0-9]*\.[0-9]*\.[0-9]*\.[0-9]* *//;" /tmp/log.txt

Open in new window


Also always use full path while using any Linux oriented commands.
Example:
unalias ls >/dev/null 2>&1
unset -f ls
ls ()
{
      echo Always use full path for ls command
      echo Julio Jose, Just FYI.
      Ret=12
      return $Ret
}
ls -latr /root
echo $? will be twelve
#Un defining that ls function
unset -f ls

Open in new window


Mentioned Linux oriented commands
>> Applicable to all operating systems:
SunOS
AIX
HP-UX
Linux ( Redhat Enterprise Linux/CentOS/OpenSUSE/UNIX/....)
Windows_Cygwin_OR_mingw
@Julio Jose,
Mentioning the related script for the same.
#!/bin/ksh
#+-----------+----------------+------------------------------------------------------------------+
#|dd/mmm/yyyy|Name            |Svn History                                                       |
#+-----------+----------------+------------------------------------------------------------------+
#|31/Mar/2017|Murugesan Dinesh|Remove string from server log file.                               |
#+-----------+----------------+------------------------------------------------------------------+
#Function Disp_Log_Msg
#        If no parameter passed
#                Script reading default Log file /tmp/Server.log
#        else
#                Script taking all parameter "$@" as Log file.
#        Disp_Log_Msg function display the log message excluding the first column
#        Removing first column due to presence of IP Address.
unset -f Disp_Log_Msg
Disp_Log_Msg ()
{
        if test 0 -eq $#
        then
                INPUT_LOG_FILE="/tmp/Server.log"
        else
                INPUT_LOG_FILE="$@"
        fi
        if test -f "$INPUT_LOG_FILE"
        then
                OUTPUT_DIR=''`/bin/dirname $INPUT_LOG_FILE`''
                while [ 1 ]
                do
                        OUTPUT_LOG_FILE="$OUTPUT_DIR/Server_"''`/bin/date "+%d_%b_%Y_%S_%N"`''".log"
                        if test -f "$OUTPUT_LOG_FILE"
                        then
                                /bin/sleep 1;
                        else
                                break
                        fi
                done
               # INPUT_LOG_FILE_NUM_OF_LN
                export INPUT_LOG_FILE_NUM_OF_LN=''`/bin/awk 'END { print NR}' "$INPUT_LOG_FILE"`''
                /bin/awk '{
                        INPUT_LOG_FILE_NUM_OF_LN=ENVIRON["INPUT_LOG_FILE_NUM_OF_LN"];
                }
                {
                        if ( 1 == NR)
                        {
                                printf( "#Date: %s %s\n", $2, $3);
                        }
                        for ( Loc=2; Loc<NF; Loc++)
                        {
                                printf( "%s ", $Loc);
                        }
                        if ( NR != INPUT_LOG_FILE_NUM_OF_LN)
                        {
                                printf( "%s\n", $Loc);
                        }
                        else
                        {
                                printf( "%s", $Loc);
                        }
                }' "$INPUT_LOG_FILE" > "$OUTPUT_LOG_FILE"
                if test -f "$OUTPUT_LOG_FILE"
                then
                        echo "Input  log file: $INPUT_LOG_FILE"
                        echo "Output log file: $OUTPUT_LOG_FILE"
                fi
        else
                echo "$INPUT_LOG_FILE No such file"
                Ret=1
                return $Ret
        fi
}
Disp_Log_Msg $@

Open in new window

Hi Murugesan,

I want to test the script

I have define INPUT_LOG_FILE full path to the login

then set  OUTPUT_DIR=''`$INPUT_LOG_FILE.$$`''

./log2.sh[25]: <the log path>.4158: not found [No such file or directory]
Use the following:
.......
then
        set  OUTPUT_DIR="$INPUT_LOG_FILE.$$"
.....

Open in new window

Reason for using different output log file:
Possibility of $$ can become same and might overwrite the old output log file.
Hence handled that error inside that script itself.
However it can be handled manually also(for you to move the output file where ever you wish to backup).
Also instead of posting only error,
post the script you have used, not sure how you handled that inside the script.
#!/bin/bash
INPUT_LOG_FILE="/tmp/Server.log"
export  OUTPUT_DIR="$INPUT_LOG_FILE.$$"
echo OUTPUT_DIR $OUTPUT_DIR
echo INPUT_LOG_FILE $INPUT_LOG_FILE
if test ! -d "$OUTPUT_DIR"
then
        echo "Need to create the output directory /bin/mkdir $OUTPUT_DIR"
        echo "I feel this is not the right way."
fi

Open in new window

sample output here:
MurugesanDinesh@TEL /home/murugesandins [ 0 ]
$ ./Julio_jose.sh
OUTPUT_DIR /tmp/Server.log.7728
INPUT_LOG_FILE /tmp/Server.log
Need to create the output directory /bin/mkdir /tmp/Server.log.7728
I feel this is not the right way.
MurugesanDinesh@TEL /home/murugesandins [ 0 ]
$

Open in new window

ASKER CERTIFIED SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Hi, I been away for sometime, I added the "set"

I got the log generate under root instead of the same directory, what is the problem?

log2.sh: line 25: /xx/xx/xx/xx/xx.log.1431: No such file or directory
Input  log file: /xx/xx/xx/xx/xx.log
Output log file: /Server_05_May_2017_16_063950669.log
I will see your current query tomorrow.
Sending this from home.
can you post the code
log2.sh
#!/bin/ksh
#+-----------+----------------+------------------------------------------------------------------+
#|dd/mmm/yyyy|Name            |Svn History                                                       |
#+-----------+----------------+------------------------------------------------------------------+
#|31/Mar/2017|Murugesan Dinesh|Remove string from server log file.                               |
#+-----------+----------------+------------------------------------------------------------------+
#Function Disp_Log_Msg
#        If no parameter passed
#                Script reading default Log file /tmp/Server.log
#        else
#                Script taking all parameter "$@" as Log file.
#        Disp_Log_Msg function display the log message excluding the first column
#        Removing first column due to presence of IP Address.
unset -f Disp_Log_Msg
Disp_Log_Msg ()
{
        if test 0 -eq $#
        then
                INPUT_LOG_FILE="/xx/xx/xx/xx/xx.log"
        else
                INPUT_LOG_FILE="$@"
        fi
        if test -f "$INPUT_LOG_FILE"
        then
                set OUTPUT_DIR=''`$INPUT_LOG_FILE.$$`''
                while [ 1 ]
                do
                        OUTPUT_LOG_FILE="$OUTPUT_DIR/Server_"''`/bin/date "+%d_%b_%Y_%S_%N"`''".log"
                        if test -f "$OUTPUT_LOG_FILE"
                        then
                                /bin/sleep 1;
                        else
                                break
                        fi
                done
               # INPUT_LOG_FILE_NUM_OF_LN
                export INPUT_LOG_FILE_NUM_OF_LN=''`/bin/awk 'END { print NR}' "$INPUT_LOG_FILE"`''
                /bin/awk '{
                        INPUT_LOG_FILE_NUM_OF_LN=ENVIRON["INPUT_LOG_FILE_NUM_OF_LN"];
                }
                {
                        if ( 1 == NR)
                        {
                                printf( "#Date: %s %s\n", $2, $3);
                        }
                        for ( Loc=2; Loc<NF; Loc++)
                        {
                                printf( "%s ", $Loc);
                        }
                        if ( NR != INPUT_LOG_FILE_NUM_OF_LN)
                        {
                                printf( "%s\n", $Loc);
                        }
                        else
                        {
                                printf( "%s", $Loc);
                        }
                }' "$INPUT_LOG_FILE" > "$OUTPUT_LOG_FILE"
                if test -f "$OUTPUT_LOG_FILE"
                then
                        echo "Input  log file: $INPUT_LOG_FILE"
                        echo "Output log file: $OUTPUT_LOG_FILE"
                fi
        else
                echo "$INPUT_LOG_FILE No such file"
                Ret=1
                return $Ret
        fi
}
Disp_Log_Msg $@

Open in new window

Hi Julio Jose,

I copied and test the code you have pasted here:
Sample output at cygwin:
$ /bin/cp -ip /bin/bash /bin/ksh
$ ./log2.sh
/xx/xx/xx/xx/xx.log No such file
$ echo $?
1
$ /bin/date
Tue May 16 10:02:29 IST 2017

Open in new window


a)
Give the exact command you are using to execute log2.sh

b)
I have also verified the type of the file being used. Give the output for following command
$ /usr/bin/file ./log2.sh
./log2.sh: Korn shell script, ASCII text executable

Open in new window


c)
Give the output for following commands:
/usr/bin/od -bc  ./log2.sh | /bin/egrep "\\\r"
echo $?

Open in new window

This is the output

#sh log2.sh
# /usr/bin/od -bc ./log2.sh | /bin/egrep "\\\r"
# /usr/bin/file ./log2.sh
./log2.sh: Korn shell script text executable
Here goes updated script:
#!/bin/ksh
#+-----------+----------------+---------------------------------------------------------------------+
#|dd/mmm/yyyy|Name            | Svn History                                                         |
#+-----------+----------------+---------------------------------------------------------------------+
#|24/May/2017|Murugesan Dinesh| Changed INPUT_LOG_FILE to have /tmp/Server.log                      |
#|                              Changed OUTPUT_DIR to point to source directory (/tmp) of input file|
#|                              Also displaying output in current terminal.                         |
#|                              Removed following variable in current update.                       |
#|                              INPUT_LOG_FILE_NUM_OF_LN                                            |
#|31/Mar/2017|Murugesan Dinesh| Remove string from server log file.                                 |
#+-----------+----------------+---------------------------------------------------------------------+
#Function Disp_Log_Msg
#        If no parameter passed
#                Script reading default Log file /tmp/Server.log
#        else
#                Script taking all parameter "$@" as Log file.
#        Disp_Log_Msg function display the log message excluding the first column
#        Removing first column due to presence of IP Address.
unset -f Disp_Log_Msg
Disp_Log_Msg ()
{
        if test 0 -eq $#
        then
                INPUT_LOG_FILE="/tmp/Server.log"
        else
                INPUT_LOG_FILE="$@"
        fi
        if test -f "$INPUT_LOG_FILE"
        then
                OUTPUT_DIR=''`/bin/dirname $INPUT_LOG_FILE`''
		if test "." = "$OUTPUT_DIR"
		then
			OUTPUT_DIR="$PWD"
		fi
                while [ 1 ]
                do
                        OUTPUT_LOG_FILE="$OUTPUT_DIR/Server_"''`/bin/date "+%d_%b_%Y_%S_%N"`''".log"
                        if test -f "$OUTPUT_LOG_FILE"
                        then
                                /bin/sleep 1;
                        else
                                break
                        fi
                done
		echo "Current output:"
		echo "------------------------:"
                /bin/awk '{
                        if ( 1 == NR)
                        {
                                printf( "#Date: %s %s\n", $2, $3);
                        }
                        for ( Loc=2; Loc<NF; Loc++)
                        {
                                printf( "%s ", $Loc);
                        }
                        printf( "%s\n", $Loc);
                }' "$INPUT_LOG_FILE" 2>&1 | /usr/bin/tee -a "$OUTPUT_LOG_FILE"
		echo "------------------------:"
                if test -f "$OUTPUT_LOG_FILE"
                then
                        echo "Input  log file: $INPUT_LOG_FILE"
                        echo "Output log file: $OUTPUT_LOG_FILE"
                fi
        else
                echo "$INPUT_LOG_FILE No such file"
                Ret=1
                return $Ret
        fi
}
Disp_Log_Msg $@

Open in new window

Related sample output here:
$ ./29012555.sh
Current output:
------------------------:
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
------------------------:
Input  log file: /tmp/Server.log
Output log file: /tmp/Server_24_May_2017_49_677778500.log
$ /bin/cat /tmp/Server_24_May_2017_49_677778500.log
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
$ ./29012555.sh  /root/Server.log
Current output:
------------------------:
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
------------------------:
Input  log file: /root/Server.log
Output log file: /root/Server_24_May_2017_19_936798800.log
$ /bin/cat /root/Server_24_May_2017_19_936798800.log
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
$

Open in new window

Test the updated code and let us know if getting expected output for the same.