bash remove string for logs

Hi, I have the log file, each of the line first few string "<space>10.99.9.99<space>" need to remove and the IP address could be different for each log files, I need the script to count the character to remove for each line

<space>10.99.9.99<space>2017-03-27 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Open in new window

After process remove the string the script need to append date time from first line of the log to the top the log file

example :

#Date: 2017-03-27 23:44:49
2017-03-27 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Open in new window

Julio JoseAsked: 2017-03-29

Who is Participating?

Experts Exchange Solution brought to you by

Enjoy your complimentary solution view.

Get this solution by purchasing an Individual license! Start your 7-day free trial.

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

LVL 37

johnsoneSenior Oracle DBACommented: 2017-03-29

If the format will always be as you show, then this should do it:

#!/usr/bin/bash

log_file="/tmp/log.txt"

dt=`head -1 ${log_file} | cut -f3-4 -d' '`

echo "#Date: ${dt}" > /tmp/log.txt.$$

cut -f3- -d' ' ${log_file} >> ${log_file}.$$

#
# Uncomment next line to replace original file with modified file
#
# mv ${log_file}.$$ ${log_file}

Open in new window

Assumes spaces separating the IP and the date/time will always be 2 fields.

Obviously, you need to modify the variable to set the name of the file.

Julio JoseAuthor Commented: 2017-03-29

Hi John,

If the IP address change like to different length like 192.168.128.193 what do I need to modify? The Date time format is not fix too.

I have tested the script, the string is removed but the date is not append at the top

LVL 37

johnsoneSenior Oracle DBACommented: 2017-03-29

You do not need to change it for length of IP address. The script is looking for spaces.

Can you provide sample data of what didn't work? This is the sample file I used:

 10.99.9.99 2017-03-27 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 10.99.9.99 2017-03-27 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 10.99.9.99 2017-03-27 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 10.99.9.99 2017-03-27 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Open in new window

Which produced this result:

#Date: 2017-03-27 23:44:49
2017-03-27 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2017-03-27 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2017-03-27 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2017-03-27 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Open in new window

If the date format changes it isn't a big deal. If the spacing changes, you would need to account for that.

LVL 11

murugesandinsShell_script Automation /bin/bash /bin/bash.exe /bin/ksh /bin/mksh.exe AIX C C++ CYGWIN_NT HP-UX Linux MINGW32 MINGW64 SunOS Windows_NT started powershellCommented: 2017-03-30

Hi Julio Jose,

You can do that using following awk:

/bin/awk '{
        if ( 1 == NR)
        {
                printf( "#Date: %s %s\n", $2, $3);
        }
        for ( Loc=2; Loc<=NF; Loc++)
        {
                printf( "%s ", $Loc);
        }
        printf( "\n");
}' /tmp/log.txt

Open in new window

Also above code take less time(most of the times) which has been tested using:

START_NANO_SECONDS=''`/bin/date "+%N"`''
#REQUIRED_COMMANDS.....
END_NANO_SECONDS=''`/bin/date "+%N"`''
echo "TIME TAKEN BY CURRENT  SCRIPT: "''`/usr/bin/expr $END_NANO_SECONDS - $START_NANO_SECONDS`''

Open in new window

This could have been done using /bin/sed alone, reason for adding /bin/awk due to your requirement:
>> to append date time from first line

LVL 11

murugesandinsShell_script Automation /bin/bash /bin/bash.exe /bin/ksh /bin/mksh.exe AIX C C++ CYGWIN_NT HP-UX Linux MINGW32 MINGW64 SunOS Windows_NT started powershellCommented: 2017-03-30

2.
Same thing using /bin/sed where total number of /bin/sed being 3 times.

/bin/sed -n 1p  /tmp/log.txt | /bin/sed "s/ *[0-9]*\.[0-9]*\.[0-9]*\.[0-9]* *\([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]\).*/#Date: \1/;"
/bin/sed "s/ *[0-9]*\.[0-9]*\.[0-9]*\.[0-9]* *//;" /tmp/log.txt

Open in new window

3. Using /bin/sed (1 time)
and /bin/awk(1 time)

/bin/awk '{
        if ( 1 == NR)
        {
                printf( "#Date: %s %s\n", $2, $3);
        }
}' /tmp/log.txt
/bin/sed "s/ *[0-9]*\.[0-9]*\.[0-9]*\.[0-9]* *//;" /tmp/log.txt

Open in new window

Also always use full path while using any Linux oriented commands.
Example:

unalias ls >/dev/null 2>&1
unset -f ls
ls ()
{
      echo Always use full path for ls command
      echo Julio Jose, Just FYI.
      Ret=12
      return $Ret
}
ls -latr /root
echo $? will be twelve
#Un defining that ls function
unset -f ls

Open in new window

Mentioned Linux oriented commands
>> Applicable to all operating systems:
SunOS
AIX
HP-UX
Linux ( Redhat Enterprise Linux/CentOS/OpenSUSE/UNIX/....)
Windows_Cygwin_OR_mingw

LVL 11

murugesandinsShell_script Automation /bin/bash /bin/bash.exe /bin/ksh /bin/mksh.exe AIX C C++ CYGWIN_NT HP-UX Linux MINGW32 MINGW64 SunOS Windows_NT started powershellCommented: 2017-03-30

@Julio Jose,
Mentioning the related script for the same.

#!/bin/ksh
#+-----------+----------------+------------------------------------------------------------------+
#|dd/mmm/yyyy|Name            |Svn History                                                       |
#+-----------+----------------+------------------------------------------------------------------+
#|31/Mar/2017|Murugesan Dinesh|Remove string from server log file.                               |
#+-----------+----------------+------------------------------------------------------------------+
#Function Disp_Log_Msg
#        If no parameter passed
#                Script reading default Log file /tmp/Server.log
#        else
#                Script taking all parameter "$@" as Log file.
#        Disp_Log_Msg function display the log message excluding the first column
#        Removing first column due to presence of IP Address.
unset -f Disp_Log_Msg
Disp_Log_Msg ()
{
        if test 0 -eq $#
        then
                INPUT_LOG_FILE="/tmp/Server.log"
        else
                INPUT_LOG_FILE="$@"
        fi
        if test -f "$INPUT_LOG_FILE"
        then
                OUTPUT_DIR=''`/bin/dirname $INPUT_LOG_FILE`''
                while [ 1 ]
                do
                        OUTPUT_LOG_FILE="$OUTPUT_DIR/Server_"''`/bin/date "+%d_%b_%Y_%S_%N"`''".log"
                        if test -f "$OUTPUT_LOG_FILE"
                        then
                                /bin/sleep 1;
                        else
                                break
                        fi
                done
               # INPUT_LOG_FILE_NUM_OF_LN
                export INPUT_LOG_FILE_NUM_OF_LN=''`/bin/awk 'END { print NR}' "$INPUT_LOG_FILE"`''
                /bin/awk '{
                        INPUT_LOG_FILE_NUM_OF_LN=ENVIRON["INPUT_LOG_FILE_NUM_OF_LN"];
                }
                {
                        if ( 1 == NR)
                        {
                                printf( "#Date: %s %s\n", $2, $3);
                        }
                        for ( Loc=2; Loc<NF; Loc++)
                        {
                                printf( "%s ", $Loc);
                        }
                        if ( NR != INPUT_LOG_FILE_NUM_OF_LN)
                        {
                                printf( "%s\n", $Loc);
                        }
                        else
                        {
                                printf( "%s", $Loc);
                        }
                }' "$INPUT_LOG_FILE" > "$OUTPUT_LOG_FILE"
                if test -f "$OUTPUT_LOG_FILE"
                then
                        echo "Input  log file: $INPUT_LOG_FILE"
                        echo "Output log file: $OUTPUT_LOG_FILE"
                fi
        else
                echo "$INPUT_LOG_FILE No such file"
                Ret=1
                return $Ret
        fi
}
Disp_Log_Msg $@

Open in new window

Julio JoseAuthor Commented: 2017-03-30

Hi Murugesan,

I want to test the script

I have define INPUT_LOG_FILE full path to the login

then set OUTPUT_DIR=''`$INPUT_LOG_FILE.$$`''

./log2.sh[25]: <the log path>.4158: not found [No such file or directory]

LVL 11

murugesandinsShell_script Automation /bin/bash /bin/bash.exe /bin/ksh /bin/mksh.exe AIX C C++ CYGWIN_NT HP-UX Linux MINGW32 MINGW64 SunOS Windows_NT started powershellCommented: 2017-03-30

Use the following:

.......
then
        set  OUTPUT_DIR="$INPUT_LOG_FILE.$$"
.....

Open in new window

LVL 11

murugesandinsShell_script Automation /bin/bash /bin/bash.exe /bin/ksh /bin/mksh.exe AIX C C++ CYGWIN_NT HP-UX Linux MINGW32 MINGW64 SunOS Windows_NT started powershellCommented: 2017-03-30

Reason for using different output log file:
Possibility of $$ can become same and might overwrite the old output log file.
Hence handled that error inside that script itself.
However it can be handled manually also(for you to move the output file where ever you wish to backup).

LVL 11

murugesandinsShell_script Automation /bin/bash /bin/bash.exe /bin/ksh /bin/mksh.exe AIX C C++ CYGWIN_NT HP-UX Linux MINGW32 MINGW64 SunOS Windows_NT started powershellCommented: 2017-03-30

Also instead of posting only error,
post the script you have used, not sure how you handled that inside the script.

#!/bin/bash
INPUT_LOG_FILE="/tmp/Server.log"
export  OUTPUT_DIR="$INPUT_LOG_FILE.$$"
echo OUTPUT_DIR $OUTPUT_DIR
echo INPUT_LOG_FILE $INPUT_LOG_FILE
if test ! -d "$OUTPUT_DIR"
then
        echo "Need to create the output directory /bin/mkdir $OUTPUT_DIR"
        echo "I feel this is not the right way."
fi

Open in new window

sample output here:

MurugesanDinesh@TEL /home/murugesandins [ 0 ]
$ ./Julio_jose.sh
OUTPUT_DIR /tmp/Server.log.7728
INPUT_LOG_FILE /tmp/Server.log
Need to create the output directory /bin/mkdir /tmp/Server.log.7728
I feel this is not the right way.
MurugesanDinesh@TEL /home/murugesandins [ 0 ]
$

Open in new window

LVL 11

murugesandinsShell_script Automation /bin/bash /bin/bash.exe /bin/ksh /bin/mksh.exe AIX C C++ CYGWIN_NT HP-UX Linux MINGW32 MINGW64 SunOS Windows_NT started powershellCommented: 2017-04-13

Reason for given error:

set  OUTPUT_DIR=''`$INPUT_LOG_FILE.$$`''
 ./log2.sh[25]: <the log path>.4158: not found [No such file or directory]

Open in new window

Meaning of following statement:
set OUTPUT_DIR=''`$INPUT_LOG_FILE.$$`''
Assume that INPUT_LOG_FILE having value "hai.txt" and $$ is 1234
Hence INPUT_LOG_FILE.$$ means hai.txt.1234
`$INPUT_LOG_FILE.$$` means execute the command hai.txt.1234
which is invalid. Since there is not file hai.txt.1234
it reported the error
<the log path>.4158: not found [No such file or directory]

Comment from author for related update or query ??

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial

LVL 11

murugesandinsShell_script Automation /bin/bash /bin/bash.exe /bin/ksh /bin/mksh.exe AIX C C++ CYGWIN_NT HP-UX Linux MINGW32 MINGW64 SunOS Windows_NT started powershellCommented: 2017-05-02

Comment if any objection.https://www.experts-exchange.com/questions/29012555/bash-remove-string-for-logs.html#

Julio JoseAuthor Commented: 2017-05-05

Hi, I been away for sometime, I added the "set"

I got the log generate under root instead of the same directory, what is the problem?

log2.sh: line 25: /xx/xx/xx/xx/xx.log.1431: No such file or directory
Input log file: /xx/xx/xx/xx/xx.log
Output log file: /Server_05_May_2017_16_063950669.log

LVL 11

murugesandinsShell_script Automation /bin/bash /bin/bash.exe /bin/ksh /bin/mksh.exe AIX C C++ CYGWIN_NT HP-UX Linux MINGW32 MINGW64 SunOS Windows_NT started powershellCommented: 2017-05-07

I will see your current query tomorrow.
Sending this from home.

LVL 11

murugesandinsShell_script Automation /bin/bash /bin/bash.exe /bin/ksh /bin/mksh.exe AIX C C++ CYGWIN_NT HP-UX Linux MINGW32 MINGW64 SunOS Windows_NT started powershellCommented: 2017-05-07

can you post the code
log2.sh

Julio JoseAuthor Commented: 2017-05-14

#!/bin/ksh
#+-----------+----------------+------------------------------------------------------------------+
#|dd/mmm/yyyy|Name            |Svn History                                                       |
#+-----------+----------------+------------------------------------------------------------------+
#|31/Mar/2017|Murugesan Dinesh|Remove string from server log file.                               |
#+-----------+----------------+------------------------------------------------------------------+
#Function Disp_Log_Msg
#        If no parameter passed
#                Script reading default Log file /tmp/Server.log
#        else
#                Script taking all parameter "$@" as Log file.
#        Disp_Log_Msg function display the log message excluding the first column
#        Removing first column due to presence of IP Address.
unset -f Disp_Log_Msg
Disp_Log_Msg ()
{
        if test 0 -eq $#
        then
                INPUT_LOG_FILE="/xx/xx/xx/xx/xx.log"
        else
                INPUT_LOG_FILE="$@"
        fi
        if test -f "$INPUT_LOG_FILE"
        then
                set OUTPUT_DIR=''`$INPUT_LOG_FILE.$$`''
                while [ 1 ]
                do
                        OUTPUT_LOG_FILE="$OUTPUT_DIR/Server_"''`/bin/date "+%d_%b_%Y_%S_%N"`''".log"
                        if test -f "$OUTPUT_LOG_FILE"
                        then
                                /bin/sleep 1;
                        else
                                break
                        fi
                done
               # INPUT_LOG_FILE_NUM_OF_LN
                export INPUT_LOG_FILE_NUM_OF_LN=''`/bin/awk 'END { print NR}' "$INPUT_LOG_FILE"`''
                /bin/awk '{
                        INPUT_LOG_FILE_NUM_OF_LN=ENVIRON["INPUT_LOG_FILE_NUM_OF_LN"];
                }
                {
                        if ( 1 == NR)
                        {
                                printf( "#Date: %s %s\n", $2, $3);
                        }
                        for ( Loc=2; Loc<NF; Loc++)
                        {
                                printf( "%s ", $Loc);
                        }
                        if ( NR != INPUT_LOG_FILE_NUM_OF_LN)
                        {
                                printf( "%s\n", $Loc);
                        }
                        else
                        {
                                printf( "%s", $Loc);
                        }
                }' "$INPUT_LOG_FILE" > "$OUTPUT_LOG_FILE"
                if test -f "$OUTPUT_LOG_FILE"
                then
                        echo "Input  log file: $INPUT_LOG_FILE"
                        echo "Output log file: $OUTPUT_LOG_FILE"
                fi
        else
                echo "$INPUT_LOG_FILE No such file"
                Ret=1
                return $Ret
        fi
}
Disp_Log_Msg $@

Open in new window

LVL 11

murugesandinsShell_script Automation /bin/bash /bin/bash.exe /bin/ksh /bin/mksh.exe AIX C C++ CYGWIN_NT HP-UX Linux MINGW32 MINGW64 SunOS Windows_NT started powershellCommented: 2017-05-15

Hi Julio Jose,

I copied and test the code you have pasted here:
Sample output at cygwin:

$ /bin/cp -ip /bin/bash /bin/ksh
$ ./log2.sh
/xx/xx/xx/xx/xx.log No such file
$ echo $?
1
$ /bin/date
Tue May 16 10:02:29 IST 2017

Open in new window

a)
Give the exact command you are using to execute log2.sh

b)
I have also verified the type of the file being used. Give the output for following command

$ /usr/bin/file ./log2.sh
./log2.sh: Korn shell script, ASCII text executable

Open in new window

c)
Give the output for following commands:

/usr/bin/od -bc  ./log2.sh | /bin/egrep "\\\r"
echo $?

Open in new window

Julio JoseAuthor Commented: 2017-05-23

This is the output

#sh log2.sh
# /usr/bin/od -bc ./log2.sh | /bin/egrep "\\\r"
# /usr/bin/file ./log2.sh
./log2.sh: Korn shell script text executable

LVL 11

murugesandinsShell_script Automation /bin/bash /bin/bash.exe /bin/ksh /bin/mksh.exe AIX C C++ CYGWIN_NT HP-UX Linux MINGW32 MINGW64 SunOS Windows_NT started powershellCommented: 2017-05-23

Here goes updated script:

#!/bin/ksh
#+-----------+----------------+---------------------------------------------------------------------+
#|dd/mmm/yyyy|Name            | Svn History                                                         |
#+-----------+----------------+---------------------------------------------------------------------+
#|24/May/2017|Murugesan Dinesh| Changed INPUT_LOG_FILE to have /tmp/Server.log                      |
#|                              Changed OUTPUT_DIR to point to source directory (/tmp) of input file|
#|                              Also displaying output in current terminal.                         |
#|                              Removed following variable in current update.                       |
#|                              INPUT_LOG_FILE_NUM_OF_LN                                            |
#|31/Mar/2017|Murugesan Dinesh| Remove string from server log file.                                 |
#+-----------+----------------+---------------------------------------------------------------------+
#Function Disp_Log_Msg
#        If no parameter passed
#                Script reading default Log file /tmp/Server.log
#        else
#                Script taking all parameter "$@" as Log file.
#        Disp_Log_Msg function display the log message excluding the first column
#        Removing first column due to presence of IP Address.
unset -f Disp_Log_Msg
Disp_Log_Msg ()
{
        if test 0 -eq $#
        then
                INPUT_LOG_FILE="/tmp/Server.log"
        else
                INPUT_LOG_FILE="$@"
        fi
        if test -f "$INPUT_LOG_FILE"
        then
                OUTPUT_DIR=''`/bin/dirname $INPUT_LOG_FILE`''
		if test "." = "$OUTPUT_DIR"
		then
			OUTPUT_DIR="$PWD"
		fi
                while [ 1 ]
                do
                        OUTPUT_LOG_FILE="$OUTPUT_DIR/Server_"''`/bin/date "+%d_%b_%Y_%S_%N"`''".log"
                        if test -f "$OUTPUT_LOG_FILE"
                        then
                                /bin/sleep 1;
                        else
                                break
                        fi
                done
		echo "Current output:"
		echo "------------------------:"
                /bin/awk '{
                        if ( 1 == NR)
                        {
                                printf( "#Date: %s %s\n", $2, $3);
                        }
                        for ( Loc=2; Loc<NF; Loc++)
                        {
                                printf( "%s ", $Loc);
                        }
                        printf( "%s\n", $Loc);
                }' "$INPUT_LOG_FILE" 2>&1 | /usr/bin/tee -a "$OUTPUT_LOG_FILE"
		echo "------------------------:"
                if test -f "$OUTPUT_LOG_FILE"
                then
                        echo "Input  log file: $INPUT_LOG_FILE"
                        echo "Output log file: $OUTPUT_LOG_FILE"
                fi
        else
                echo "$INPUT_LOG_FILE No such file"
                Ret=1
                return $Ret
        fi
}
Disp_Log_Msg $@

Open in new window

Related sample output here:

$ ./29012555.sh
Current output:
------------------------:
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
------------------------:
Input  log file: /tmp/Server.log
Output log file: /tmp/Server_24_May_2017_49_677778500.log
$ /bin/cat /tmp/Server_24_May_2017_49_677778500.log
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
$ ./29012555.sh  /root/Server.log
Current output:
------------------------:
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
------------------------:
Input  log file: /root/Server.log
Output log file: /root/Server_24_May_2017_19_936798800.log
$ /bin/cat /root/Server_24_May_2017_19_936798800.log
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
$

Open in new window

Test the updated code and let us know if getting expected output for the same.

It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

Bash

From novice to tech pro — start learning today.

Programming

By:

Chris Penick

Perl Fundamentals

i

Members can enroll in this course at no extra cost.

51 lessons
Microsoft Applications

By:

Patrick Loner

Certification
Certification: MOS Microsoft Office Specialist - Excel 2016 Part …

i

Members can enroll in this course at no extra cost.

129 lessons
System Programming

By:

Don Kiely

SQL Server Core 2016

i

Members can enroll in this course at no extra cost.

81 lessons
Programming

By:

Don Kiely

Programming C# 6: Intermediate

i

Members can enroll in this course at no extra cost.

90 lessons
Microsoft Applications

By:

Sandra Batakis

Certification
Certification: MOS Microsoft Office Specialist - Excel 2007 Adva…

i

Members can enroll in this course at no extra cost.

188 lessons
R

By:

Thomas LeBlanc

Exploring SQL Server 2016: Intermediate

i

Members can enroll in this course at no extra cost.

40 lessons
Cloud Computing

By:

Ken Mayer

Certification
Monitoring and Operating a Private Cloud with System Center 2012 R2

i

Members can enroll in this course at no extra cost.

273 lessons
IT Administration

By:

Tracy Preston

Certification
Certification: ISCAP Information Systems Certification and Accreditati…

i

Members can enroll in this course at no extra cost.

16 lessons
Programming

By:

Peter Thorsteinson

C++ 11: Fundamentals

i

Members can enroll in this course at no extra cost.

29 lessons
Disaster Recovery

By:

Rodney Barnhardt

Migrating and Managing Backup Exec 2014

i

Members can enroll in this course at no extra cost.

3 lessons