Avatar of Julio Jose
Julio Jose
Flag for Malaysia asked on

bash remove string for logs

Hi, I have the log file, each of the line first few string "<space>10.99.9.99<space>" need to remove and the IP address could be different for each log files, I need the script to count the character to remove for each line

<space>10.99.9.99<space>2017-03-27 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Open in new window


After process remove the string the script need to append date time from first line of the log to the top the log file

example :

#Date: 2017-03-27 23:44:49
2017-03-27 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Open in new window

* BashScripting LanguagesShell Scripting

Avatar of undefined
Last Comment
MURUGESAN N

8/22/2022 - Mon
johnsone

If the format will always be as you show, then this should do it:
#!/usr/bin/bash

log_file="/tmp/log.txt"

dt=`head -1 ${log_file} | cut -f3-4 -d' '`

echo "#Date: ${dt}" > /tmp/log.txt.$$

cut -f3- -d' ' ${log_file} >> ${log_file}.$$

#
# Uncomment next line to replace original file with modified file
#
# mv ${log_file}.$$ ${log_file}

Open in new window

Assumes spaces separating the IP and the date/time will always be 2 fields.

Obviously, you need to modify the variable to set the name of the file.
Julio Jose

ASKER
Hi John,

If the IP address change like to different length like 192.168.128.193 what do I need to modify? The Date time format is not fix too.

I have tested the script, the string is removed but the date is not append at the top
SOLUTION
johnsone

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
MURUGESAN N

Hi Julio Jose,

You can do that using following awk:
/bin/awk '{
        if ( 1 == NR)
        {
                printf( "#Date: %s %s\n", $2, $3);
        }
        for ( Loc=2; Loc<=NF; Loc++)
        {
                printf( "%s ", $Loc);
        }
        printf( "\n");
}' /tmp/log.txt

Open in new window


Also above code take less time(most of the times) which has been tested using:
START_NANO_SECONDS=''`/bin/date "+%N"`''
#REQUIRED_COMMANDS.....
END_NANO_SECONDS=''`/bin/date "+%N"`''
echo "TIME TAKEN BY CURRENT  SCRIPT: "''`/usr/bin/expr $END_NANO_SECONDS - $START_NANO_SECONDS`''

Open in new window


This could have been done using /bin/sed alone, reason for adding /bin/awk due to your requirement:
>> to append date time from first line
This is the best money I have ever spent. I cannot not tell you how many times these folks have saved my bacon. I learn so much from the contributors.
rwheeler23
MURUGESAN N

2.
Same thing using /bin/sed where total number of /bin/sed being 3 times.
/bin/sed -n 1p  /tmp/log.txt | /bin/sed "s/ *[0-9]*\.[0-9]*\.[0-9]*\.[0-9]* *\([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]\).*/#Date: \1/;"
/bin/sed "s/ *[0-9]*\.[0-9]*\.[0-9]*\.[0-9]* *//;" /tmp/log.txt

Open in new window

3. Using /bin/sed (1 time)
and /bin/awk(1 time)
/bin/awk '{
        if ( 1 == NR)
        {
                printf( "#Date: %s %s\n", $2, $3);
        }
}' /tmp/log.txt
/bin/sed "s/ *[0-9]*\.[0-9]*\.[0-9]*\.[0-9]* *//;" /tmp/log.txt

Open in new window


Also always use full path while using any Linux oriented commands.
Example:
unalias ls >/dev/null 2>&1
unset -f ls
ls ()
{
      echo Always use full path for ls command
      echo Julio Jose, Just FYI.
      Ret=12
      return $Ret
}
ls -latr /root
echo $? will be twelve
#Un defining that ls function
unset -f ls

Open in new window


Mentioned Linux oriented commands
>> Applicable to all operating systems:
SunOS
AIX
HP-UX
Linux ( Redhat Enterprise Linux/CentOS/OpenSUSE/UNIX/....)
Windows_Cygwin_OR_mingw
MURUGESAN N

@Julio Jose,
Mentioning the related script for the same.
#!/bin/ksh
#+-----------+----------------+------------------------------------------------------------------+
#|dd/mmm/yyyy|Name            |Svn History                                                       |
#+-----------+----------------+------------------------------------------------------------------+
#|31/Mar/2017|Murugesan Dinesh|Remove string from server log file.                               |
#+-----------+----------------+------------------------------------------------------------------+
#Function Disp_Log_Msg
#        If no parameter passed
#                Script reading default Log file /tmp/Server.log
#        else
#                Script taking all parameter "$@" as Log file.
#        Disp_Log_Msg function display the log message excluding the first column
#        Removing first column due to presence of IP Address.
unset -f Disp_Log_Msg
Disp_Log_Msg ()
{
        if test 0 -eq $#
        then
                INPUT_LOG_FILE="/tmp/Server.log"
        else
                INPUT_LOG_FILE="$@"
        fi
        if test -f "$INPUT_LOG_FILE"
        then
                OUTPUT_DIR=''`/bin/dirname $INPUT_LOG_FILE`''
                while [ 1 ]
                do
                        OUTPUT_LOG_FILE="$OUTPUT_DIR/Server_"''`/bin/date "+%d_%b_%Y_%S_%N"`''".log"
                        if test -f "$OUTPUT_LOG_FILE"
                        then
                                /bin/sleep 1;
                        else
                                break
                        fi
                done
               # INPUT_LOG_FILE_NUM_OF_LN
                export INPUT_LOG_FILE_NUM_OF_LN=''`/bin/awk 'END { print NR}' "$INPUT_LOG_FILE"`''
                /bin/awk '{
                        INPUT_LOG_FILE_NUM_OF_LN=ENVIRON["INPUT_LOG_FILE_NUM_OF_LN"];
                }
                {
                        if ( 1 == NR)
                        {
                                printf( "#Date: %s %s\n", $2, $3);
                        }
                        for ( Loc=2; Loc<NF; Loc++)
                        {
                                printf( "%s ", $Loc);
                        }
                        if ( NR != INPUT_LOG_FILE_NUM_OF_LN)
                        {
                                printf( "%s\n", $Loc);
                        }
                        else
                        {
                                printf( "%s", $Loc);
                        }
                }' "$INPUT_LOG_FILE" > "$OUTPUT_LOG_FILE"
                if test -f "$OUTPUT_LOG_FILE"
                then
                        echo "Input  log file: $INPUT_LOG_FILE"
                        echo "Output log file: $OUTPUT_LOG_FILE"
                fi
        else
                echo "$INPUT_LOG_FILE No such file"
                Ret=1
                return $Ret
        fi
}
Disp_Log_Msg $@

Open in new window

Julio Jose

ASKER
Hi Murugesan,

I want to test the script

I have define INPUT_LOG_FILE full path to the login

then set  OUTPUT_DIR=''`$INPUT_LOG_FILE.$$`''

./log2.sh[25]: <the log path>.4158: not found [No such file or directory]
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
MURUGESAN N

Use the following:
.......
then
        set  OUTPUT_DIR="$INPUT_LOG_FILE.$$"
.....

Open in new window

MURUGESAN N

Reason for using different output log file:
Possibility of $$ can become same and might overwrite the old output log file.
Hence handled that error inside that script itself.
However it can be handled manually also(for you to move the output file where ever you wish to backup).
MURUGESAN N

Also instead of posting only error,
post the script you have used, not sure how you handled that inside the script.
#!/bin/bash
INPUT_LOG_FILE="/tmp/Server.log"
export  OUTPUT_DIR="$INPUT_LOG_FILE.$$"
echo OUTPUT_DIR $OUTPUT_DIR
echo INPUT_LOG_FILE $INPUT_LOG_FILE
if test ! -d "$OUTPUT_DIR"
then
        echo "Need to create the output directory /bin/mkdir $OUTPUT_DIR"
        echo "I feel this is not the right way."
fi

Open in new window

sample output here:
MurugesanDinesh@TEL /home/murugesandins [ 0 ]
$ ./Julio_jose.sh
OUTPUT_DIR /tmp/Server.log.7728
INPUT_LOG_FILE /tmp/Server.log
Need to create the output directory /bin/mkdir /tmp/Server.log.7728
I feel this is not the right way.
MurugesanDinesh@TEL /home/murugesandins [ 0 ]
$

Open in new window

All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
ASKER CERTIFIED SOLUTION
Log in to continue reading
Log In
Sign up - Free for 7 days
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
MURUGESAN N

Julio Jose

ASKER
Hi, I been away for sometime, I added the "set"

I got the log generate under root instead of the same directory, what is the problem?

log2.sh: line 25: /xx/xx/xx/xx/xx.log.1431: No such file or directory
Input  log file: /xx/xx/xx/xx/xx.log
Output log file: /Server_05_May_2017_16_063950669.log
MURUGESAN N

I will see your current query tomorrow.
Sending this from home.
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
MURUGESAN N

can you post the code
log2.sh
Julio Jose

ASKER
#!/bin/ksh
#+-----------+----------------+------------------------------------------------------------------+
#|dd/mmm/yyyy|Name            |Svn History                                                       |
#+-----------+----------------+------------------------------------------------------------------+
#|31/Mar/2017|Murugesan Dinesh|Remove string from server log file.                               |
#+-----------+----------------+------------------------------------------------------------------+
#Function Disp_Log_Msg
#        If no parameter passed
#                Script reading default Log file /tmp/Server.log
#        else
#                Script taking all parameter "$@" as Log file.
#        Disp_Log_Msg function display the log message excluding the first column
#        Removing first column due to presence of IP Address.
unset -f Disp_Log_Msg
Disp_Log_Msg ()
{
        if test 0 -eq $#
        then
                INPUT_LOG_FILE="/xx/xx/xx/xx/xx.log"
        else
                INPUT_LOG_FILE="$@"
        fi
        if test -f "$INPUT_LOG_FILE"
        then
                set OUTPUT_DIR=''`$INPUT_LOG_FILE.$$`''
                while [ 1 ]
                do
                        OUTPUT_LOG_FILE="$OUTPUT_DIR/Server_"''`/bin/date "+%d_%b_%Y_%S_%N"`''".log"
                        if test -f "$OUTPUT_LOG_FILE"
                        then
                                /bin/sleep 1;
                        else
                                break
                        fi
                done
               # INPUT_LOG_FILE_NUM_OF_LN
                export INPUT_LOG_FILE_NUM_OF_LN=''`/bin/awk 'END { print NR}' "$INPUT_LOG_FILE"`''
                /bin/awk '{
                        INPUT_LOG_FILE_NUM_OF_LN=ENVIRON["INPUT_LOG_FILE_NUM_OF_LN"];
                }
                {
                        if ( 1 == NR)
                        {
                                printf( "#Date: %s %s\n", $2, $3);
                        }
                        for ( Loc=2; Loc<NF; Loc++)
                        {
                                printf( "%s ", $Loc);
                        }
                        if ( NR != INPUT_LOG_FILE_NUM_OF_LN)
                        {
                                printf( "%s\n", $Loc);
                        }
                        else
                        {
                                printf( "%s", $Loc);
                        }
                }' "$INPUT_LOG_FILE" > "$OUTPUT_LOG_FILE"
                if test -f "$OUTPUT_LOG_FILE"
                then
                        echo "Input  log file: $INPUT_LOG_FILE"
                        echo "Output log file: $OUTPUT_LOG_FILE"
                fi
        else
                echo "$INPUT_LOG_FILE No such file"
                Ret=1
                return $Ret
        fi
}
Disp_Log_Msg $@

Open in new window

MURUGESAN N

Hi Julio Jose,

I copied and test the code you have pasted here:
Sample output at cygwin:
$ /bin/cp -ip /bin/bash /bin/ksh
$ ./log2.sh
/xx/xx/xx/xx/xx.log No such file
$ echo $?
1
$ /bin/date
Tue May 16 10:02:29 IST 2017

Open in new window


a)
Give the exact command you are using to execute log2.sh

b)
I have also verified the type of the file being used. Give the output for following command
$ /usr/bin/file ./log2.sh
./log2.sh: Korn shell script, ASCII text executable

Open in new window


c)
Give the output for following commands:
/usr/bin/od -bc  ./log2.sh | /bin/egrep "\\\r"
echo $?

Open in new window

I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
Julio Jose

ASKER
This is the output

#sh log2.sh
# /usr/bin/od -bc ./log2.sh | /bin/egrep "\\\r"
# /usr/bin/file ./log2.sh
./log2.sh: Korn shell script text executable
MURUGESAN N

Here goes updated script:
#!/bin/ksh
#+-----------+----------------+---------------------------------------------------------------------+
#|dd/mmm/yyyy|Name            | Svn History                                                         |
#+-----------+----------------+---------------------------------------------------------------------+
#|24/May/2017|Murugesan Dinesh| Changed INPUT_LOG_FILE to have /tmp/Server.log                      |
#|                              Changed OUTPUT_DIR to point to source directory (/tmp) of input file|
#|                              Also displaying output in current terminal.                         |
#|                              Removed following variable in current update.                       |
#|                              INPUT_LOG_FILE_NUM_OF_LN                                            |
#|31/Mar/2017|Murugesan Dinesh| Remove string from server log file.                                 |
#+-----------+----------------+---------------------------------------------------------------------+
#Function Disp_Log_Msg
#        If no parameter passed
#                Script reading default Log file /tmp/Server.log
#        else
#                Script taking all parameter "$@" as Log file.
#        Disp_Log_Msg function display the log message excluding the first column
#        Removing first column due to presence of IP Address.
unset -f Disp_Log_Msg
Disp_Log_Msg ()
{
        if test 0 -eq $#
        then
                INPUT_LOG_FILE="/tmp/Server.log"
        else
                INPUT_LOG_FILE="$@"
        fi
        if test -f "$INPUT_LOG_FILE"
        then
                OUTPUT_DIR=''`/bin/dirname $INPUT_LOG_FILE`''
		if test "." = "$OUTPUT_DIR"
		then
			OUTPUT_DIR="$PWD"
		fi
                while [ 1 ]
                do
                        OUTPUT_LOG_FILE="$OUTPUT_DIR/Server_"''`/bin/date "+%d_%b_%Y_%S_%N"`''".log"
                        if test -f "$OUTPUT_LOG_FILE"
                        then
                                /bin/sleep 1;
                        else
                                break
                        fi
                done
		echo "Current output:"
		echo "------------------------:"
                /bin/awk '{
                        if ( 1 == NR)
                        {
                                printf( "#Date: %s %s\n", $2, $3);
                        }
                        for ( Loc=2; Loc<NF; Loc++)
                        {
                                printf( "%s ", $Loc);
                        }
                        printf( "%s\n", $Loc);
                }' "$INPUT_LOG_FILE" 2>&1 | /usr/bin/tee -a "$OUTPUT_LOG_FILE"
		echo "------------------------:"
                if test -f "$OUTPUT_LOG_FILE"
                then
                        echo "Input  log file: $INPUT_LOG_FILE"
                        echo "Output log file: $OUTPUT_LOG_FILE"
                fi
        else
                echo "$INPUT_LOG_FILE No such file"
                Ret=1
                return $Ret
        fi
}
Disp_Log_Msg $@

Open in new window

Related sample output here:
$ ./29012555.sh
Current output:
------------------------:
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
------------------------:
Input  log file: /tmp/Server.log
Output log file: /tmp/Server_24_May_2017_49_677778500.log
$ /bin/cat /tmp/Server_24_May_2017_49_677778500.log
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
$ ./29012555.sh  /root/Server.log
Current output:
------------------------:
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
------------------------:
Input  log file: /root/Server.log
Output log file: /root/Server_24_May_2017_19_936798800.log
$ /bin/cat /root/Server_24_May_2017_19_936798800.log
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
$

Open in new window

Test the updated code and let us know if getting expected output for the same.