We help IT Professionals succeed at work.

bash remove string for logs

278 Views
Last Modified: 2017-05-23
Hi, I have the log file, each of the line first few string "<space>10.99.9.99<space>" need to remove and the IP address could be different for each log files, I need the script to count the character to remove for each line

<space>10.99.9.99<space>2017-03-27 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Open in new window


After process remove the string the script need to append date time from first line of the log to the top the log file

example :

#Date: 2017-03-27 23:44:49
2017-03-27 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Open in new window

Comment
Watch Question

johnsoneSenior Oracle DBA
CERTIFIED EXPERT

Commented:
If the format will always be as you show, then this should do it:
#!/usr/bin/bash

log_file="/tmp/log.txt"

dt=`head -1 ${log_file} | cut -f3-4 -d' '`

echo "#Date: ${dt}" > /tmp/log.txt.$$

cut -f3- -d' ' ${log_file} >> ${log_file}.$$

#
# Uncomment next line to replace original file with modified file
#
# mv ${log_file}.$$ ${log_file}

Open in new window

Assumes spaces separating the IP and the date/time will always be 2 fields.

Obviously, you need to modify the variable to set the name of the file.

Author

Commented:
Hi John,

If the IP address change like to different length like 192.168.128.193 what do I need to modify? The Date time format is not fix too.

I have tested the script, the string is removed but the date is not append at the top
johnsoneSenior Oracle DBA
CERTIFIED EXPERT
Commented:
This problem has been solved!
(Unlock this solution with a 7-day Free Trial)
UNLOCK SOLUTION
MURUGESAN NApplication Development and Automation Management
CERTIFIED EXPERT

Commented:
Hi Julio Jose,

You can do that using following awk:
/bin/awk '{
        if ( 1 == NR)
        {
                printf( "#Date: %s %s\n", $2, $3);
        }
        for ( Loc=2; Loc<=NF; Loc++)
        {
                printf( "%s ", $Loc);
        }
        printf( "\n");
}' /tmp/log.txt

Open in new window


Also above code take less time(most of the times) which has been tested using:
START_NANO_SECONDS=''`/bin/date "+%N"`''
#REQUIRED_COMMANDS.....
END_NANO_SECONDS=''`/bin/date "+%N"`''
echo "TIME TAKEN BY CURRENT  SCRIPT: "''`/usr/bin/expr $END_NANO_SECONDS - $START_NANO_SECONDS`''

Open in new window


This could have been done using /bin/sed alone, reason for adding /bin/awk due to your requirement:
>> to append date time from first line
MURUGESAN NApplication Development and Automation Management
CERTIFIED EXPERT

Commented:
2.
Same thing using /bin/sed where total number of /bin/sed being 3 times.
/bin/sed -n 1p  /tmp/log.txt | /bin/sed "s/ *[0-9]*\.[0-9]*\.[0-9]*\.[0-9]* *\([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]\).*/#Date: \1/;"
/bin/sed "s/ *[0-9]*\.[0-9]*\.[0-9]*\.[0-9]* *//;" /tmp/log.txt

Open in new window

3. Using /bin/sed (1 time)
and /bin/awk(1 time)
/bin/awk '{
        if ( 1 == NR)
        {
                printf( "#Date: %s %s\n", $2, $3);
        }
}' /tmp/log.txt
/bin/sed "s/ *[0-9]*\.[0-9]*\.[0-9]*\.[0-9]* *//;" /tmp/log.txt

Open in new window


Also always use full path while using any Linux oriented commands.
Example:
unalias ls >/dev/null 2>&1
unset -f ls
ls ()
{
      echo Always use full path for ls command
      echo Julio Jose, Just FYI.
      Ret=12
      return $Ret
}
ls -latr /root
echo $? will be twelve
#Un defining that ls function
unset -f ls

Open in new window


Mentioned Linux oriented commands
>> Applicable to all operating systems:
SunOS
AIX
HP-UX
Linux ( Redhat Enterprise Linux/CentOS/OpenSUSE/UNIX/....)
Windows_Cygwin_OR_mingw
MURUGESAN NApplication Development and Automation Management
CERTIFIED EXPERT

Commented:
@Julio Jose,
Mentioning the related script for the same.
#!/bin/ksh
#+-----------+----------------+------------------------------------------------------------------+
#|dd/mmm/yyyy|Name            |Svn History                                                       |
#+-----------+----------------+------------------------------------------------------------------+
#|31/Mar/2017|Murugesan Dinesh|Remove string from server log file.                               |
#+-----------+----------------+------------------------------------------------------------------+
#Function Disp_Log_Msg
#        If no parameter passed
#                Script reading default Log file /tmp/Server.log
#        else
#                Script taking all parameter "$@" as Log file.
#        Disp_Log_Msg function display the log message excluding the first column
#        Removing first column due to presence of IP Address.
unset -f Disp_Log_Msg
Disp_Log_Msg ()
{
        if test 0 -eq $#
        then
                INPUT_LOG_FILE="/tmp/Server.log"
        else
                INPUT_LOG_FILE="$@"
        fi
        if test -f "$INPUT_LOG_FILE"
        then
                OUTPUT_DIR=''`/bin/dirname $INPUT_LOG_FILE`''
                while [ 1 ]
                do
                        OUTPUT_LOG_FILE="$OUTPUT_DIR/Server_"''`/bin/date "+%d_%b_%Y_%S_%N"`''".log"
                        if test -f "$OUTPUT_LOG_FILE"
                        then
                                /bin/sleep 1;
                        else
                                break
                        fi
                done
               # INPUT_LOG_FILE_NUM_OF_LN
                export INPUT_LOG_FILE_NUM_OF_LN=''`/bin/awk 'END { print NR}' "$INPUT_LOG_FILE"`''
                /bin/awk '{
                        INPUT_LOG_FILE_NUM_OF_LN=ENVIRON["INPUT_LOG_FILE_NUM_OF_LN"];
                }
                {
                        if ( 1 == NR)
                        {
                                printf( "#Date: %s %s\n", $2, $3);
                        }
                        for ( Loc=2; Loc<NF; Loc++)
                        {
                                printf( "%s ", $Loc);
                        }
                        if ( NR != INPUT_LOG_FILE_NUM_OF_LN)
                        {
                                printf( "%s\n", $Loc);
                        }
                        else
                        {
                                printf( "%s", $Loc);
                        }
                }' "$INPUT_LOG_FILE" > "$OUTPUT_LOG_FILE"
                if test -f "$OUTPUT_LOG_FILE"
                then
                        echo "Input  log file: $INPUT_LOG_FILE"
                        echo "Output log file: $OUTPUT_LOG_FILE"
                fi
        else
                echo "$INPUT_LOG_FILE No such file"
                Ret=1
                return $Ret
        fi
}
Disp_Log_Msg $@

Open in new window

Author

Commented:
Hi Murugesan,

I want to test the script

I have define INPUT_LOG_FILE full path to the login

then set  OUTPUT_DIR=''`$INPUT_LOG_FILE.$$`''

./log2.sh[25]: <the log path>.4158: not found [No such file or directory]
MURUGESAN NApplication Development and Automation Management
CERTIFIED EXPERT

Commented:
Use the following:
.......
then
        set  OUTPUT_DIR="$INPUT_LOG_FILE.$$"
.....

Open in new window

MURUGESAN NApplication Development and Automation Management
CERTIFIED EXPERT

Commented:
Reason for using different output log file:
Possibility of $$ can become same and might overwrite the old output log file.
Hence handled that error inside that script itself.
However it can be handled manually also(for you to move the output file where ever you wish to backup).
MURUGESAN NApplication Development and Automation Management
CERTIFIED EXPERT

Commented:
Also instead of posting only error,
post the script you have used, not sure how you handled that inside the script.
#!/bin/bash
INPUT_LOG_FILE="/tmp/Server.log"
export  OUTPUT_DIR="$INPUT_LOG_FILE.$$"
echo OUTPUT_DIR $OUTPUT_DIR
echo INPUT_LOG_FILE $INPUT_LOG_FILE
if test ! -d "$OUTPUT_DIR"
then
        echo "Need to create the output directory /bin/mkdir $OUTPUT_DIR"
        echo "I feel this is not the right way."
fi

Open in new window

sample output here:
MurugesanDinesh@TEL /home/murugesandins [ 0 ]
$ ./Julio_jose.sh
OUTPUT_DIR /tmp/Server.log.7728
INPUT_LOG_FILE /tmp/Server.log
Need to create the output directory /bin/mkdir /tmp/Server.log.7728
I feel this is not the right way.
MurugesanDinesh@TEL /home/murugesandins [ 0 ]
$

Open in new window

Application Development and Automation Management
CERTIFIED EXPERT
Commented:
This problem has been solved!
(Unlock this solution with a 7-day Free Trial)
UNLOCK SOLUTION
MURUGESAN NApplication Development and Automation Management
CERTIFIED EXPERT

Commented:

Author

Commented:
Hi, I been away for sometime, I added the "set"

I got the log generate under root instead of the same directory, what is the problem?

log2.sh: line 25: /xx/xx/xx/xx/xx.log.1431: No such file or directory
Input  log file: /xx/xx/xx/xx/xx.log
Output log file: /Server_05_May_2017_16_063950669.log
MURUGESAN NApplication Development and Automation Management
CERTIFIED EXPERT

Commented:
I will see your current query tomorrow.
Sending this from home.
MURUGESAN NApplication Development and Automation Management
CERTIFIED EXPERT

Commented:
can you post the code
log2.sh

Author

Commented:
#!/bin/ksh
#+-----------+----------------+------------------------------------------------------------------+
#|dd/mmm/yyyy|Name            |Svn History                                                       |
#+-----------+----------------+------------------------------------------------------------------+
#|31/Mar/2017|Murugesan Dinesh|Remove string from server log file.                               |
#+-----------+----------------+------------------------------------------------------------------+
#Function Disp_Log_Msg
#        If no parameter passed
#                Script reading default Log file /tmp/Server.log
#        else
#                Script taking all parameter "$@" as Log file.
#        Disp_Log_Msg function display the log message excluding the first column
#        Removing first column due to presence of IP Address.
unset -f Disp_Log_Msg
Disp_Log_Msg ()
{
        if test 0 -eq $#
        then
                INPUT_LOG_FILE="/xx/xx/xx/xx/xx.log"
        else
                INPUT_LOG_FILE="$@"
        fi
        if test -f "$INPUT_LOG_FILE"
        then
                set OUTPUT_DIR=''`$INPUT_LOG_FILE.$$`''
                while [ 1 ]
                do
                        OUTPUT_LOG_FILE="$OUTPUT_DIR/Server_"''`/bin/date "+%d_%b_%Y_%S_%N"`''".log"
                        if test -f "$OUTPUT_LOG_FILE"
                        then
                                /bin/sleep 1;
                        else
                                break
                        fi
                done
               # INPUT_LOG_FILE_NUM_OF_LN
                export INPUT_LOG_FILE_NUM_OF_LN=''`/bin/awk 'END { print NR}' "$INPUT_LOG_FILE"`''
                /bin/awk '{
                        INPUT_LOG_FILE_NUM_OF_LN=ENVIRON["INPUT_LOG_FILE_NUM_OF_LN"];
                }
                {
                        if ( 1 == NR)
                        {
                                printf( "#Date: %s %s\n", $2, $3);
                        }
                        for ( Loc=2; Loc<NF; Loc++)
                        {
                                printf( "%s ", $Loc);
                        }
                        if ( NR != INPUT_LOG_FILE_NUM_OF_LN)
                        {
                                printf( "%s\n", $Loc);
                        }
                        else
                        {
                                printf( "%s", $Loc);
                        }
                }' "$INPUT_LOG_FILE" > "$OUTPUT_LOG_FILE"
                if test -f "$OUTPUT_LOG_FILE"
                then
                        echo "Input  log file: $INPUT_LOG_FILE"
                        echo "Output log file: $OUTPUT_LOG_FILE"
                fi
        else
                echo "$INPUT_LOG_FILE No such file"
                Ret=1
                return $Ret
        fi
}
Disp_Log_Msg $@

Open in new window

MURUGESAN NApplication Development and Automation Management
CERTIFIED EXPERT

Commented:
Hi Julio Jose,

I copied and test the code you have pasted here:
Sample output at cygwin:
$ /bin/cp -ip /bin/bash /bin/ksh
$ ./log2.sh
/xx/xx/xx/xx/xx.log No such file
$ echo $?
1
$ /bin/date
Tue May 16 10:02:29 IST 2017

Open in new window


a)
Give the exact command you are using to execute log2.sh

b)
I have also verified the type of the file being used. Give the output for following command
$ /usr/bin/file ./log2.sh
./log2.sh: Korn shell script, ASCII text executable

Open in new window


c)
Give the output for following commands:
/usr/bin/od -bc  ./log2.sh | /bin/egrep "\\\r"
echo $?

Open in new window

Author

Commented:
This is the output

#sh log2.sh
# /usr/bin/od -bc ./log2.sh | /bin/egrep "\\\r"
# /usr/bin/file ./log2.sh
./log2.sh: Korn shell script text executable
MURUGESAN NApplication Development and Automation Management
CERTIFIED EXPERT

Commented:
Here goes updated script:
#!/bin/ksh
#+-----------+----------------+---------------------------------------------------------------------+
#|dd/mmm/yyyy|Name            | Svn History                                                         |
#+-----------+----------------+---------------------------------------------------------------------+
#|24/May/2017|Murugesan Dinesh| Changed INPUT_LOG_FILE to have /tmp/Server.log                      |
#|                              Changed OUTPUT_DIR to point to source directory (/tmp) of input file|
#|                              Also displaying output in current terminal.                         |
#|                              Removed following variable in current update.                       |
#|                              INPUT_LOG_FILE_NUM_OF_LN                                            |
#|31/Mar/2017|Murugesan Dinesh| Remove string from server log file.                                 |
#+-----------+----------------+---------------------------------------------------------------------+
#Function Disp_Log_Msg
#        If no parameter passed
#                Script reading default Log file /tmp/Server.log
#        else
#                Script taking all parameter "$@" as Log file.
#        Disp_Log_Msg function display the log message excluding the first column
#        Removing first column due to presence of IP Address.
unset -f Disp_Log_Msg
Disp_Log_Msg ()
{
        if test 0 -eq $#
        then
                INPUT_LOG_FILE="/tmp/Server.log"
        else
                INPUT_LOG_FILE="$@"
        fi
        if test -f "$INPUT_LOG_FILE"
        then
                OUTPUT_DIR=''`/bin/dirname $INPUT_LOG_FILE`''
		if test "." = "$OUTPUT_DIR"
		then
			OUTPUT_DIR="$PWD"
		fi
                while [ 1 ]
                do
                        OUTPUT_LOG_FILE="$OUTPUT_DIR/Server_"''`/bin/date "+%d_%b_%Y_%S_%N"`''".log"
                        if test -f "$OUTPUT_LOG_FILE"
                        then
                                /bin/sleep 1;
                        else
                                break
                        fi
                done
		echo "Current output:"
		echo "------------------------:"
                /bin/awk '{
                        if ( 1 == NR)
                        {
                                printf( "#Date: %s %s\n", $2, $3);
                        }
                        for ( Loc=2; Loc<NF; Loc++)
                        {
                                printf( "%s ", $Loc);
                        }
                        printf( "%s\n", $Loc);
                }' "$INPUT_LOG_FILE" 2>&1 | /usr/bin/tee -a "$OUTPUT_LOG_FILE"
		echo "------------------------:"
                if test -f "$OUTPUT_LOG_FILE"
                then
                        echo "Input  log file: $INPUT_LOG_FILE"
                        echo "Output log file: $OUTPUT_LOG_FILE"
                fi
        else
                echo "$INPUT_LOG_FILE No such file"
                Ret=1
                return $Ret
        fi
}
Disp_Log_Msg $@

Open in new window

Related sample output here:
$ ./29012555.sh
Current output:
------------------------:
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
------------------------:
Input  log file: /tmp/Server.log
Output log file: /tmp/Server_24_May_2017_49_677778500.log
$ /bin/cat /tmp/Server_24_May_2017_49_677778500.log
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
$ ./29012555.sh  /root/Server.log
Current output:
------------------------:
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
------------------------:
Input  log file: /root/Server.log
Output log file: /root/Server_24_May_2017_19_936798800.log
$ /bin/cat /root/Server_24_May_2017_19_936798800.log
#Date: 23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23:44:49 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
$

Open in new window

Test the updated code and let us know if getting expected output for the same.

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions