?
Solved

Log File Formatting

Posted on 2009-12-27
15
Medium Priority
?
348 Views
Last Modified: 2013-11-10
Hello Experts,

I have several log files (text files) that I would like to reformat to look readable. I would like to know if there is a automatic way to have my text file do a down arrow 3 times and a backspace.

So the function I am looking for is:

Down, Down, Down, Backspace
Down, Down, Down, Backspace

and repeat until the end of the text file.

Can anyone help me reformat these log files? I am trying to do it manually and it is taking a long time to finish even one.

Thanks!

Here is an excerpt of several lines:


HK Retiremen:
ALARM:            Normal Zone    EZU1 Zone6 ZONE 6, W-WING 3RD FLR. S/D, P/S, H/
D    11:27 12/20/09

HK Retiremen:
ALARM:            Normal Zone    EZU2 Zone5 ZONE 13, N-WING 3RD FLR. S/D, P/S, H
/D   11:27 12/20/09

HK Retiremen:
RESET:                                      via SCU
     11:27 12/20/09

I would like it to show in this readable format:

HK Retiremen:
ALARM:            Normal Zone    EZU1 Zone6 ZONE 6, W-WING 3RD FLR. S/D, P/S, H/D    11:27 12/20/09

HK Retiremen:
ALARM:            Normal Zone    EZU2 Zone5 ZONE 13, N-WING 3RD FLR. S/D, P/S, H/D   11:27 12/20/09

HK Retiremen:
RESET:                                      via SCU     11:27 12/20/09

Open in new window

0
Comment
Question by:katredrum
15 Comments
 
LVL 3

Expert Comment

by:roeib
ID: 26129875
well you dnot have somthing like this just out of nothing, you can write a small application to read the file and work on it

use a csv reader library like this one:

http://www.codeproject.com/KB/database/CsvReader.aspx
split the columns by Tab Delimiter and rebuild it as you need.

or you can just read the file to memory and rewrite it row by row with your logic of disply.
0
 
LVL 1

Author Comment

by:katredrum
ID: 26129886
woah, Iroeib, I do not know how to code or program. I was hoping there was a way to do this in a simple way.

Your second suggestion seems easier than the first one, can you elaborate? The log file is coming out of a fire alarm panel so I probably would not be able to (even if I knew how) program the output.
0
 
LVL 31

Expert Comment

by:moorhouselondon
ID: 26129980
How about a program such as Monarch

http://www.datawatch.com/_products/monarch_overview.php
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
LVL 3

Expert Comment

by:eitama
ID: 26130554
I would not use keystrokes to re-format a log file.
Better use regular expression.

In the above file you posted, how would you like it to look after the reformatting?
0
 
LVL 3

Expert Comment

by:eitama
ID: 26130777
Hi,

Being a TCL Freak I am, I like solving these issues using TCL Scripts.

I wrote a script for you.

1. Copy the code below, and save it into a file called change.tcl (a text file with extension tcl)
2. Install activestate tcl 8.5 or 8.4 from www.activestate.com (free), this is the TCL interpreter that will execute the script.
3. Copy the change.tcl file to the same directory where you log files are.
4. Open a CMD, navigate to that place, and use the script :
    C:\>tclsh85 change.tcl myLogFile.log
5. The converted log file will be saved in new.log
6. If you want it to overwrite, edit the change.tcl file, and choose line 38 instead of 37.

Good luck.
Eitam.
# Get filename to change from argument
if {[lindex $argv 0] eq ""} {
	puts "Usage:\n  tclsh change.tcl <filename>\n\nExample:\n  tclsh change.tcl myLog.log"
	exit
}
set filename [lindex $argv 0] 

# Open for reading
set f [open $filename r]

# Read the data from the file
set data [read $f]

# Close the file
close $f

# Set counter for number of replacments.
set i 0

# Replace as long as there are matches:
while {true} {
	# If there is a match of a broken log line
	if {[regexp -- {HK Retiremen:\n([^\n]+)\n([^\n]+)} $data temp 1st 2nd]} {
		# Increase the countera
		incr i
		# Replace with corrent syntax
		regsub -- {HK\sRetiremen:\n[^\n]+\n[^\n]+} $data "HK Retiremen:\n$1st$2nd" data
	} else {
		# Done.
		puts "No more matches, replaced $i times"
		# Leave the loop
		break
	}
}

# Write the new file into a temp file, if you want to overwrite thee old file, comment line 37, and uncomment line 38
set f [open new.log w+]
#set f [open $filename w+]
puts -nonewline $f $data
close $f

Open in new window

0
 
LVL 1

Author Comment

by:katredrum
ID: 26132795
I will try this and let you know eitama.
0
 
LVL 1

Author Comment

by:katredrum
ID: 26133697
eitama,

When I run the script, it gives me this error.

Error.jpg
0
 
LVL 3

Expert Comment

by:eitama
ID: 26134084
Hmmm,

That is odd, I have ran TCL on 8 different OSs, all versions of windows and some linux distros,
I have never had such an error.

What os do you have? looks like XP, but just to be sure.

Also I know it's a hastle but remove TCL 8.5 and install 8.4
If that doesn't work, so I wrote a script in vain ):

By the way, the script needs an input file, exactly like you wrote.
If it will work good in tcl 8.4, i can change it to work on all the log files in the current dir.
0
 
LVL 3

Expert Comment

by:eitama
ID: 26134087
Wait!

I just thought, how big are the files you are dealing with???
0
 
LVL 1

Author Comment

by:katredrum
ID: 26134304
I am running Windows XP 32-bit. The log files are 213KB, but has a lot of entries.  I will try to use a sample set with a quarter of the log and will let you know. Thanks for doing this. I really appreciate it.
0
 
LVL 3

Expert Comment

by:eitama
ID: 26134349
I found the problem, I am fixing the script.

I had a command, reading all the file at once, and then running a regular expression on it,
No way it can work like that (:

Anyway i'm fixing it, i'll upload a fix soon.
0
 
LVL 3

Accepted Solution

by:
eitama earned 2000 total points
ID: 26134647
Ok,

I completely changed the code, it works differently now.

Usage: The same as before, you can also run it now with -overwrite
to make it overwrite the old file when you are confident that it will work well.

C:\>tclsh85 change2.tcl
Usage:
  tclsh change.tcl <filename> ?-overwrite?

Example:
  tclsh change.tcl myLog.log
  tclsh change.tcl myLog.log -overwrite


What it does: Take a line "HK Retiremen:", and the 2 lines after and writes it to
a new file with the 2 lines combined.

Limitations:
  - "HK Retiremen:" Must appear, or the script will fail,
     if you are now parsing a different log file, edit the script and change it.
  - It will always take 2 lines, according to the example you posted at start and combine them.
     if you need 3 lines or more, you need to change the script.

Should work on any size of file now.
# Get filename to change from argument
if {[lindex $argv 0] eq ""} {
        puts "Usage:\n  tclsh change.tcl <filename> ?-overwrite?\n\nExample:\n  tclsh change.tcl myLog.log\n  tclsh change.tcl myLog.log -overwrite"
        exit
}

set overwrite 0
if {[lindex $argv 1] eq "-overwrite"} {
        set overwrite 1
}

set filename [lindex $argv 0] 

# Open for reading
set f [open $filename r]
fconfigure $f -buffering line
set outf [open $filename.out w+]

# Set counter for number of replacments.
set replaced 0
set done 0
set 1st ""
set 2nd ""
set gotHeader 0

while {!$done} {
	set failSafe 1000
	while {$failSafe > 0} {
		incr failSafe -1
		set temp [gets $f]
		if {$temp eq "HK Retiremen:"} {
			set 1st $temp
			if {[eof $f]} {
				set done 1
				break
			} else {
				set temp [gets $f]
				set 2nd $temp
				if {[eof $f]} {
					set done 1
					break
				} else {
					set temp [gets $f]
					append 2nd $temp
					puts $outf "$1st\n$2nd\n"
					flush $outf
					incr replaced
					break 
				}
			}
		} 
	}
	if {$failSafe == 0} {break}
}

close $f
close $outf
if {$overwrite} {
	file delete $filename
	file rename $filename.out $filename
	puts "Done. Replaced $replaced times, old file was overwritten."
} else {
	puts "Done. Replaced $replaced times, new file is named $filename.out"
}

Open in new window

Untitled.png
0
 
LVL 1

Author Comment

by:katredrum
ID: 26134851
eitama, you are the best! this is exactly what I needed.

Success.jpg
0
 
LVL 1

Author Closing Comment

by:katredrum
ID: 31670326
You saved the day!!! Thank you thank you thank you!
0
 
LVL 3

Expert Comment

by:eitama
ID: 26136180
(:

Glad I could help.

P.S.
Take your time and view the script, TCL is very easy, you can learn it in just a few weeks, it can serve you so much!
This script does the following :
- Opens both old file and new file
- Reads 3 the old file, line by line, till it finds "HK Retiremen:"
- It then reads 2 more lines
- It always check to see if it has reached the end of the file, cause then it needs to stop.
- When it has gotten 3 lines, the 1st being "HK Retiremen:" it writes to the new file, 2 lines.
   the 2nd is made from 2 lines which were read from the old file.
- And so on, till the end of the old file.
- When it's done, it checks if you put the -overwrite parameter, if you did, it will delete the old, and rename the new.

Cheers,
Take Care.
Eitam.
0

Featured Post

Vote for the Most Valuable Expert

It’s time to recognize experts that go above and beyond with helpful solutions and engagement on site. Choose from the top experts in the Hall of Fame or on the right rail of your favorite topic page. Look for the blue “Nominate” button on their profile to vote.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this post we will learn different types of Android Layout and some basics of an Android App.
Gain an elementary understanding of Blockchain technology.
Simple Linear Regression
Loops Section Overview

750 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question