Solved

Parse .log file for SQL import

Posted on 2016-07-30
13
63 Views
Last Modified: 2016-08-04
Trying to parse/prepare below format so that I can pull it into SQL, I had this topic open under Crystal reports and SQL but I'm still having challenges just importing it into 5 columns. (Text in red font)

C--Users-John-S-Pretorius-Desktop-20.jpg
0
Comment
Question by:John-S Pretorius
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 3
  • 3
  • +1
13 Comments
 
LVL 12

Expert Comment

by:tel2
ID: 41736213
Hi John,

Is there just going to be 1 record?  If not, pls give an example with at least 2 records.

What output from the script would you like, exactly?  Give us an example which is based on the input data example you've supplied.

Will you accept Perl solutions, or Bash, PowerShell, or what?
0
 

Author Comment

by:John-S Pretorius
ID: 41736383
Thank you for your feedback, see attached. From my research and other comments it sounds that best approach would be to get it into a readable format for SQL, the first step would be to parse the information from the attached log file into a tab delimited file to look like :- (PowerShell sounds like a great approach but it truly does not matter)

07/29/2016 00:02:03    000022673665    012    550    4095.00

C--Users-John-S-Pretorius-Desktop-Ba.log

My approach/wish list was to have a SSIS package that drops each file directly into sql from the log file but I'm stuck as I'm not familiar with the process. I have to implement the solution/method on the machine/network you use to get the file processed on a daily basis so be mindful of that. Will you be able to handle a directory or should I find a way to copy the single file into a separate folder everyday and delete after your process is complete ?
0
 
LVL 5

Assisted Solution

by:Abhimanyu Suri
Abhimanyu Suri earned 100 total points
ID: 41736475
Here is the Script to convert log files content into desired output.
Assumption : Logfile will have same contents as attached

#######################
# USAGE = ./get_sqldata.sh logfile_name
#!/usr/bin/sh
get_sqldata()
{
cat ${1} |awk 'NF'|while read column1 remainder
do
case $column1 in
[0-9]*)
var_date=`echo ${column1} ${remainder}|cut -d ":" -f1,2,3`
;;
MerchantID)
var_Merchant=`echo ${remainder}|cut -d "-" -f2`
;;
TerminalID)
var_TerminalID=`echo ${remainder}|cut -d "-" -f2`
;;
BatchItemCount)
var_BatchItemCount=`echo ${remainder}|cut -d "-" -f2`
;;
NetBatchTotal)
var_NetBatchTotal=`echo ${remainder}|cut -d "-" -f2`
echo ${var_date} $var_Merchant $var_TerminalID $var_BatchItemCount $var_NetBatchTotal
;;
esac
done
}

get_sqldata $1
############################

Now assuming you have multiple logfiles and you want to parse all of them
To achieve that create a list file with name of all logfiles as mentioned below

:/bash>cat list_file
abc.txt
abc.txt1

Once created execute below

cat list_file |while read line
do
. /bash/get_sqldata.sh $line
done

O/P
07/29/2016 00:02:03 000022673665 012 550 4095.00
07/29/2016 00:02:03 000022673665 012 550 4095.00
0
Salesforce Has Never Been Easier

Improve and reinforce salesforce training & adoption using WalkMe's digital adoption platform. Start saving on costly employee training by creating fast intuitive Walk-Thrus for Salesforce. Claim your Free Account Now

 

Author Comment

by:John-S Pretorius
ID: 41736487
Thank you - WOW!!

Can I assume I would need an application to run this or will this be a .bat file ?
0
 
LVL 5

Expert Comment

by:Abhimanyu Suri
ID: 41736504
.bat file will work fine. All you need is shell :)
0
 

Author Comment

by:John-S Pretorius
ID: 41736507
I opened Windows PowerShell ISE and also get errors when I run see below :

C--Users-John-S-Pretorius-Desktop-20.jpg
0
 
LVL 5

Expert Comment

by:Abhimanyu Suri
ID: 41736523
The script provided was for bash/korn shell ( Unix not windows )
0
 

Author Comment

by:John-S Pretorius
ID: 41736526
I need windows please.
0
 
LVL 12

Expert Comment

by:tel2
ID: 41736829
Hi John,

When I asked this question:
 "Will you accept Perl solutions, or Bash, PowerShell, or what?"
and you responded with this answer:
 "PowerShell sounds like a great approach but it truly does not matter"
then some experts could assume that it truly doesn't matter whether they give you a bash solution (that's a UNIX/Linux Borne Again SHell script, which is very similar to a Korn shell script).  I expect that is what Abhimanyu has concluded, and has wasted his time based on that misinformation.

The fact that one of the topic areas you selected was "Shell Scripting", is also not helpful, because if this is Windows then the only type of shell is Powershell, so you should select the "Powershell" topic area so that you target experts in that, rather than general shell scripting experts, most of whom are probably UNIX/Linux shell script writers.
1
 

Author Comment

by:John-S Pretorius
ID: 41736861
My Apologies.
0
 
LVL 12

Expert Comment

by:tel2
ID: 41736874
I don't know Powershell, John, but Perl can run on Windows, and I do know Perl.  Do you have Perl installed on the machine that will be running this?  Please be completely sure of your answer before responding, otherwise a lot more time could be wasted going down the wrong path.
0
 
LVL 54

Accepted Solution

by:
Bill Prew earned 400 total points
ID: 41736886
Here is a small BAT script that should get you there, give it a try.

@echo off
setlocal EnableDelayedExpansion

set FileIn=in.log
set FileOut=out.txt

if exist "%FileOut%" del "%FileOut%"

for /f "usebackq tokens=1-2 delims=- " %%A in ("%FileIn%") do (
  set Token1=%%A
  if "!Token1:~2,1!" == "/" set DateTime=%%A %%B
  if /i "%%A" == "MerchantID" set MerchantID=%%B
  if /i "%%A" == "TerminalID" set TerminalID=%%B
  if /i "%%A" == "BatchItemCount" set BatchItemCount=%%B
  if /i "%%A" == "NetBatchTotal" (
    set NetBatchTotal=%%B
    echo !DateTime!,!MerchantID!,!TerminalID!,!BatchItemCount!,!NetBatchTotal!>>"%FileOut%"
  )
)

Open in new window

~bp
1
 

Author Closing Comment

by:John-S Pretorius
ID: 41736897
Thank you Bill, that worked perfect.

Again I apologize for giving wrong information.
0

Featured Post

MS Dynamics Made Instantly Simpler

Make Your Microsoft Dynamics Investment Count  & Drastically Decrease Training Time by Providing Intuitive Step-By-Step WalkThru Tutorials.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Slowly Changing Dimension Transformation component in data task flow is very useful for us to manage and control how data changes in SSIS.
Load balancing is the method of dividing the total amount of work performed by one computer between two or more computers. Its aim is to get more work done in the same amount of time, ensuring that all the users get served faster.
Using examples as well as descriptions, and references to Books Online, show the different Recovery Models available in SQL Server and explain, as well as show how full, differential and transaction log backups are performed
Viewers will learn how to use the SELECT statement in SQL and will be exposed to the many uses the SELECT statement has.

732 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question