Solved

Automatically fetch .log file to push text into a SQL table

Posted on 2016-07-26
8
35 Views
Last Modified: 2016-07-29
I have a batch file that generates every morning :- BatchClose_07252016.log and looking for a way to automatically pull the information into key fields. I would need to create the following fields/columns in SQL first and then append values on a daily basis (See image and numbering below ) :-

(1) DateTime, first line '07/25/2016 13:08:22'
(2) MerchantID, portion after the - '720000309317'
(3) BatchNo, portion after the - '207001'
(4) BatchItemCount, portion after the - '000005'
(5) NetBatchTotal, portion after the - '50.00'

C--Users-John-S-Pretorius-Desktop-ex.jpgC--Users-John-S-Pretorius-Desktop-Ex.log
0
Comment
Question by:John-S Pretorius
  • 4
  • 3
8 Comments
 
LVL 12

Expert Comment

by:funwithdotnet
Comment Utility
The first step is to get the log data into a format SQL Server can consume. Can the format of the log be changed? If so, an XML format, among others, would allow a pure SQL Server solution using a scheduled task or a trigger.

If the format is fixed, the data needs to be transformed into something SQL Server can use. There has to be infinite ways to manipulate the log file into a database format. The trick is what is convenient and economical.  if the log was fixed-format, I would write a parser in .NET that inserted the log entries directly into the database and run that as a scheduled task or manually.
0
 

Author Comment

by:John-S Pretorius
Comment Utility
Thank you, this is a set .log file. Looking online I found some instructions that sounded straight forward to script sql to pull specific field values after the tables are created. I'm not versed in SQL so this is my challenge.

I'm hoping someone can assist me to extract the values as required from the file structure as mentioned above.
0
 
LVL 26

Expert Comment

by:Chris Luttrell
Comment Utility
What "straight forward" script did you find?
The first question is how is it getting into the database?  Are your scripts parsing the data first and presenting the clean data to the database or do they operate on strings once they are in the database?
What is going to trigger the file getting loaded each morning, you may want to look at a scheduled job that uses SSIS to pull from a directory, or are you going to run some script each day?
0
 

Author Comment

by:John-S Pretorius
Comment Utility
Straight forward is overstating it probably, one of the solutions that seems what I was looking for was a parser.  I also saw someone write scripts to pull the data from the specific placements in the log file and append them to the already created tables. At this time the first challenge would be to just this. I will move on thereafter. I will use Crystal reports to create the reports needed, I already found a way to pull the same log file data in into Crystal using ODBC Microsoft text file but this is not best approach in the long term which is why i'm looking towards SQL.

https://ask.sqlservercentral.com/questions/112479/how-do-import-data-into-sql-server-from-log-file.html
0
Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

 
LVL 12

Expert Comment

by:funwithdotnet
Comment Utility
I remember using Log Parser. Wow, it's a very old tool (nothing wrong with that). I don't think Log Parser is going to help much with that log in its current format.

The way I see it, you have two tasks to accomplish.

The first task is to parse the original log into a format that looks something like tab-delimited:
07/25/2016 13:08:22      720000309317      207001      000005      50.00

or markup:
<record>
<date>07/25/2016 13:08:22</date>

<MerchantID>720000309317</MerchantID>
<BatchNo>207001</BatchNo>
<BatchItemCount>000005</BatchItemCount>
<NetBatchTotal>50.00</NetBatchTotal>
</record>

The second task is to import the transformed data into SQL Server. Just like the first task, there are many ways to import the data into SQL Server. Don't worry about the SQL importing process just yet. Once task #1 is accomplished, task #2 will be easy.
 
I think you'll find that Log Parser will have issues similar to SQL Server reading the log. So if it's nice for Log Parser, SQL Server can handle it alone.

Of course, the trick is to find a solution that's easy and painless for you.  Still thinking about it.
0
 
LVL 12

Accepted Solution

by:
funwithdotnet earned 500 total points
Comment Utility
I found a page that describes how to do exactly what you want in SQL Server.
1
 

Author Comment

by:John-S Pretorius
Comment Utility
Thank you, I agree with this.
0
 

Author Closing Comment

by:John-S Pretorius
Comment Utility
Great, this is what I was hoping to see.
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

I wrote this interesting script that really help me find jobs or procedures when working in a huge environment. I could I have written it as a Procedure but then I would have to have it on each machine or have a link to a server-related search that …
Let's review the features of new SQL Server 2012 (Denali CTP3). It listed as below: PERCENT_RANK(): PERCENT_RANK() function will returns the percentage value of rank of the values among its group. PERCENT_RANK() function value always in be…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Via a live example, show how to backup a database, simulate a failure backup the tail of the database transaction log and perform the restore.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

7 Experts available now in Live!

Get 1:1 Help Now