Solved

Automatically fetch .log file to push text into a SQL table

Posted on 2016-07-26
8
50 Views
Last Modified: 2016-07-29
I have a batch file that generates every morning :- BatchClose_07252016.log and looking for a way to automatically pull the information into key fields. I would need to create the following fields/columns in SQL first and then append values on a daily basis (See image and numbering below ) :-

(1) DateTime, first line '07/25/2016 13:08:22'
(2) MerchantID, portion after the - '720000309317'
(3) BatchNo, portion after the - '207001'
(4) BatchItemCount, portion after the - '000005'
(5) NetBatchTotal, portion after the - '50.00'

C--Users-John-S-Pretorius-Desktop-ex.jpgC--Users-John-S-Pretorius-Desktop-Ex.log
0
Comment
Question by:John-S Pretorius
  • 4
  • 3
8 Comments
 
LVL 12

Expert Comment

by:funwithdotnet
ID: 41730582
The first step is to get the log data into a format SQL Server can consume. Can the format of the log be changed? If so, an XML format, among others, would allow a pure SQL Server solution using a scheduled task or a trigger.

If the format is fixed, the data needs to be transformed into something SQL Server can use. There has to be infinite ways to manipulate the log file into a database format. The trick is what is convenient and economical.  if the log was fixed-format, I would write a parser in .NET that inserted the log entries directly into the database and run that as a scheduled task or manually.
0
 

Author Comment

by:John-S Pretorius
ID: 41731023
Thank you, this is a set .log file. Looking online I found some instructions that sounded straight forward to script sql to pull specific field values after the tables are created. I'm not versed in SQL so this is my challenge.

I'm hoping someone can assist me to extract the values as required from the file structure as mentioned above.
0
 
LVL 26

Expert Comment

by:Chris Luttrell
ID: 41732323
What "straight forward" script did you find?
The first question is how is it getting into the database?  Are your scripts parsing the data first and presenting the clean data to the database or do they operate on strings once they are in the database?
What is going to trigger the file getting loaded each morning, you may want to look at a scheduled job that uses SSIS to pull from a directory, or are you going to run some script each day?
0
VMware Disaster Recovery and Data Protection

In this expert guide, you’ll learn about the components of a Modern Data Center. You will use cases for the value-added capabilities of Veeam®, including combining backup and replication for VMware disaster recovery and using replication for data center migration.

 

Author Comment

by:John-S Pretorius
ID: 41732864
Straight forward is overstating it probably, one of the solutions that seems what I was looking for was a parser.  I also saw someone write scripts to pull the data from the specific placements in the log file and append them to the already created tables. At this time the first challenge would be to just this. I will move on thereafter. I will use Crystal reports to create the reports needed, I already found a way to pull the same log file data in into Crystal using ODBC Microsoft text file but this is not best approach in the long term which is why i'm looking towards SQL.

https://ask.sqlservercentral.com/questions/112479/how-do-import-data-into-sql-server-from-log-file.html
0
 
LVL 12

Expert Comment

by:funwithdotnet
ID: 41734021
I remember using Log Parser. Wow, it's a very old tool (nothing wrong with that). I don't think Log Parser is going to help much with that log in its current format.

The way I see it, you have two tasks to accomplish.

The first task is to parse the original log into a format that looks something like tab-delimited:
07/25/2016 13:08:22      720000309317      207001      000005      50.00

or markup:
<record>
<date>07/25/2016 13:08:22</date>

<MerchantID>720000309317</MerchantID>
<BatchNo>207001</BatchNo>
<BatchItemCount>000005</BatchItemCount>
<NetBatchTotal>50.00</NetBatchTotal>
</record>

The second task is to import the transformed data into SQL Server. Just like the first task, there are many ways to import the data into SQL Server. Don't worry about the SQL importing process just yet. Once task #1 is accomplished, task #2 will be easy.
 
I think you'll find that Log Parser will have issues similar to SQL Server reading the log. So if it's nice for Log Parser, SQL Server can handle it alone.

Of course, the trick is to find a solution that's easy and painless for you.  Still thinking about it.
0
 
LVL 12

Accepted Solution

by:
funwithdotnet earned 500 total points
ID: 41734038
I found a page that describes how to do exactly what you want in SQL Server.
1
 

Author Comment

by:John-S Pretorius
ID: 41734835
Thank you, I agree with this.
0
 

Author Closing Comment

by:John-S Pretorius
ID: 41734838
Great, this is what I was hoping to see.
0

Featured Post

What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

JSON is being used more and more, besides XML, and you surely wanted to parse the data out into SQL instead of doing it in some Javascript. The below function in SQL Server can do the job for you, returning a quick table with the parsed data.
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
Via a live example, show how to setup several different housekeeping processes for a SQL Server.
Viewers will learn how to use the INSERT statement to insert data into their tables. It will also introduce the NULL statement, to show them what happens when no value is giving for any given column.

778 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question