Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Automatically fetch .log file to push text into a SQL table

Posted on 2016-07-26
8
Medium Priority
?
58 Views
Last Modified: 2016-07-29
I have a batch file that generates every morning :- BatchClose_07252016.log and looking for a way to automatically pull the information into key fields. I would need to create the following fields/columns in SQL first and then append values on a daily basis (See image and numbering below ) :-

(1) DateTime, first line '07/25/2016 13:08:22'
(2) MerchantID, portion after the - '720000309317'
(3) BatchNo, portion after the - '207001'
(4) BatchItemCount, portion after the - '000005'
(5) NetBatchTotal, portion after the - '50.00'

C--Users-John-S-Pretorius-Desktop-ex.jpgC--Users-John-S-Pretorius-Desktop-Ex.log
0
Comment
Question by:John-S Pretorius
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
8 Comments
 
LVL 12

Expert Comment

by:funwithdotnet
ID: 41730582
The first step is to get the log data into a format SQL Server can consume. Can the format of the log be changed? If so, an XML format, among others, would allow a pure SQL Server solution using a scheduled task or a trigger.

If the format is fixed, the data needs to be transformed into something SQL Server can use. There has to be infinite ways to manipulate the log file into a database format. The trick is what is convenient and economical.  if the log was fixed-format, I would write a parser in .NET that inserted the log entries directly into the database and run that as a scheduled task or manually.
0
 

Author Comment

by:John-S Pretorius
ID: 41731023
Thank you, this is a set .log file. Looking online I found some instructions that sounded straight forward to script sql to pull specific field values after the tables are created. I'm not versed in SQL so this is my challenge.

I'm hoping someone can assist me to extract the values as required from the file structure as mentioned above.
0
 
LVL 27

Expert Comment

by:Chris Luttrell
ID: 41732323
What "straight forward" script did you find?
The first question is how is it getting into the database?  Are your scripts parsing the data first and presenting the clean data to the database or do they operate on strings once they are in the database?
What is going to trigger the file getting loaded each morning, you may want to look at a scheduled job that uses SSIS to pull from a directory, or are you going to run some script each day?
0
Veeam Disaster Recovery in Microsoft Azure

Veeam PN for Microsoft Azure is a FREE solution designed to simplify and automate the setup of a DR site in Microsoft Azure using lightweight software-defined networking. It reduces the complexity of VPN deployments and is designed for businesses of ALL sizes.

 

Author Comment

by:John-S Pretorius
ID: 41732864
Straight forward is overstating it probably, one of the solutions that seems what I was looking for was a parser.  I also saw someone write scripts to pull the data from the specific placements in the log file and append them to the already created tables. At this time the first challenge would be to just this. I will move on thereafter. I will use Crystal reports to create the reports needed, I already found a way to pull the same log file data in into Crystal using ODBC Microsoft text file but this is not best approach in the long term which is why i'm looking towards SQL.

https://ask.sqlservercentral.com/questions/112479/how-do-import-data-into-sql-server-from-log-file.html
0
 
LVL 12

Expert Comment

by:funwithdotnet
ID: 41734021
I remember using Log Parser. Wow, it's a very old tool (nothing wrong with that). I don't think Log Parser is going to help much with that log in its current format.

The way I see it, you have two tasks to accomplish.

The first task is to parse the original log into a format that looks something like tab-delimited:
07/25/2016 13:08:22      720000309317      207001      000005      50.00

or markup:
<record>
<date>07/25/2016 13:08:22</date>

<MerchantID>720000309317</MerchantID>
<BatchNo>207001</BatchNo>
<BatchItemCount>000005</BatchItemCount>
<NetBatchTotal>50.00</NetBatchTotal>
</record>

The second task is to import the transformed data into SQL Server. Just like the first task, there are many ways to import the data into SQL Server. Don't worry about the SQL importing process just yet. Once task #1 is accomplished, task #2 will be easy.
 
I think you'll find that Log Parser will have issues similar to SQL Server reading the log. So if it's nice for Log Parser, SQL Server can handle it alone.

Of course, the trick is to find a solution that's easy and painless for you.  Still thinking about it.
0
 
LVL 12

Accepted Solution

by:
funwithdotnet earned 2000 total points
ID: 41734038
I found a page that describes how to do exactly what you want in SQL Server.
1
 

Author Comment

by:John-S Pretorius
ID: 41734835
Thank you, I agree with this.
0
 

Author Closing Comment

by:John-S Pretorius
ID: 41734838
Great, this is what I was hoping to see.
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In the first part of this tutorial we will cover the prerequisites for installing SQL Server vNext on Linux.
In part one, we reviewed the prerequisites required for installing SQL Server vNext. In this part we will explore how to install Microsoft's SQL Server on Ubuntu 16.04.
Familiarize people with the process of utilizing SQL Server functions from within Microsoft Access. Microsoft Access is a very powerful client/server development tool. One of the SQL Server objects that you can interact with from within Microsoft Ac…
Viewers will learn how the fundamental information of how to create a table.

636 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question