We help IT Professionals succeed at work.

Reading a logfile

243 Views
Last Modified: 2014-05-10
I have  a client with an application that in one week can create over a million lines in a tab delimited text file where the respective tab delimitations on a single record could exceed 455 columns.  

I need to create a solution where I can select the text logfile and search for a term and write it to report.  

Some of the text files exceed 100mg.  To read them even into excel I have had to use a tool to slice up the text files.

The logfile, for legal reasons, cannot leave the network it  is in.  The creator of the application knows that they have a problem and they have not created a solution.  

I am open to any solutions someone might have to offer.  Right now I do not have access to a server running a scripting language in the network that I could use to write a web based app.  I have asked the IT person to see if they can get that for me...I'm not holding my breath.

Thank you!
Comment
Watch Question

Darrell PorterEnterprise Business Process Architect
CERTIFIED EXPERT
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION
Bill PrewTest your restores, not your backups...
CERTIFIED EXPERT
Expert of the Year 2019
Top Expert 2016

Commented:
You might want to take a look at the Microsoft LogParser utility.  I'm not sure how well it will perform on a CSV file of 100MB, but it can give you a slick way to pull the data out.  Basically it provides a SQL based command set against non database data, in your case a CSV file.  For certain tasks it's a killer utility, but you will need to experiment with it a bit to determine if it could work for you.

Here's a wiki page about it, and then you can link to the Microsoft download page.

http://en.wikipedia.org/wiki/Logparser

~bp
billprew : excellent point.  That parser is awesome if you need to actually do work on the data at the same time.  It proved very useful in dealing with IIS logs for me.

If you're after just a quickie extract the grep tool is crazy fast though.  Not sure the native Windows tool was so much slower.
Darrell PorterEnterprise Business Process Architect
CERTIFIED EXPERT

Commented:
There's also grepWin at
http://stefanstools.sourceforge.net/grepWin.html
which can search using a string, phrase, or regular expression.
Bill PrewTest your restores, not your backups...
CERTIFIED EXPERT
Expert of the Year 2019
Top Expert 2016

Commented:
@vmoore99,

Can you tell us a little more about the use case?  Meaning how do you need to query the data specifically and what do you want the output to be - full original records, or just selected columns?  Is there a column title header row as part of the data?  Are you just doing simple string searches across all the columns, or limited to specific columns?  Any need for regex type searches?  How often do you run these searches, is it frequent or rarely?  Etc...

~bp
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION
CERTIFIED EXPERT
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
There is no header record.  There is no definition of the columns.  I do have some idea of what the first 12 columns are but the rest of it is just a recording of every movement a person makes in this application.  The data they see, the records they access.  

The query I would need would be to see all records that have a particular string anywhere in the record.  It could be column 9 or column 125.  I would then need to print out the entire record to see the movement across the system based on that string search.

The searches would be needed frequently enough (a couple times a week) to monitor employee movement and legal searches.  There's a whole body of law that must be adhered to when accessing this application.  

I am impressed at the ready response.  Thank you.  It will take me a couple of days to wander through the respective solutions.
Test your restores, not your backups...
CERTIFIED EXPERT
Expert of the Year 2019
Top Expert 2016
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION
Surranoapplication manager

Commented:
Why don't you introduce rotation on a daily basis, handling one magnitude smaller files at one step?

That said, grep is indeed your best bet if you know exactly (at least you can build up a decent regular expression) what to look for.

Author

Commented:
The beta version of the fileseek application worked great and I can turn queries over to the non-technical.  I appreciate all the responses.
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.