Powershell: Reading part of huge file into array

Posted on 2013-09-24
Medium Priority
Last Modified: 2014-01-03
I'm prepping some log files for insertion into a SQL table, but the log files aren't delimited properly.

So I'm running a powershell script similar to this, to get the delimiters in the right place:

$lines = @(gc infile.log)

foreach ($line in $lines)
      $line = $line.replace(" File:",",")
      $line = $line.replace(" From:<",",")
      $line = $line.replace("> To:<",",")
      add-content outfile.txt $line

This works great on test files, but when munching through a normal log file (50 MB), it uses up a lot of RAM and CPU resources, and takes forever.  And some of my log files are up to 1 GB.

Is there any way to read 100 lines at a time into the array, operate on those lines and then get the next 100 until the end of file is reached, so that we're not trying to load the entire file into memory all at once?
Question by:dustygulleson
  • 3
  • 2
LVL 42

Expert Comment

ID: 39519353
try the following.  It pushes everything through the pipeline as it comes in so memory usage should be less, and you won't have the outfile being opened and written to repeatedly which will help a lot.
gc infile.log | foreach `
      $_.replace(" File:",",")
      $_.replace(" From:<",",")
      $_.replace("> To:<",",")
} | add-content outfile.txt

Open in new window


Author Comment

ID: 39521669
I tried this...the response from PS is

"cmdlet ForEach-Object at command pipeline position 2
Supply values for the following parameters:

With a cursor after the colon.

LVL 42

Expert Comment

ID: 39522016
Make sure you have the backtick, or move the opening brace to just after the ForEach instead of on the next line.
We Need Your Input!

WatchGuard is currently running a beta program for our new macOS Host Sensor for our Threat Detection and Response service. We're looking for more macOS users to help provide insight and feedback to help us make the product even better. Please sign up for our beta program today!

LVL 72

Accepted Solution

Qlemo earned 800 total points
ID: 39522947
Doing it with this line should be even better:
gc infile.log | % { $_ -replace ' File:| From:<|> To:<', ',' } | out-file outfile.txt

Open in new window

BTW, [String]::Replace doesn't change the original string - it creates a copy, which is the result of the method call. So footech's code will result in 3 (!) lines per source line, each with a different replacement string ;-).
LVL 42

Expert Comment

ID: 39522984
Hah, that's funny! (laughing at myself)
I can't believe I let that slip by me...
LVL 72

Expert Comment

ID: 39523001
You just wanted to make it three times better :D

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

In a question here at Experts Exchange, a member was looking for "a little app that would allow sound to be turned OFF and ON by simply clicking on an icon in the system tray". This article shows how to achieve that, as well as providing the same OF…
Nuance's PaperPort may display this error message: PaperPort appears to be running Windows XP Compatibility Mode which may result in errors. We recommend disabling Compatibility Mode for the PaprPort.exe program, see Technote 6629. This articl…
This Micro Tutorial hows how you can integrate  Mac OSX to a Windows Active Directory Domain. Apple has made it easy to allow users to bind their macs to a windows domain with relative ease. The following video show how to bind OSX Mavericks to …
With the advent of Windows 10, Microsoft is pushing a Get Windows 10 icon into the notification area (system tray) of qualifying computers. There are many reasons for wanting to remove this icon. This two-part Experts Exchange video Micro Tutorial s…

619 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question