asked on

Powershell: Reading part of huge file into array

I'm prepping some log files for insertion into a SQL table, but the log files aren't delimited properly.

So I'm running a powershell script similar to this, to get the delimiters in the right place:

$lines = @(gc infile.log)

foreach ($line in $lines)
{
      $line = $line.replace(" File:",",")
      $line = $line.replace(" From:<",",")
      $line = $line.replace("> To:<",",")
      add-content outfile.txt $line
}

This works great on test files, but when munching through a normal log file (50 MB), it uses up a lot of RAM and CPU resources, and takes forever. And some of my log files are up to 1 GB.

Is there any way to read 100 lines at a time into the array, operate on those lines and then get the next 100 until the end of file is reached, so that we're not trying to load the entire file into memory all at once?

footech

try the following. It pushes everything through the pipeline as it comes in so memory usage should be less, and you won't have the outfile being opened and written to repeatedly which will help a lot.

gc infile.log | foreach `
{
      $_.replace(" File:",",")
      $_.replace(" From:<",",")
      $_.replace("> To:<",",")
} | add-content outfile.txt

Open in new window

dustygulleson

ASKER

I tried this...the response from PS is

"cmdlet ForEach-Object at command pipeline position 2
Supply values for the following parameters:
Proccess[0]:"

With a cursor after the colon.

?

footech

Make sure you have the backtick, or move the opening brace to just after the ForEach instead of on the next line.

ASKER CERTIFIED SOLUTION

Qlemo

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

footech

Hah, that's funny! (laughing at myself)
I can't believe I let that slip by me...

Qlemo

You just wanted to make it three times better :D