Powershell: Reading part of huge file into array

I'm prepping some log files for insertion into a SQL table, but the log files aren't delimited properly.

So I'm running a powershell script similar to this, to get the delimiters in the right place:

$lines = @(gc infile.log)

foreach ($line in $lines)
{
      $line = $line.replace(" File:",",")
      $line = $line.replace(" From:<",",")
      $line = $line.replace("> To:<",",")
      add-content outfile.txt $line
}

This works great on test files, but when munching through a normal log file (50 MB), it uses up a lot of RAM and CPU resources, and takes forever.  And some of my log files are up to 1 GB.

Is there any way to read 100 lines at a time into the array, operate on those lines and then get the next 100 until the end of file is reached, so that we're not trying to load the entire file into memory all at once?
dustygullesonAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

footechCommented:
try the following.  It pushes everything through the pipeline as it comes in so memory usage should be less, and you won't have the outfile being opened and written to repeatedly which will help a lot.
gc infile.log | foreach `
{
      $_.replace(" File:",",")
      $_.replace(" From:<",",")
      $_.replace("> To:<",",")
} | add-content outfile.txt

Open in new window

0
dustygullesonAuthor Commented:
I tried this...the response from PS is

"cmdlet ForEach-Object at command pipeline position 2
Supply values for the following parameters:
Proccess[0]:"

With a cursor after the colon.

?
0
footechCommented:
Make sure you have the backtick, or move the opening brace to just after the ForEach instead of on the next line.
0
Simplify Active Directory Administration

Administration of Active Directory does not have to be hard.  Too often what should be a simple task is made more difficult than it needs to be.The solution?  Hyena from SystemTools Software.  With ease-of-use as well as powerful importing and bulk updating capabilities.

Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
Doing it with this line should be even better:
gc infile.log | % { $_ -replace ' File:| From:<|> To:<', ',' } | out-file outfile.txt

Open in new window

BTW, [String]::Replace doesn't change the original string - it creates a copy, which is the result of the method call. So footech's code will result in 3 (!) lines per source line, each with a different replacement string ;-).
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
footechCommented:
Hah, that's funny! (laughing at myself)
I can't believe I let that slip by me...
0
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
You just wanted to make it three times better :D
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Powershell

From novice to tech pro — start learning today.