Solved

Read large XML file with Powershell

Posted on 2010-08-30
4
2,947 Views
Last Modified: 2013-12-04
I have a 500mb+ xml file I need to break apart, but every time I do Get-Content file it is too large for memory. Is there any way to process each node in the root one at a time similar to reading a stream?

The error: Exception of type 'System.OutOfMemoryException' was thrown.
0
Comment
Question by:RyanAndres
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
4 Comments
 
LVL 65

Expert Comment

by:rockiroads
ID: 33561176
0
 
LVL 8

Author Comment

by:RyanAndres
ID: 33563459
No, it uses $xml = [xml](gc file) ....

This is what I tried and it will load the entire file into memory.. which will cause the exception.
0
 
LVL 71

Accepted Solution

by:
Chris Dent earned 500 total points
ID: 33565457

You should be able to initialise an instance of System.Xml.XmlReader using the StreamReader.

e.g.

$Reader = New-Object IO.StreamReader("FullPathToFile")
$XmlReader = [Xml.XmlReader]::Create($Reader)

Since it's reading it as a stream rather than loading the entire file into memory it "should" work better. However, I cannot test that theory since I don't have any large XML files.

Chris
0
 
LVL 8

Author Closing Comment

by:RyanAndres
ID: 33568419
Cool I used XmlTextReader and Writer.
0

Featured Post

Are You Headed to Black Hat USA 2017?

Getting ready for Black Hat next week? Kick things off with the WatchGuard Badge Challenge and test your puzzle and cipher skills. Do you have what it takes to earn our limited edition Firebox Badge? Get started today - https://crimsonthorn.net

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Utilizing an array to gracefully append to a list of EmailAddresses
Previously, on our Nano Server Deployment series, we've created a new nano server image and deployed it on a physical server in part 2. Now we will go through configuration.
This is Part 3 in a 3-part series on Experts Exchange to discuss error handling in VBA code written for Excel. Part 1 of this series discussed basic error handling code using VBA. http://www.experts-exchange.com/videos/1478/Excel-Error-Handlin…
Exchange organizations may use the Journaling Agent of the Transport Service to archive messages going through Exchange. However, if the Transport Service is integrated with some email content management application (such as an antispam), the admini…

617 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question