Avatar of egarrison
Flag for United States of America asked on

Importing large / complex XML

I need a little direction on a project.  An import is going from PIPE delimited to XML.  The Daily files are averaging 4GB.  I need to know the fastest way to import this large XML file. Not fast from setup; I have time to do this right, I mean fastest for the actually import.

It seems like XML Bulk Load should be one of the fastest:

My issue is that the file is much more complex.  Some records may have 1 detail node, some 2000.  The data is Invoices (header info) and the Invoice details (1 to many rows).  

The second option is OpenXML.  http://msdn.microsoft.com/en-us/library/ms186918.aspx
This option seems like a close fit, but I don't have any experience or benchmarks for it.  The problem is that XML will be more flexible and better in the long run, but I need the speed I get from Bulk Load mode and BCP.  

Third option ? Suggestions. Maybe Third party software.  
I have tried SpectralCore's Full Convert Enterprise.  It could not handle the file.

Hardware: I cannot get much better hardware.  HP DL380G8.  128GB Ram, Netapp Fiber Storage.  SQL Server 2012 Enterprise.
Microsoft SQL Server 2008SSIS

Avatar of undefined
Last Comment
Anthony Perkins

8/22/2022 - Mon


Oleggold - I tried the import.  I am getting this error:

Msg 6365, Level 16, State 1, Line 5
An XML operation resulted an XML data type exceeding 2GB in size. Operation aborted.

Is there a 2GB limit on XML Files?

Does anyone know the best way to Import 4-6GB XML Files?
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
Anthony Perkins

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question