Avatar of egarrison
egarrisonFlag for United States of America

asked on 

Importing large / complex XML

I need a little direction on a project.  An import is going from PIPE delimited to XML.  The Daily files are averaging 4GB.  I need to know the fastest way to import this large XML file. Not fast from setup; I have time to do this right, I mean fastest for the actually import.

It seems like XML Bulk Load should be one of the fastest:
http://support.microsoft.com/kb/316005

My issue is that the file is much more complex.  Some records may have 1 detail node, some 2000.  The data is Invoices (header info) and the Invoice details (1 to many rows).  

The second option is OpenXML.  http://msdn.microsoft.com/en-us/library/ms186918.aspx
This option seems like a close fit, but I don't have any experience or benchmarks for it.  The problem is that XML will be more flexible and better in the long run, but I need the speed I get from Bulk Load mode and BCP.  

Third option ? Suggestions. Maybe Third party software.  
I have tried SpectralCore's Full Convert Enterprise.  It could not handle the file.

Hardware: I cannot get much better hardware.  HP DL380G8.  128GB Ram, Netapp Fiber Storage.  SQL Server 2012 Enterprise.
Microsoft SQL Server 2008SSIS

Avatar of undefined
Last Comment
Anthony Perkins
Avatar of oleggold
oleggold
Flag of United States of America image

Avatar of egarrison
egarrison
Flag of United States of America image

ASKER

Oleggold - I tried the import.  I am getting this error:

Msg 6365, Level 16, State 1, Line 5
An XML operation resulted an XML data type exceeding 2GB in size. Operation aborted.

Is there a 2GB limit on XML Files?
Avatar of egarrison
egarrison
Flag of United States of America image

ASKER

Does anyone know the best way to Import 4-6GB XML Files?
ASKER CERTIFIED SOLUTION
Avatar of Anthony Perkins
Anthony Perkins
Flag of United States of America image

Blurred text
THIS SOLUTION IS ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
Microsoft SQL Server 2008
Microsoft SQL Server 2008

Microsoft SQL Server 2008 is a suite of relational database management system (RDBMS) products providing multi-user database access functionality.Component services include integration (SSIS), reporting (SSRS), analysis (SSAS), data quality, master data, T-SQL and performance tuning. Major improvements include the Always On technologies and support for unstructured data types.

50K
Questions
--
Followers
--
Top Experts
Get a personalized solution from industry experts
Ask the experts
Read over 600 more reviews

TRUSTED BY

IBM logoIntel logoMicrosoft logoUbisoft logoSAP logo
Qualcomm logoCitrix Systems logoWorkday logoErnst & Young logo
High performer badgeUsers love us badge
LinkedIn logoFacebook logoX logoInstagram logoTikTok logoYouTube logo