• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 644
  • Last Modified:

Handling very large data imports and exports with DataSet.WriteXml and ReadXml

I am investigating creating an import and export utility using the WriteXml and ReadXml methods on the DataSet class.

In my investigations I am trying to see if there is some way to batch up the data being processed, so that when exporting for example (WriteXml) there is a way to get the data in batches and process a batch at a time instead of trying to write all the data at once.

I can see the SqlDataAdapter has some batch properties and methods, but I can't find any explanation of how to use them if it they are suitable for what I want.

Essentially I would like to say "get xxx rows from the database, and write them to this file, then get the next xxx rows from the database and either append the existing file or write to a new file" and continue until all the rows in the table have been processed.
0
purplesoup
Asked:
purplesoup
1 Solution
 
jppampinCommented:
In previous version of DataAdapter (.Net 1.0 and 1.1) the SqlDataAdpater execute 1 command per DataRow. In ADO .Net 2.0 SqlDataAdapter has this porperty to set the number of concurrent commands to execute. You can set it to tell SqlDataAdapter to autoconfigure itself ( Default ) setting it to 0 or to you can set the Window Size to any number, for example 5000.
If you are building an Importer tool, i recomend the Integration Services of Sql Server 2005, it has a very performance set of componentes to do every thing than you want.
0

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now