Handling very large data imports and exports with DataSet.WriteXml and ReadXml

Posted on 2009-02-09
Last Modified: 2012-06-22
I am investigating creating an import and export utility using the WriteXml and ReadXml methods on the DataSet class.

In my investigations I am trying to see if there is some way to batch up the data being processed, so that when exporting for example (WriteXml) there is a way to get the data in batches and process a batch at a time instead of trying to write all the data at once.

I can see the SqlDataAdapter has some batch properties and methods, but I can't find any explanation of how to use them if it they are suitable for what I want.

Essentially I would like to say "get xxx rows from the database, and write them to this file, then get the next xxx rows from the database and either append the existing file or write to a new file" and continue until all the rows in the table have been processed.
Question by:purplesoup
    1 Comment
    LVL 4

    Accepted Solution

    In previous version of DataAdapter (.Net 1.0 and 1.1) the SqlDataAdpater execute 1 command per DataRow. In ADO .Net 2.0 SqlDataAdapter has this porperty to set the number of concurrent commands to execute. You can set it to tell SqlDataAdapter to autoconfigure itself ( Default ) setting it to 0 or to you can set the Window Size to any number, for example 5000.
    If you are building an Importer tool, i recomend the Integration Services of Sql Server 2005, it has a very performance set of componentes to do every thing than you want.

    Featured Post

    Live: Real-Time Solutions, Start Here

    Receive instant 1:1 support from technology experts, using our real-time conversation and whiteboard interface. Your first 5 minutes are always free.

    Join & Write a Comment

    For those of you who don't follow the news, or just happen to live under rocks, Microsoft Research released a beta SDK ( for the Xbox 360 Kinect. If you don't know what a Kinect is (http:…
    More often than not, we developers are confronted with a need: a need to make some kind of magic happen via code. Whether it is for a client, for the boss, or for our own personal projects, the need must be satisfied. Most of the time, the Framework…
    Internet Business Fax to Email Made Easy - With eFax Corporate (, you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…
    Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

    755 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    16 Experts available now in Live!

    Get 1:1 Help Now