third party .Net Data providers

I’m writing a desktop application that process a large amount of data, currently I use XML file and DataSet.ReadXML() and DataSet.WriteXML(). But due to the large size of XML files (~10MB) the performance drag and resources usage is more than a lot of users would be happy with.

Now, I was planning to use n MS Access database file, but I skipped that too since you need to have MDAC 2.8 and Jet installed and the download size is around 8.

I need a solution either something inside .Net framework, or a third-party data provider which would be able to provide the data to my application in reasonable speed without using a lot of resources. Data source shouldn’t require any kind of server installed. It could have any type of database format (its own, xml, ms access…). The component and all its requirements shouldn't be big (less than ~3MB) preferably open source or free, if it’s not free a trail version should be available for download.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.


One thing you can do is to keep using your Xml Files, which is gonna be the most simple thing to put on users computers, sincez you don't have anything to install.

The point is to avoid using DataSets with ReadXml and WriteXml, which are pretty memory and time consuming if you are using large amounts of data.

For my part, I still consider 10MB is a small amount of data, 100MB to 1GB is getting a big amount of data.

Now, for reading the data and processing it, you can use XmlTextReader and XmlTextWriter to read the data line by line and have it processed.

Now If you have Constraints and RelationShips, this method is gonna be tough and maybe slower than DataSets since you will read the data a lot of times.

A good thing with these readers is that it can still be applied to Database commands through the ExecuteXml method, so if you find a DBLike server suiting your needs, you can keep using this method.


I'd use sqllite, which is a very lightweight database and it has a .NET provider.
It comes as a single dll without external dependencies, and if you wish an additional command line exe to manipulate the database.
the databases are contained in a single .db file and it's blazingly fast with a very low footprint, but yet its still fully transactional
it's free too

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
forgot the best part: sqlites license allows you to distribute the dll with your application without the need for users to install anything, just drop it in your bin
Introducing Cloud Class® training courses

Tech changes fast. You can learn faster. That’s why we’re bringing professional training courses to Experts Exchange. With a subscription, you can access all the Cloud Class® courses to expand your education, prep for certifications, and get top-notch instructions.

saturn_oneAuthor Commented:
doesn't sqlite require MDAC or jet? and would you have any documents on how to use it in a .net application?
the .NET provider site has docs
saturn_oneAuthor Commented:
sqlite seems to be a really good option. there is only one problem, i can't find any programs to design a sqlite(version 3) database in windows, there is "DBTools Manager Professional" but thats only for version 2, i can't  even find a program that would convert database (MS Access or sqlite v2) to version 3.
sqllite isn't very strong on the gui side, you probably could use access or similar as a front end to it but i usually just use sql with the command line client.
You trade some user friendlyness for a small footprint maybe
saturn_oneAuthor Commented:
I just finished porting a small part of my application to SQLite and WOW, i don't know how to say thank you, you saved me ALOT of headaces. 2000 records read in less than a second. and barely no footprind, i used to have a 100MB  foot print and 30-60 seconds loadtime.

Thank you.
you're welcome

there are plenty of small open source projects which offer very good value in specific situations, the problem is to find them
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.