• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 296
  • Last Modified:

third party .Net Data providers

I’m writing a desktop application that process a large amount of data, currently I use XML file and DataSet.ReadXML() and DataSet.WriteXML(). But due to the large size of XML files (~10MB) the performance drag and resources usage is more than a lot of users would be happy with.

Now, I was planning to use n MS Access database file, but I skipped that too since you need to have MDAC 2.8 and Jet installed and the download size is around 8.

I need a solution either something inside .Net framework, or a third-party data provider which would be able to provide the data to my application in reasonable speed without using a lot of resources. Data source shouldn’t require any kind of server installed. It could have any type of database format (its own, xml, ms access…). The component and all its requirements shouldn't be big (less than ~3MB) preferably open source or free, if it’s not free a trail version should be available for download.
0
saturn_one
Asked:
saturn_one
  • 5
  • 3
1 Solution
 
_kiwi_Commented:
Hi,

One thing you can do is to keep using your Xml Files, which is gonna be the most simple thing to put on users computers, sincez you don't have anything to install.

The point is to avoid using DataSets with ReadXml and WriteXml, which are pretty memory and time consuming if you are using large amounts of data.

For my part, I still consider 10MB is a small amount of data, 100MB to 1GB is getting a big amount of data.

Now, for reading the data and processing it, you can use XmlTextReader and XmlTextWriter to read the data line by line and have it processed.

Now If you have Constraints and RelationShips, this method is gonna be tough and maybe slower than DataSets since you will read the data a lot of times.

A good thing with these readers is that it can still be applied to Database commands through the ExecuteXml method, so if you find a DBLike server suiting your needs, you can keep using this method.

Cheers,
Julien

0
 
tomvergoteCommented:
I'd use sqllite, which is a very lightweight database and it has a .NET provider.
It comes as a single dll without external dependencies, and if you wish an additional command line exe to manipulate the database.
the databases are contained in a single .db file and it's blazingly fast with a very low footprint, but yet its still fully transactional
it's free too
http://www.sqlite.org/
http://sourceforge.net/projects/adodotnetsqlite
0
 
tomvergoteCommented:
forgot the best part: sqlites license allows you to distribute the dll with your application without the need for users to install anything, just drop it in your bin
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
saturn_oneAuthor Commented:
doesn't sqlite require MDAC or jet? and would you have any documents on how to use it in a .net application?
0
 
tomvergoteCommented:
http://sourceforge.net/docman/?group_id=94056
the .NET provider site has docs
0
 
saturn_oneAuthor Commented:
sqlite seems to be a really good option. there is only one problem, i can't find any programs to design a sqlite(version 3) database in windows, there is "DBTools Manager Professional" but thats only for version 2, i can't  even find a program that would convert database (MS Access or sqlite v2) to version 3.
0
 
tomvergoteCommented:
sqllite isn't very strong on the gui side, you probably could use access or similar as a front end to it but i usually just use sql with the command line client.
You trade some user friendlyness for a small footprint maybe
0
 
saturn_oneAuthor Commented:
I just finished porting a small part of my application to SQLite and WOW, i don't know how to say thank you, you saved me ALOT of headaces. 2000 records read in less than a second. and barely no footprind, i used to have a 100MB  foot print and 30-60 seconds loadtime.

Thank you.
0
 
tomvergoteCommented:
you're welcome

there are plenty of small open source projects which offer very good value in specific situations, the problem is to find them
0

Featured Post

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

  • 5
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now