We help IT Professionals succeed at work.

What is better to use a text file or sql db for data intensive app?

216 Views
Last Modified: 2012-05-06
I'm building an application that will allow the user to import and name a hand full of data files (Excel data) each with a max of approx 10000 records, then the application will:
- compare record keys across the imported data files and report out what is missing between each data file and what is extra
- do some summing, division, max, mins, averages across all records in all tables
- remove and add several records across all data files depending on what's missing or extra across all data files
- rebuild all of the imported data files into a one file output

The user will import all of this data, run the application and take the output, then will do another project the same way, and then another and another each with different data and data files...sort of like creating Excel spreadsheets for various projects...each file is a totally different project, not the re-use of the same data or data structure

My question is: What is the best type of flat DB to use in terms of speed for user, ease to build the application? : text files, sql db, other?

Thank you very much!
Comment
Watch Question

CERTIFIED EXPERT
Awarded 2008
Awarded 2008

Commented:
sql database...FOR SURE.  you can use sql express...its free!
CERTIFIED EXPERT
Awarded 2008
Awarded 2008

Commented:

Commented:
I agree with chapmandew...text file only to use for keeping very small pieces of data. For structured data, use an database engine like SQL Express. As said, it's free and ideal for small applications.

jppinto

Author

Commented:
If the user begins a project which generates a SQL DB, can they have complete freedom to save the SQL DB anywhere on their computer and then possibly copy and paste it to a new location and reopen as easily as a text file or as easily as an MS Office app such as Excel, PowerPt etc?
Scott McDaniel (EE MVE )Infotrakker Software
CERTIFIED EXPERT
Most Valuable Expert 2012
Top Expert 2014

Commented:
Not with SQL Server. Of course, why would you need to do this with a relational database?

You can do this with an Access database ...
CERTIFIED EXPERT
Most Valuable Expert 2015
Distinguished Expert 2018

Commented:
No, doable but not easily.

For 6 times 10000 records Access/JET will definitely be the fastest datastore.
And this single mdb file can be copied and moved just like any text file.

/gustav

Commented:
For what you want to do...Access is the best option for you. It's fully integrated with other Office tools.

Author

Commented:
I want to stay away from a paid licensed DB provider such as Access because not all users of the this app will have an Access license.  From what I understand if the app uses a SQL DB, anyone who downloads SQL Server Express for free from the MS website will be able to use this app without having to purchase any kind of DB license...am I wrong?
CERTIFIED EXPERT
Most Valuable Expert 2015
Distinguished Expert 2018
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION
Scott McDaniel (EE MVE )Infotrakker Software
CERTIFIED EXPERT
Most Valuable Expert 2012
Top Expert 2014
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
So what is the advantage of using a MDB (Access like DB) versus a MDF (SQL DB)?   Which one offers more file portability for the app user (VB.Net app I'm building) and which one has more robust analytics (VB.Net app will process data faster and is easier for me to build)?

Thanks again...I've already sunk a number of hours into this app and just want to make sure I'm on the right  path;)
CERTIFIED EXPERT
Awarded 2008
Awarded 2008
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
Thanks for your help!
CERTIFIED EXPERT
Awarded 2008
Awarded 2008

Commented:
These were not B answers....

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.