Solved and databases - create new databases/results

Posted on 2007-08-10
Last Modified: 2013-11-07
I've a application that connects to an external device and gathers data in tab delimited forms every 2 secs. The data comes simply as a long string with 50 values tab separated for processing.
This data needs to be stored along with calculated results.
The user will create different "projects" on each run on the software, by clicking "Start recording data" to start retrieving the data every two seconds.
I'm not sure whether the data retrieved (and results calculated from it) should be dumped into an SQL server database - in which case a blank database needs created every times the user clicks "Start recording data" or
into an XML file...
or any other database for that matter.

Whats are the thoughts of experts on way to go - server 2005 or XML or other?  

If server 2005 I'd need to know how to create new replica of blank database (included in installer) each time the user clicks "Start recording data"
If as an XML file, should the file be saved each 2 seconds, or just at end of recording, or at certain  intervals?
Recording may be up to two hours, 50 fields each 2 seconds.

Thanks in advance for help and advice
Question by:rwallacej
    LVL 32

    Accepted Solution

    If you're going to store the data for future analysis, I'd stick it in a database.  XML is great for transmitting data between systems and storing small amounts (e.g. application settings).

    If you're going to get any significant quantity of data, you want a DB.

    Now ... why do you need a separate database every time recording starts?  Why can't you just add a field to your table to differentiate between batches of data & populate that with something unique each time?

    Author Comment

    thanks for advce.

    "stick it in a database" - yeah, that's what I was thinking but wanted confirmation

    "add a field" - I thought the databases could get very big , so separate databases for each run might be better?  ideally stuff would be kept in one DB though.

    however, there's the issue that before each run a new build would need to be created as each run might require more fields. maybe fields can be added automatically though?
    LVL 32

    Expert Comment

    by:Daniel Wilson
    Very big ... How many GB are we talking about?

    50 fields every 2 seconds for 2 hours ... That's only 1800 rows.  How long are the fields?  Assuming each field is 1 KB (which it probably isn't that big) ... that's still only a matter of MB's on each run.

    After several runs, I might consider archiving the data to different TABLE(s), but would not look at running multiple DB's until I ran into 10's of GB's.


    Author Comment

    I'll go for the single dbase option for now and deal later if it gets too big....but that'll be a LONG time away

    Featured Post

    Better Security Awareness With Threat Intelligence

    See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

    Join & Write a Comment

    by Mark Wills Attending one of Rob Farley's seminars the other day, I heard the phrase "The Accidental DBA" and fell in love with it. It got me thinking about the plight of the newcomer to SQL Server...  So if you are the accidental DBA, or, simp…
    In SQL Server, when rows are selected from a table, does it retrieve data in the order in which it is inserted?  Many believe this is the case. Let us try to examine for ourselves with an example. To get started, use the following script, wh…
    Internet Business Fax to Email Made Easy - With eFax Corporate (, you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…
    Here's a very brief overview of the methods PRTG Network Monitor ( offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…

    745 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    15 Experts available now in Live!

    Get 1:1 Help Now