fastest or most efficient way to store lots of data

jscharpf used Ask the Experts™
I have a VB program that will be generating alot of raw data.
For example.. it will be running a battery test, and I will be logging about 100 single precision values 4 times per second for several days...
I started by just adding records to and Access database, but it appears that could get very large.. and maybe slow down??

Is using a text file faster than Access? Does it take more or less memory to store the same amount of data?


Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
I recommend with that kind of data, if you don't need to access the data in real-time, that you gather the data in local memory until you reach a threshold, then pass that data off to be saved.

With your 400 Single entries per second, you may want to write out (asynchronously) the 24000 entries to a DB every minute.  With memory being cheap and the assumption that the box will probably not be in use during these tests by someone, this is a workable solution.  I recommend a DB for the full-features you will want to evaluate these numbers.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial