Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

Calculation speed using db or application

Posted on 2006-07-07
7
Medium Priority
?
401 Views
Last Modified: 2011-09-20
Dear all,

A bit lazy to find this out by testing so here it goes

-> i have a sql db of 12000 items
-> each has 6 numeric properties
-> for period ranges up to 50000 records per item

now i need to calculate several statistics on the properties over the complete range per item

can do this on the sql db using stored procedures but was wondering if bringing over the data to .NET and do the calculations in there
and upload the completed dataset per item to the db would be faster maybe using multiple threads?

would appreciate some good arguments using this or that way

Thanks
Brian
0
Comment
Question by:bruintje
  • 3
  • 3
7 Comments
 
LVL 37

Assisted Solution

by:gregoryyoung
gregoryyoung earned 1000 total points
ID: 17058548
I I understand you properly, it will be faster to run this on the db in 99% of cases simply because the data throughput you would need to the sqlserver. If you had an extremely heavy calculation (such as a non-linear regression of the datapoints) it might be worthwhile to farm it out to a few machines, i.e. schedule it out to say 4-8+ machines to utilize the multiple processors.

Generally speaking it takes a rather hefty calculation to make this worthwhile though.

Cheers,

Greg
0
 
LVL 5

Accepted Solution

by:
bastibartel earned 1000 total points
ID: 17058643
Hi bruintje,

It is almost impossible to say without having an idea of how 'expensive' the computation is.
If you can imlement a fake algorithm (in the db) that performs a similiar number of (senseless) operations as your intended algorithm,
than you'll have an estimate for the computational expense.

If that compares to the time needed to pull the data from the db - never mind.
If calculation takes significantly longer, then you should consider writing a separate statistics module.

At that point, you have to ask yourself, what kind of operations your algorithm will perform.

Say, you have two huge tables, 1 column each  and need to mupltiply every value in the 1st table with every value in the 2nd table.
Then, a C/C++ programm will outperform the database by several orders of magnitude.

If you depend on random access via conditional statements (query-like access to rows in a set) than the db might be better.
We have had some experience with the first kind of algorithm: 10 sec in DB vs. 0.2 sec in C++.


I hope I could help a bit.

Cheers!
Sebastian
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 17058836
Sebastian .. I think you missed a part ..

"can do this on the sql db using stored procedures but was wondering if bringing over the data to .NET and do the calculations in there
and upload the completed dataset per item to the db would be faster maybe using multiple threads?"

inserting the data back to the db .. it takes a huge amount of calculation to overcome the network overhead associated with a large read combined with n writes over the network.
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
LVL 44

Author Comment

by:bruintje
ID: 17059155
hmmm both of you confirmed my fears ;-)

saving the network traffic will be worthy the dread of writing stored procedures

thanks for the input and have a great weekend!
0
 
LVL 44

Author Comment

by:bruintje
ID: 17059176
screwed the acceptance, should've been just the other way around sorry
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 17059217
no worries .. I think most are here just to help:) if I traded in all my points .. I think I could get 30 tickets for the skeeball store (and we all know what 30 tickets gets you)

0
 
LVL 44

Author Comment

by:bruintje
ID: 17059252
:)
0

Featured Post

[Webinar On Demand] Database Backup and Recovery

Does your company store data on premises, off site, in the cloud, or a combination of these? If you answered “yes”, you need a data backup recovery plan that fits each and every platform. Watch now as as Percona teaches us how to build agile data backup recovery plan.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article introduced a TextBox that supports transparent background.   Introduction TextBox is the most widely used control component in GUI design. Most GUI controls do not support transparent background and more or less do not have the…
Calculating holidays and working days is a function that is often needed yet it is not one found within the Framework. This article presents one approach to building a working-day calculator for use in .NET.
this video summaries big data hadoop online training demo (http://onlineitguru.com/big-data-hadoop-online-training-placement.html) , and covers basics in big data hadoop .
Kernel Data Recovery is a renowned Data Recovery solution provider which offers wide range of softwares for both enterprise and home users with its cost-effective solutions. Let's have a quick overview of the journey and data recovery tools range he…
Suggested Courses

572 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question