Solved

Calculation speed using db or application

Posted on 2006-07-07
7
396 Views
Last Modified: 2011-09-20
Dear all,

A bit lazy to find this out by testing so here it goes

-> i have a sql db of 12000 items
-> each has 6 numeric properties
-> for period ranges up to 50000 records per item

now i need to calculate several statistics on the properties over the complete range per item

can do this on the sql db using stored procedures but was wondering if bringing over the data to .NET and do the calculations in there
and upload the completed dataset per item to the db would be faster maybe using multiple threads?

would appreciate some good arguments using this or that way

Thanks
Brian
0
Comment
Question by:bruintje
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
7 Comments
 
LVL 37

Assisted Solution

by:gregoryyoung
gregoryyoung earned 250 total points
ID: 17058548
I I understand you properly, it will be faster to run this on the db in 99% of cases simply because the data throughput you would need to the sqlserver. If you had an extremely heavy calculation (such as a non-linear regression of the datapoints) it might be worthwhile to farm it out to a few machines, i.e. schedule it out to say 4-8+ machines to utilize the multiple processors.

Generally speaking it takes a rather hefty calculation to make this worthwhile though.

Cheers,

Greg
0
 
LVL 5

Accepted Solution

by:
bastibartel earned 250 total points
ID: 17058643
Hi bruintje,

It is almost impossible to say without having an idea of how 'expensive' the computation is.
If you can imlement a fake algorithm (in the db) that performs a similiar number of (senseless) operations as your intended algorithm,
than you'll have an estimate for the computational expense.

If that compares to the time needed to pull the data from the db - never mind.
If calculation takes significantly longer, then you should consider writing a separate statistics module.

At that point, you have to ask yourself, what kind of operations your algorithm will perform.

Say, you have two huge tables, 1 column each  and need to mupltiply every value in the 1st table with every value in the 2nd table.
Then, a C/C++ programm will outperform the database by several orders of magnitude.

If you depend on random access via conditional statements (query-like access to rows in a set) than the db might be better.
We have had some experience with the first kind of algorithm: 10 sec in DB vs. 0.2 sec in C++.


I hope I could help a bit.

Cheers!
Sebastian
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 17058836
Sebastian .. I think you missed a part ..

"can do this on the sql db using stored procedures but was wondering if bringing over the data to .NET and do the calculations in there
and upload the completed dataset per item to the db would be faster maybe using multiple threads?"

inserting the data back to the db .. it takes a huge amount of calculation to overcome the network overhead associated with a large read combined with n writes over the network.
0
Salesforce Has Never Been Easier

Improve and reinforce salesforce training & adoption using WalkMe's digital adoption platform. Start saving on costly employee training by creating fast intuitive Walk-Thrus for Salesforce. Claim your Free Account Now

 
LVL 44

Author Comment

by:bruintje
ID: 17059155
hmmm both of you confirmed my fears ;-)

saving the network traffic will be worthy the dread of writing stored procedures

thanks for the input and have a great weekend!
0
 
LVL 44

Author Comment

by:bruintje
ID: 17059176
screwed the acceptance, should've been just the other way around sorry
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 17059217
no worries .. I think most are here just to help:) if I traded in all my points .. I think I could get 30 tickets for the skeeball store (and we all know what 30 tickets gets you)

0
 
LVL 44

Author Comment

by:bruintje
ID: 17059252
:)
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction This article series is supposed to shed some light on the use of IDisposable and objects that inherit from it. In essence, a more apt title for this article would be: using (IDisposable) {}. I’m just not sure how many people would ge…
We all know that functional code is the leg that any good program stands on when it comes right down to it, however, if your program lacks a good user interface your product may not have the appeal needed to keep your customers happy. This issue can…
In this video we outline the Physical Segments view of NetCrunch network monitor. By following this brief how-to video, you will be able to learn how NetCrunch visualizes your network, how granular is the information collected, as well as where to f…
Monitoring a network: how to monitor network services and why? Michael Kulchisky, MCSE, MCSA, MCP, VTSP, VSP, CCSP outlines the philosophy behind service monitoring and why a handshake validation is critical in network monitoring. Software utilized …

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question