Solved

Calculation speed using db or application

Posted on 2006-07-07
7
392 Views
Last Modified: 2011-09-20
Dear all,

A bit lazy to find this out by testing so here it goes

-> i have a sql db of 12000 items
-> each has 6 numeric properties
-> for period ranges up to 50000 records per item

now i need to calculate several statistics on the properties over the complete range per item

can do this on the sql db using stored procedures but was wondering if bringing over the data to .NET and do the calculations in there
and upload the completed dataset per item to the db would be faster maybe using multiple threads?

would appreciate some good arguments using this or that way

Thanks
Brian
0
Comment
Question by:bruintje
  • 3
  • 3
7 Comments
 
LVL 37

Assisted Solution

by:gregoryyoung
gregoryyoung earned 250 total points
ID: 17058548
I I understand you properly, it will be faster to run this on the db in 99% of cases simply because the data throughput you would need to the sqlserver. If you had an extremely heavy calculation (such as a non-linear regression of the datapoints) it might be worthwhile to farm it out to a few machines, i.e. schedule it out to say 4-8+ machines to utilize the multiple processors.

Generally speaking it takes a rather hefty calculation to make this worthwhile though.

Cheers,

Greg
0
 
LVL 5

Accepted Solution

by:
bastibartel earned 250 total points
ID: 17058643
Hi bruintje,

It is almost impossible to say without having an idea of how 'expensive' the computation is.
If you can imlement a fake algorithm (in the db) that performs a similiar number of (senseless) operations as your intended algorithm,
than you'll have an estimate for the computational expense.

If that compares to the time needed to pull the data from the db - never mind.
If calculation takes significantly longer, then you should consider writing a separate statistics module.

At that point, you have to ask yourself, what kind of operations your algorithm will perform.

Say, you have two huge tables, 1 column each  and need to mupltiply every value in the 1st table with every value in the 2nd table.
Then, a C/C++ programm will outperform the database by several orders of magnitude.

If you depend on random access via conditional statements (query-like access to rows in a set) than the db might be better.
We have had some experience with the first kind of algorithm: 10 sec in DB vs. 0.2 sec in C++.


I hope I could help a bit.

Cheers!
Sebastian
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 17058836
Sebastian .. I think you missed a part ..

"can do this on the sql db using stored procedures but was wondering if bringing over the data to .NET and do the calculations in there
and upload the completed dataset per item to the db would be faster maybe using multiple threads?"

inserting the data back to the db .. it takes a huge amount of calculation to overcome the network overhead associated with a large read combined with n writes over the network.
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 
LVL 44

Author Comment

by:bruintje
ID: 17059155
hmmm both of you confirmed my fears ;-)

saving the network traffic will be worthy the dread of writing stored procedures

thanks for the input and have a great weekend!
0
 
LVL 44

Author Comment

by:bruintje
ID: 17059176
screwed the acceptance, should've been just the other way around sorry
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 17059217
no worries .. I think most are here just to help:) if I traded in all my points .. I think I could get 30 tickets for the skeeball store (and we all know what 30 tickets gets you)

0
 
LVL 44

Author Comment

by:bruintje
ID: 17059252
:)
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
creating a flowchart from an algorithm 5 30
which object should i use (efficiency) 5 46
Icons and Colors for Terms 3 24
Please explain "Multi-Tenant Services" 5 63
Introduction Although it is an old technology, serial ports are still being used by many hardware manufacturers. If you develop applications in C#, Microsoft .NET framework has SerialPort class to communicate with the serial ports.  I needed to…
It was really hard time for me to get the understanding of Delegates in C#. I went through many websites and articles but I found them very clumsy. After going through those sites, I noted down the points in a easy way so here I am sharing that unde…
With the power of JIRA, there's an unlimited number of ways you can customize it, use it and benefit from it. With that in mind, there's bound to be things that I wasn't able to cover in this course. With this summary we'll look at some places to go…
Both in life and business – not all partnerships are created equal. As the demand for cloud services increases, so do the number of self-proclaimed cloud partners. Asking the right questions up front in the partnership, will enable both parties …

920 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now