Solved

performance - sql server select vs. select on in memory datatable

Posted on 2011-02-23
8
716 Views
Last Modified: 2012-08-13
Hi,
I’m looking for the best way to do the following thing in terms of performance time/memory usage.
I’ve a windows app. build with C#. the machine is win server 2003 with sql server 2005 DB.

I’ve a DB with a few tables, some have 1000's rows and some 10,000's rows.
in my application I need to do many select commands.
there're 2 approaches I’m aware of and I’m looking for the best one in terms of performance:
1. load the tables into datatables at the beginning of the app. run using SqlDataAdapters and then use datatable.select(xxx) to find the rows I need (no primary key selects usually select on text).
in this method I approach sql server only in the beginning of the app., loads the data to the memory and query on the datatables.
2. each time I’ve to query use a stored procedure on the DB. which means 10000's of queries on the DB.

from test I’ve made the 2nd method looks faster, but on task manager it looks like the sqlservr.exe is working very hard and takes a lot of CPU.

so what is the best way? is there another solution which is better than my solutions?
what about LINQ?

Thanks,
Assaf.
0
Comment
Question by:AssafST
  • 4
  • 2
  • 2
8 Comments
 
LVL 53

Expert Comment

by:Dhaest
Comment Utility
Some remarks:

1. Loading the data at startup - what if data has changed ?
2. How big is the return ? Do you return 10 rows, 100 rows, 10000 rows, ... The higher you go, the inneffecient your design is made. Perhaps you can use paging to return the data in smaller pieces ...

0
 

Author Comment

by:AssafST
Comment Utility
1. the data is changed/updated at the end of the App. run - this App. is used to update the data in the DB, so there's no fear that the data isn't correct.
2. i need all the tables data in order to run the App., so there're 6 tables with ~10,000 rows each.
0
 
LVL 53

Accepted Solution

by:
Dhaest earned 250 total points
Comment Utility
>> 2. i need all the tables data in order to run the App., so there're 6 tables with ~10,000 rows each.

Do you mean that you update all the tables, all the records ?
0
 

Author Comment

by:AssafST
Comment Utility
no. i need all the data in order to update 3 tables (not all the records).
i don't do the update in memory but create a long string of SP commands and at the end of the run update the DB.
something like:
string updateDBstr = "exec SP1 p1, p2,p3; exec SP1 p5, p6,p7; exec SP2 p1, p2,p3;" etc...
objCommO.CommandText = updateDBstr;
objCommO.ExecuteNonQuery();
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 
LVL 39

Expert Comment

by:lcohan
Comment Utility
If you have enough memory on the SQL box you should not worry about that as SQL will put those tables in cache and keep them as long as they are needed (especially on not so large tables as yours). Until SQL 2005 you could have use DBCC PINTABLE sql command to force SQL keep those tables in memory but was discontinued.

you should check your page life expectancy/SQL Server Buffer Cache Hit Ratio and if less than 90% you may need better indexing/stats or more RAM on the box
0
 

Author Comment

by:AssafST
Comment Utility
1. what do you mean by "put those tables in cache"? put them in datatables as i do?
2. "you should check your page life expectancy/SQL Server Buffer Cache Hit Ratio"
how do i check this?

10x
0
 
LVL 39

Assisted Solution

by:lcohan
lcohan earned 250 total points
Comment Utility
SQL server will put the data in cache and as your tables are small is likely they will be entirely cached on servers with sufficient RAM.

You can run command below to check the values in SQL:


select * from sys.sysperfinfo
   where counter_name in ('Buffer cache hit ratio','Page life expectancy' )
order by counter_name



0
 

Author Closing Comment

by:AssafST
Comment Utility
not a final solution
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

Wouldn’t it be nice if you could test whether an element is contained in an array by using a Contains method just like the one available on List objects? Wouldn’t it be good if you could write code like this? (CODE) In .NET 3.5, this is possible…
A long time ago (May 2011), I have written an article showing you how to create a DLL using Visual Studio 2005 to be hosted in SQL Server 2005. That was valid at that time and it is still valid if you are still using these versions. You can still re…
Illustrator's Shape Builder tool will let you combine shapes visually and interactively. This video shows the Mac version, but the tool works the same way in Windows. To follow along with this video, you can draw your own shapes or download the file…
This tutorial demonstrates a quick way of adding group price to multiple Magento products.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

6 Experts available now in Live!

Get 1:1 Help Now