Solved

performance - sql server select vs. select on in memory datatable

Posted on 2011-02-23
8
720 Views
Last Modified: 2012-08-13
Hi,
I’m looking for the best way to do the following thing in terms of performance time/memory usage.
I’ve a windows app. build with C#. the machine is win server 2003 with sql server 2005 DB.

I’ve a DB with a few tables, some have 1000's rows and some 10,000's rows.
in my application I need to do many select commands.
there're 2 approaches I’m aware of and I’m looking for the best one in terms of performance:
1. load the tables into datatables at the beginning of the app. run using SqlDataAdapters and then use datatable.select(xxx) to find the rows I need (no primary key selects usually select on text).
in this method I approach sql server only in the beginning of the app., loads the data to the memory and query on the datatables.
2. each time I’ve to query use a stored procedure on the DB. which means 10000's of queries on the DB.

from test I’ve made the 2nd method looks faster, but on task manager it looks like the sqlservr.exe is working very hard and takes a lot of CPU.

so what is the best way? is there another solution which is better than my solutions?
what about LINQ?

Thanks,
Assaf.
0
Comment
Question by:AssafST
  • 4
  • 2
  • 2
8 Comments
 
LVL 53

Expert Comment

by:Dhaest
ID: 34967709
Some remarks:

1. Loading the data at startup - what if data has changed ?
2. How big is the return ? Do you return 10 rows, 100 rows, 10000 rows, ... The higher you go, the inneffecient your design is made. Perhaps you can use paging to return the data in smaller pieces ...

0
 

Author Comment

by:AssafST
ID: 34967742
1. the data is changed/updated at the end of the App. run - this App. is used to update the data in the DB, so there's no fear that the data isn't correct.
2. i need all the tables data in order to run the App., so there're 6 tables with ~10,000 rows each.
0
 
LVL 53

Accepted Solution

by:
Dhaest earned 250 total points
ID: 34967781
>> 2. i need all the tables data in order to run the App., so there're 6 tables with ~10,000 rows each.

Do you mean that you update all the tables, all the records ?
0
Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

 

Author Comment

by:AssafST
ID: 34967820
no. i need all the data in order to update 3 tables (not all the records).
i don't do the update in memory but create a long string of SP commands and at the end of the run update the DB.
something like:
string updateDBstr = "exec SP1 p1, p2,p3; exec SP1 p5, p6,p7; exec SP2 p1, p2,p3;" etc...
objCommO.CommandText = updateDBstr;
objCommO.ExecuteNonQuery();
0
 
LVL 39

Expert Comment

by:lcohan
ID: 34970787
If you have enough memory on the SQL box you should not worry about that as SQL will put those tables in cache and keep them as long as they are needed (especially on not so large tables as yours). Until SQL 2005 you could have use DBCC PINTABLE sql command to force SQL keep those tables in memory but was discontinued.

you should check your page life expectancy/SQL Server Buffer Cache Hit Ratio and if less than 90% you may need better indexing/stats or more RAM on the box
0
 

Author Comment

by:AssafST
ID: 34971946
1. what do you mean by "put those tables in cache"? put them in datatables as i do?
2. "you should check your page life expectancy/SQL Server Buffer Cache Hit Ratio"
how do i check this?

10x
0
 
LVL 39

Assisted Solution

by:lcohan
lcohan earned 250 total points
ID: 34973237
SQL server will put the data in cache and as your tables are small is likely they will be entirely cached on servers with sufficient RAM.

You can run command below to check the values in SQL:


select * from sys.sysperfinfo
   where counter_name in ('Buffer cache hit ratio','Page life expectancy' )
order by counter_name



0
 

Author Closing Comment

by:AssafST
ID: 35139243
not a final solution
0

Featured Post

The Eight Noble Truths of Backup and Recovery

How can IT departments tackle the challenges of a Big Data world? This white paper provides a roadmap to success and helps companies ensure that all their data is safe and secure, no matter if it resides on-premise with physical or virtual machines or in the cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Wouldn’t it be nice if you could test whether an element is contained in an array by using a Contains method just like the one available on List objects? Wouldn’t it be good if you could write code like this? (CODE) In .NET 3.5, this is possible…
This article shows how to deploy dynamic backgrounds to computers depending on the aspect ratio of display
This Micro Tutorial will teach you how to censor certain areas of your screen. The example in this video will show a little boy's face being blurred. This will be demonstrated using Adobe Premiere Pro CS6.
This Micro Tutorial hows how you can integrate  Mac OSX to a Windows Active Directory Domain. Apple has made it easy to allow users to bind their macs to a windows domain with relative ease. The following video show how to bind OSX Mavericks to …

786 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question