Improve company productivity with a Business Account.Sign Up

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 245
  • Last Modified:

Loading rows from a view (no index)

hi guys

we have a remote view from which we are pulling data into our local table.

The remote view is :REMOTE_CUSTOMER
and our local table is :LOCAL_CUSTOMER

This is what we are doing right now.

We first delete data from our local table
DELETE Top (50000) FROM LOCAL_CUSTOMER where Id = '209'

Then we do a select to pull the data from the view (the remote view does not have any indexes, can we assign a index to view? Also the remote view doesnt have a primary key)

SELECT FirstName,LastName,BusinessDescription,ProductDescription,ProductSourceMap,updateBy,updateTimestamp FROM remote.dbview.REMOTE_CUSTOMER

then we insert into our local table

INSERT INTO LOCAL_CUSTOMER (FirstName,LastName,BusinessDescription,ProductDescription,ProductSourceMap,updateBy,updateTimestamp,Id)
VALUES(:FirstName, :LastName, :BusinessDescription, :ProductDescription, :ProductSourceMap, :updateBy,:updateTimestamp,:Id)

There are over a million rows in the REMOTE_CUSTOMER view and this process takes around 30 minutes to get the data from the view and populate into our table.

Is there a way to improve the speed of this entire process?

thanks
0
royjayd
Asked:
royjayd
  • 4
  • 3
1 Solution
 
yechanCommented:
If you have any indexes on the LOCAL_CUSTOMER table my suggestion is to drop those indexes, load your data, and then re-create those indexes.
0
 
royjaydAuthor Commented:
ok, i dropped the index, but it takes the same time to insert the rows into
our LOCAL_CUSTOMER table (around 30 minutes)
Any other advice other than dropping the indexes ?

Can i create multiple threads and do parallel processing in sql server 2008 to increase the speed?

thanks
0
 
yechanCommented:
Do you have a WHERE clause against the REMOTE_CUSTOMER table/view?  If yes, then I think adding an index would be very beneficial.

Other things to check for is that the LOCAL_CUSTOMER table does not have any triggers.  Change the "Recovery Level" of the database to Simple.  

Using SSMS to create a multiple threads would be difficult unless you open several tabs =)

I think if you want to do some parallel processing, using SSIS is the way to go.
0
Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
royjaydAuthor Commented:
hi

actually there is no Where clause in the select statement to query the view.
its just a simple sql

SELECT FirstName,LastName,BusinessDescription,ProductDescription,ProductSourceMap,updateBy,updateTimestamp FROM remote.dbview.REMOTE_CUSTOMER


Also
LOCAL_CUSTOMER  also does not have any triggers
Recovery Model of the database is Simple

Can you tell me how i can do SSIS  for parallel processing?

thanks.
0
 
yechanCommented:
In SSIS, for example, you could have two "Execute SQL" tasks and each task would have a query that would look something like the following:

DECLARE @myCount AS INT =( SELECT COUNT(*) FROM REMOTE_CUSTOMER );

WITH myCTE as
(
   SELECT 	
      col1, 
      col2, 
      etc,
      rn = ROW_NUMBER() OVER (ORDER BY (SELECT 0))
FROM 
     REMOTE_CUSTOMER
)

INSERT INTO LOCAL_CUSTOMER
SELECT  * FROM mycte
WHERE rn BETWEEN 1  AND 0.5 * @myCount
--WHERE rn BETWEEN 0.5 * @myCount AND @myCount  query to be used in the second SSIS task

Open in new window

0
 
royjaydAuthor Commented:
ok thanks

So is the above sql being run by multiple threads ?
Can you explain a little what the sql does?
0
 
yechanCommented:
What the SQL code does is create a  Common Table Expression (CTE for short) that selects all REMOTE_CUSTOMER records along with a row number using the ROW_NUMBER() function.

After that, the code inserts the first 50% of the records in the myCTE table into the LOCAL_CUSTOMER table.  You can execute this logic by inserting this SQL code into a "Execute SQL Task".
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

  • 4
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now