Solved

Loading rows from a view (no index)

Posted on 2013-06-19
7
237 Views
Last Modified: 2016-02-11
hi guys

we have a remote view from which we are pulling data into our local table.

The remote view is :REMOTE_CUSTOMER
and our local table is :LOCAL_CUSTOMER

This is what we are doing right now.

We first delete data from our local table
DELETE Top (50000) FROM LOCAL_CUSTOMER where Id = '209'

Then we do a select to pull the data from the view (the remote view does not have any indexes, can we assign a index to view? Also the remote view doesnt have a primary key)

SELECT FirstName,LastName,BusinessDescription,ProductDescription,ProductSourceMap,updateBy,updateTimestamp FROM remote.dbview.REMOTE_CUSTOMER

then we insert into our local table

INSERT INTO LOCAL_CUSTOMER (FirstName,LastName,BusinessDescription,ProductDescription,ProductSourceMap,updateBy,updateTimestamp,Id)
VALUES(:FirstName, :LastName, :BusinessDescription, :ProductDescription, :ProductSourceMap, :updateBy,:updateTimestamp,:Id)

There are over a million rows in the REMOTE_CUSTOMER view and this process takes around 30 minutes to get the data from the view and populate into our table.

Is there a way to improve the speed of this entire process?

thanks
0
Comment
Question by:royjayd
  • 4
  • 3
7 Comments
 
LVL 1

Expert Comment

by:yechan
ID: 39260305
If you have any indexes on the LOCAL_CUSTOMER table my suggestion is to drop those indexes, load your data, and then re-create those indexes.
0
 

Author Comment

by:royjayd
ID: 39265735
ok, i dropped the index, but it takes the same time to insert the rows into
our LOCAL_CUSTOMER table (around 30 minutes)
Any other advice other than dropping the indexes ?

Can i create multiple threads and do parallel processing in sql server 2008 to increase the speed?

thanks
0
 
LVL 1

Expert Comment

by:yechan
ID: 39265796
Do you have a WHERE clause against the REMOTE_CUSTOMER table/view?  If yes, then I think adding an index would be very beneficial.

Other things to check for is that the LOCAL_CUSTOMER table does not have any triggers.  Change the "Recovery Level" of the database to Simple.  

Using SSMS to create a multiple threads would be difficult unless you open several tabs =)

I think if you want to do some parallel processing, using SSIS is the way to go.
0
Ransomware-A Revenue Bonanza for Service Providers

Ransomware – malware that gets on your customers’ computers, encrypts their data, and extorts a hefty ransom for the decryption keys – is a surging new threat.  The purpose of this eBook is to educate the reader about ransomware attacks.

 

Author Comment

by:royjayd
ID: 39265899
hi

actually there is no Where clause in the select statement to query the view.
its just a simple sql

SELECT FirstName,LastName,BusinessDescription,ProductDescription,ProductSourceMap,updateBy,updateTimestamp FROM remote.dbview.REMOTE_CUSTOMER


Also
LOCAL_CUSTOMER  also does not have any triggers
Recovery Model of the database is Simple

Can you tell me how i can do SSIS  for parallel processing?

thanks.
0
 
LVL 1

Accepted Solution

by:
yechan earned 500 total points
ID: 39266645
In SSIS, for example, you could have two "Execute SQL" tasks and each task would have a query that would look something like the following:

DECLARE @myCount AS INT =( SELECT COUNT(*) FROM REMOTE_CUSTOMER );

WITH myCTE as
(
   SELECT 	
      col1, 
      col2, 
      etc,
      rn = ROW_NUMBER() OVER (ORDER BY (SELECT 0))
FROM 
     REMOTE_CUSTOMER
)

INSERT INTO LOCAL_CUSTOMER
SELECT  * FROM mycte
WHERE rn BETWEEN 1  AND 0.5 * @myCount
--WHERE rn BETWEEN 0.5 * @myCount AND @myCount  query to be used in the second SSIS task

Open in new window

0
 

Author Comment

by:royjayd
ID: 39267024
ok thanks

So is the above sql being run by multiple threads ?
Can you explain a little what the sql does?
0
 
LVL 1

Expert Comment

by:yechan
ID: 39270986
What the SQL code does is create a  Common Table Expression (CTE for short) that selects all REMOTE_CUSTOMER records along with a row number using the ROW_NUMBER() function.

After that, the code inserts the first 50% of the records in the myCTE table into the LOCAL_CUSTOMER table.  You can execute this logic by inserting this SQL code into a "Execute SQL Task".
0

Featured Post

Zoho SalesIQ

Hassle-free live chat software re-imagined for business growth. 2 users, always free.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Error in query 3 52
Determine log file requirements 7 34
Implementing SQL Server Data Files in Azure 1 35
Generate Weekly Schedule 15 16
How to leverage one TLS certificate to encrypt Microsoft SQL traffic and Remote Desktop Services, versus creating multiple tickets for the same server.
I have a large data set and a SSIS package. How can I load this file in multi threading?
Internet Business Fax to Email Made Easy - With  eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, f…
Many functions in Excel can make decisions. The most simple of these is the IF function: it returns a value depending on whether a condition you describe is true or false. Once you get the hang of using the IF function, you will find it easier to us…

911 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now