Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Loading rows from a view (no index)

Posted on 2013-06-19
7
Medium Priority
?
243 Views
Last Modified: 2016-02-11
hi guys

we have a remote view from which we are pulling data into our local table.

The remote view is :REMOTE_CUSTOMER
and our local table is :LOCAL_CUSTOMER

This is what we are doing right now.

We first delete data from our local table
DELETE Top (50000) FROM LOCAL_CUSTOMER where Id = '209'

Then we do a select to pull the data from the view (the remote view does not have any indexes, can we assign a index to view? Also the remote view doesnt have a primary key)

SELECT FirstName,LastName,BusinessDescription,ProductDescription,ProductSourceMap,updateBy,updateTimestamp FROM remote.dbview.REMOTE_CUSTOMER

then we insert into our local table

INSERT INTO LOCAL_CUSTOMER (FirstName,LastName,BusinessDescription,ProductDescription,ProductSourceMap,updateBy,updateTimestamp,Id)
VALUES(:FirstName, :LastName, :BusinessDescription, :ProductDescription, :ProductSourceMap, :updateBy,:updateTimestamp,:Id)

There are over a million rows in the REMOTE_CUSTOMER view and this process takes around 30 minutes to get the data from the view and populate into our table.

Is there a way to improve the speed of this entire process?

thanks
0
Comment
Question by:royjayd
  • 4
  • 3
7 Comments
 
LVL 1

Expert Comment

by:yechan
ID: 39260305
If you have any indexes on the LOCAL_CUSTOMER table my suggestion is to drop those indexes, load your data, and then re-create those indexes.
0
 

Author Comment

by:royjayd
ID: 39265735
ok, i dropped the index, but it takes the same time to insert the rows into
our LOCAL_CUSTOMER table (around 30 minutes)
Any other advice other than dropping the indexes ?

Can i create multiple threads and do parallel processing in sql server 2008 to increase the speed?

thanks
0
 
LVL 1

Expert Comment

by:yechan
ID: 39265796
Do you have a WHERE clause against the REMOTE_CUSTOMER table/view?  If yes, then I think adding an index would be very beneficial.

Other things to check for is that the LOCAL_CUSTOMER table does not have any triggers.  Change the "Recovery Level" of the database to Simple.  

Using SSMS to create a multiple threads would be difficult unless you open several tabs =)

I think if you want to do some parallel processing, using SSIS is the way to go.
0
Nothing ever in the clear!

This technical paper will help you implement VMware’s VM encryption as well as implement Veeam encryption which together will achieve the nothing ever in the clear goal. If a bad guy steals VMs, backups or traffic they get nothing.

 

Author Comment

by:royjayd
ID: 39265899
hi

actually there is no Where clause in the select statement to query the view.
its just a simple sql

SELECT FirstName,LastName,BusinessDescription,ProductDescription,ProductSourceMap,updateBy,updateTimestamp FROM remote.dbview.REMOTE_CUSTOMER


Also
LOCAL_CUSTOMER  also does not have any triggers
Recovery Model of the database is Simple

Can you tell me how i can do SSIS  for parallel processing?

thanks.
0
 
LVL 1

Accepted Solution

by:
yechan earned 2000 total points
ID: 39266645
In SSIS, for example, you could have two "Execute SQL" tasks and each task would have a query that would look something like the following:

DECLARE @myCount AS INT =( SELECT COUNT(*) FROM REMOTE_CUSTOMER );

WITH myCTE as
(
   SELECT 	
      col1, 
      col2, 
      etc,
      rn = ROW_NUMBER() OVER (ORDER BY (SELECT 0))
FROM 
     REMOTE_CUSTOMER
)

INSERT INTO LOCAL_CUSTOMER
SELECT  * FROM mycte
WHERE rn BETWEEN 1  AND 0.5 * @myCount
--WHERE rn BETWEEN 0.5 * @myCount AND @myCount  query to be used in the second SSIS task

Open in new window

0
 

Author Comment

by:royjayd
ID: 39267024
ok thanks

So is the above sql being run by multiple threads ?
Can you explain a little what the sql does?
0
 
LVL 1

Expert Comment

by:yechan
ID: 39270986
What the SQL code does is create a  Common Table Expression (CTE for short) that selects all REMOTE_CUSTOMER records along with a row number using the ROW_NUMBER() function.

After that, the code inserts the first 50% of the records in the myCTE table into the LOCAL_CUSTOMER table.  You can execute this logic by inserting this SQL code into a "Execute SQL Task".
0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Occasionally there is a need to clean table columns, especially if you have inherited legacy data. There are obviously many ways to accomplish that, including elaborate UPDATE queries with anywhere from one to numerous REPLACE functions (even within…
My client sends a request to me that they want me to load data, which will be returned by Web Service APIs, and do some transformation before importing to database. In this article, I will provide an approach to load data with Web Service Task and X…
This lesson discusses how to use a Mainform + Subforms in Microsoft Access to find and enter data for payments on orders. The sample data comes from a custom shop that builds and sells movable storage structures that are delivered to your property. …
When cloud platforms entered the scene, users and companies jumped on board to take advantage of the many benefits, like the ability to work and connect with company information from various locations. What many didn't foresee was the increased risk…

926 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question