Solved

Loading rows from a view (no index)

Posted on 2013-06-19
7
236 Views
Last Modified: 2016-02-11
hi guys

we have a remote view from which we are pulling data into our local table.

The remote view is :REMOTE_CUSTOMER
and our local table is :LOCAL_CUSTOMER

This is what we are doing right now.

We first delete data from our local table
DELETE Top (50000) FROM LOCAL_CUSTOMER where Id = '209'

Then we do a select to pull the data from the view (the remote view does not have any indexes, can we assign a index to view? Also the remote view doesnt have a primary key)

SELECT FirstName,LastName,BusinessDescription,ProductDescription,ProductSourceMap,updateBy,updateTimestamp FROM remote.dbview.REMOTE_CUSTOMER

then we insert into our local table

INSERT INTO LOCAL_CUSTOMER (FirstName,LastName,BusinessDescription,ProductDescription,ProductSourceMap,updateBy,updateTimestamp,Id)
VALUES(:FirstName, :LastName, :BusinessDescription, :ProductDescription, :ProductSourceMap, :updateBy,:updateTimestamp,:Id)

There are over a million rows in the REMOTE_CUSTOMER view and this process takes around 30 minutes to get the data from the view and populate into our table.

Is there a way to improve the speed of this entire process?

thanks
0
Comment
Question by:royjayd
  • 4
  • 3
7 Comments
 
LVL 1

Expert Comment

by:yechan
ID: 39260305
If you have any indexes on the LOCAL_CUSTOMER table my suggestion is to drop those indexes, load your data, and then re-create those indexes.
0
 

Author Comment

by:royjayd
ID: 39265735
ok, i dropped the index, but it takes the same time to insert the rows into
our LOCAL_CUSTOMER table (around 30 minutes)
Any other advice other than dropping the indexes ?

Can i create multiple threads and do parallel processing in sql server 2008 to increase the speed?

thanks
0
 
LVL 1

Expert Comment

by:yechan
ID: 39265796
Do you have a WHERE clause against the REMOTE_CUSTOMER table/view?  If yes, then I think adding an index would be very beneficial.

Other things to check for is that the LOCAL_CUSTOMER table does not have any triggers.  Change the "Recovery Level" of the database to Simple.  

Using SSMS to create a multiple threads would be difficult unless you open several tabs =)

I think if you want to do some parallel processing, using SSIS is the way to go.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:royjayd
ID: 39265899
hi

actually there is no Where clause in the select statement to query the view.
its just a simple sql

SELECT FirstName,LastName,BusinessDescription,ProductDescription,ProductSourceMap,updateBy,updateTimestamp FROM remote.dbview.REMOTE_CUSTOMER


Also
LOCAL_CUSTOMER  also does not have any triggers
Recovery Model of the database is Simple

Can you tell me how i can do SSIS  for parallel processing?

thanks.
0
 
LVL 1

Accepted Solution

by:
yechan earned 500 total points
ID: 39266645
In SSIS, for example, you could have two "Execute SQL" tasks and each task would have a query that would look something like the following:

DECLARE @myCount AS INT =( SELECT COUNT(*) FROM REMOTE_CUSTOMER );

WITH myCTE as
(
   SELECT 	
      col1, 
      col2, 
      etc,
      rn = ROW_NUMBER() OVER (ORDER BY (SELECT 0))
FROM 
     REMOTE_CUSTOMER
)

INSERT INTO LOCAL_CUSTOMER
SELECT  * FROM mycte
WHERE rn BETWEEN 1  AND 0.5 * @myCount
--WHERE rn BETWEEN 0.5 * @myCount AND @myCount  query to be used in the second SSIS task

Open in new window

0
 

Author Comment

by:royjayd
ID: 39267024
ok thanks

So is the above sql being run by multiple threads ?
Can you explain a little what the sql does?
0
 
LVL 1

Expert Comment

by:yechan
ID: 39270986
What the SQL code does is create a  Common Table Expression (CTE for short) that selects all REMOTE_CUSTOMER records along with a row number using the ROW_NUMBER() function.

After that, the code inserts the first 50% of the records in the myCTE table into the LOCAL_CUSTOMER table.  You can execute this logic by inserting this SQL code into a "Execute SQL Task".
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

Lessons learned during ten years of interviewing for SQL Server Integration Services (SSIS) and other Extract-Transform-Load (ETL) contract roles and two years of staff manager interviewing contractors.
My client sends a request to me that they want me to load data, which will be returned by Web Service APIs, and do some transformation before importing to database. In this article, I will provide an approach to load data with Web Service Task and X…
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now