Posted on 2010-11-20
Last Modified: 2013-11-10
We have a package where it copies about 100 million records each day from the OLTP to the OLAP enviornment.We are using OLDEDB Destination with fast load to do this.It works ok till 40 million and after that it slows down drastically and almost stalled at somepoint.
Is there anything that Im missing.
Currently I have row commit size to infinite and row per batch size to 1 million.
Question by:venk_r
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
LVL 23

Accepted Solution

Racim BOUDJAKDJI earned 500 total points
ID: 34179952
<<Currently I have row commit size to infinite and row per batch size to 1 million.>>
Reduce both to 10000.  Also, please post the INSERT and SELECT queries so that we can help further.  Finally make sure your tempdb is well sized and that you do not have too frequent extent.  hope this helps...
LVL 30

Expert Comment

by:Reza Rad
ID: 34180267
I strongly recommend to read this article carefully, and raise up performance of your data flow with it:
LVL 12

Expert Comment

ID: 34229093
How long and when is the insert running? Common problem could be:
1. Any other processes running on source/destination box at the same time? Backup perhaps?
2. Doeas destination DB have enough free space? Could be that after first 40 mil. rows, DB space increasing causes additional IO.
3. Network load is fine?

Author Closing Comment

ID: 34409466

Featured Post

Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

JSON is being used more and more, besides XML, and you surely wanted to parse the data out into SQL instead of doing it in some Javascript. The below function in SQL Server can do the job for you, returning a quick table with the parsed data.
The Delta outage: 650 cancelled flights, more than 1200 delayed flights, thousands of frustrated customers, tens of millions of dollars in damages – plus untold reputational damage to one of the world’s most trusted airlines. All due to a catastroph…
This video shows, step by step, how to configure Oracle Heterogeneous Services via the Generic Gateway Agent in order to make a connection from an Oracle session and access a remote SQL Server database table.
Via a live example combined with referencing Books Online, show some of the information that can be extracted from the Catalog Views in SQL Server.

738 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question