Solved

Load bulk data using SSIS

Posted on 2010-08-16
3
537 Views
Last Modified: 2013-11-10
Hi Experts,

   I want to retrieve data from XML file (typically large data) and need to do so many calculations on that data and finally load into our data warehouse. Data will be very large(every etl run, 200000 rows). What is the best way to ETL this?

How to convert relational database to dimensional model?
0
Comment
Question by:rajvja
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
3 Comments
 
LVL 30

Expert Comment

by:Reza Rad
ID: 33450947
Hi,
200,000 rows of data is not very large actually, and you can do any transformation as you need in data flow.
what to you mean by convert OLTP to OLAP? did you designed your data warehouse? did you created dimensions and fact tables?
if yes, you just need to transfer data, you can do this simply by any transformation as your requirements.
of course there are some special transformation for DataWarehouse ( like SCD - Slowly Change Dimension ) which can help you much in dimensions.

let me know what you mean exactly here? and what is the problem you faced in ETL from OLTP to OLAP? I think this way is better to pass this problem.

0
 
LVL 11

Author Comment

by:rajvja
ID: 33452380
HI,

  Thanks for the reply.
 
There is already a relational database. The data coming from external source is growing large and large.
It will be 2m rows per day. It is a bus ticket information.

The requirement is we need to create a dimensional model coz there are many reports to be developed(KPIs).

What are the steps to follow to create a dimensional model from existing relational model.

If the data is 2m rows, what is the best way to do ETL with rich performance?
0
 
LVL 30

Accepted Solution

by:
Reza Rad earned 500 total points
ID: 33452506
OK,
First and foremost, you need to read books and articles about designing a Data Warehouse. In fact there is no wizard tools to create an OLAP database from OLTP model.
for start I suggest these links:
Kimball books about data warehouse is great resource:
http://www.ralphkimball.com/html/books.html

also microsoft links can help you in this area too:
http://msdn.microsoft.com/en-us/library/aa902672%28SQL.80%29.aspx

Note that designing a Data warehouse is most important step for your OLAP solution.

and in the next step, there are ways to transfer huge amount of data. but this depends on you business.
for example this is one way:
http://siddhumehta.blogspot.com/2009/03/improve-performance-of-etl-ssis-load.html
also using SCD is one of steps you probably need in your ETL, you can deal with performance there too( because SCD is quite slow ).
all these ways can be helpful when your data warehouse designed appropriate. So go on Data Warehouse first.


0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

For both online and offline retail, the cross-channel business is the most recent pattern in the B2C trade space.
What if you have to shut down the entire Citrix infrastructure for hardware maintenance, software upgrades or "the unknown"? I developed this plan for "the unknown" and hope that it helps you as well. This article explains how to properly shut down …
Via a live example, show how to setup several different housekeeping processes for a SQL Server.
Using examples as well as descriptions, and references to Books Online, show the documentation available for datatypes, explain the available data types and show how data can be passed into and out of variables.

627 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question