Solved

Load bulk data using SSIS

Posted on 2010-08-16
3
533 Views
Last Modified: 2013-11-10
Hi Experts,

   I want to retrieve data from XML file (typically large data) and need to do so many calculations on that data and finally load into our data warehouse. Data will be very large(every etl run, 200000 rows). What is the best way to ETL this?

How to convert relational database to dimensional model?
0
Comment
Question by:rajvja
  • 2
3 Comments
 
LVL 30

Expert Comment

by:Reza Rad
Comment Utility
Hi,
200,000 rows of data is not very large actually, and you can do any transformation as you need in data flow.
what to you mean by convert OLTP to OLAP? did you designed your data warehouse? did you created dimensions and fact tables?
if yes, you just need to transfer data, you can do this simply by any transformation as your requirements.
of course there are some special transformation for DataWarehouse ( like SCD - Slowly Change Dimension ) which can help you much in dimensions.

let me know what you mean exactly here? and what is the problem you faced in ETL from OLTP to OLAP? I think this way is better to pass this problem.

0
 
LVL 11

Author Comment

by:rajvja
Comment Utility
HI,

  Thanks for the reply.
 
There is already a relational database. The data coming from external source is growing large and large.
It will be 2m rows per day. It is a bus ticket information.

The requirement is we need to create a dimensional model coz there are many reports to be developed(KPIs).

What are the steps to follow to create a dimensional model from existing relational model.

If the data is 2m rows, what is the best way to do ETL with rich performance?
0
 
LVL 30

Accepted Solution

by:
Reza Rad earned 500 total points
Comment Utility
OK,
First and foremost, you need to read books and articles about designing a Data Warehouse. In fact there is no wizard tools to create an OLAP database from OLTP model.
for start I suggest these links:
Kimball books about data warehouse is great resource:
http://www.ralphkimball.com/html/books.html

also microsoft links can help you in this area too:
http://msdn.microsoft.com/en-us/library/aa902672%28SQL.80%29.aspx

Note that designing a Data warehouse is most important step for your OLAP solution.

and in the next step, there are ways to transfer huge amount of data. but this depends on you business.
for example this is one way:
http://siddhumehta.blogspot.com/2009/03/improve-performance-of-etl-ssis-load.html
also using SCD is one of steps you probably need in your ETL, you can deal with performance there too( because SCD is quite slow ).
all these ways can be helpful when your data warehouse designed appropriate. So go on Data Warehouse first.


0

Featured Post

What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

Join & Write a Comment

Suggested Solutions

In this article—a derivative of my DaytaBase.org blog post (http://daytabase.org/2011/06/18/what-week-is-it/)—I will explore a few different perspectives on which week today's date falls within using Microsoft SQL Server. First, to frame this stu…
Slowly Changing Dimension Transformation component in data task flow is very useful for us to manage and control how data changes in SSIS.
Familiarize people with the process of retrieving data from SQL Server using an Access pass-thru query. Microsoft Access is a very powerful client/server development tool. One of the ways that you can retrieve data from a SQL Server is by using a pa…
Using examples as well as descriptions, and references to Books Online, show the different Recovery Models available in SQL Server and explain, as well as show how full, differential and transaction log backups are performed

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now