Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Load bulk data using SSIS

Posted on 2010-08-16
3
Medium Priority
?
538 Views
Last Modified: 2013-11-10
Hi Experts,

   I want to retrieve data from XML file (typically large data) and need to do so many calculations on that data and finally load into our data warehouse. Data will be very large(every etl run, 200000 rows). What is the best way to ETL this?

How to convert relational database to dimensional model?
0
Comment
Question by:rajvja
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
3 Comments
 
LVL 30

Expert Comment

by:Reza Rad
ID: 33450947
Hi,
200,000 rows of data is not very large actually, and you can do any transformation as you need in data flow.
what to you mean by convert OLTP to OLAP? did you designed your data warehouse? did you created dimensions and fact tables?
if yes, you just need to transfer data, you can do this simply by any transformation as your requirements.
of course there are some special transformation for DataWarehouse ( like SCD - Slowly Change Dimension ) which can help you much in dimensions.

let me know what you mean exactly here? and what is the problem you faced in ETL from OLTP to OLAP? I think this way is better to pass this problem.

0
 
LVL 11

Author Comment

by:rajvja
ID: 33452380
HI,

  Thanks for the reply.
 
There is already a relational database. The data coming from external source is growing large and large.
It will be 2m rows per day. It is a bus ticket information.

The requirement is we need to create a dimensional model coz there are many reports to be developed(KPIs).

What are the steps to follow to create a dimensional model from existing relational model.

If the data is 2m rows, what is the best way to do ETL with rich performance?
0
 
LVL 30

Accepted Solution

by:
Reza Rad earned 2000 total points
ID: 33452506
OK,
First and foremost, you need to read books and articles about designing a Data Warehouse. In fact there is no wizard tools to create an OLAP database from OLTP model.
for start I suggest these links:
Kimball books about data warehouse is great resource:
http://www.ralphkimball.com/html/books.html

also microsoft links can help you in this area too:
http://msdn.microsoft.com/en-us/library/aa902672%28SQL.80%29.aspx

Note that designing a Data warehouse is most important step for your OLAP solution.

and in the next step, there are ways to transfer huge amount of data. but this depends on you business.
for example this is one way:
http://siddhumehta.blogspot.com/2009/03/improve-performance-of-etl-ssis-load.html
also using SCD is one of steps you probably need in your ETL, you can deal with performance there too( because SCD is quite slow ).
all these ways can be helpful when your data warehouse designed appropriate. So go on Data Warehouse first.


0

Featured Post

Get your Disaster Recovery as a Service basics

Disaster Recovery as a Service is one go-to solution that revolutionizes DR planning. Implementing DRaaS could be an efficient process, easily accessible to non-DR experts. Learn about monitoring, testing, executing failovers and failbacks to ensure a "healthy" DR environment.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
What if you have to shut down the entire Citrix infrastructure for hardware maintenance, software upgrades or "the unknown"? I developed this plan for "the unknown" and hope that it helps you as well. This article explains how to properly shut down …
Familiarize people with the process of utilizing SQL Server functions from within Microsoft Access. Microsoft Access is a very powerful client/server development tool. One of the SQL Server objects that you can interact with from within Microsoft Ac…
This videos aims to give the viewer a basic demonstration of how a user can query current session information by using the SYS_CONTEXT function

660 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question