Solved

Load bulk data using SSIS

Posted on 2010-08-16
3
534 Views
Last Modified: 2013-11-10
Hi Experts,

   I want to retrieve data from XML file (typically large data) and need to do so many calculations on that data and finally load into our data warehouse. Data will be very large(every etl run, 200000 rows). What is the best way to ETL this?

How to convert relational database to dimensional model?
0
Comment
Question by:rajvja
  • 2
3 Comments
 
LVL 30

Expert Comment

by:Reza Rad
ID: 33450947
Hi,
200,000 rows of data is not very large actually, and you can do any transformation as you need in data flow.
what to you mean by convert OLTP to OLAP? did you designed your data warehouse? did you created dimensions and fact tables?
if yes, you just need to transfer data, you can do this simply by any transformation as your requirements.
of course there are some special transformation for DataWarehouse ( like SCD - Slowly Change Dimension ) which can help you much in dimensions.

let me know what you mean exactly here? and what is the problem you faced in ETL from OLTP to OLAP? I think this way is better to pass this problem.

0
 
LVL 11

Author Comment

by:rajvja
ID: 33452380
HI,

  Thanks for the reply.
 
There is already a relational database. The data coming from external source is growing large and large.
It will be 2m rows per day. It is a bus ticket information.

The requirement is we need to create a dimensional model coz there are many reports to be developed(KPIs).

What are the steps to follow to create a dimensional model from existing relational model.

If the data is 2m rows, what is the best way to do ETL with rich performance?
0
 
LVL 30

Accepted Solution

by:
Reza Rad earned 500 total points
ID: 33452506
OK,
First and foremost, you need to read books and articles about designing a Data Warehouse. In fact there is no wizard tools to create an OLAP database from OLTP model.
for start I suggest these links:
Kimball books about data warehouse is great resource:
http://www.ralphkimball.com/html/books.html

also microsoft links can help you in this area too:
http://msdn.microsoft.com/en-us/library/aa902672%28SQL.80%29.aspx

Note that designing a Data warehouse is most important step for your OLAP solution.

and in the next step, there are ways to transfer huge amount of data. but this depends on you business.
for example this is one way:
http://siddhumehta.blogspot.com/2009/03/improve-performance-of-etl-ssis-load.html
also using SCD is one of steps you probably need in your ETL, you can deal with performance there too( because SCD is quite slow ).
all these ways can be helpful when your data warehouse designed appropriate. So go on Data Warehouse first.


0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Returning data in a different format 8 45
VB6 ListBox Question 4 30
Time Duration able to handle overflow of 24+ hours 5 39
SQL - SP needs a little help 9 19
Ever needed a SQL 2008 Database replicated/mirrored/log shipped on another server but you can't take the downtime inflicted by initial snapshot or disconnect while T-logs are restored or mirror applied? You can use SQL Server Initialize from Backup…
Ever wondered why sometimes your SQL Server is slow or unresponsive with connections spiking up but by the time you go in, all is well? The following article will show you how to install and configure a SQL job that will send you email alerts includ…
Via a live example, show how to shrink a transaction log file down to a reasonable size.
Viewers will learn how the fundamental information of how to create a table.

914 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now