Load bulk data using SSIS

Hi Experts,

   I want to retrieve data from XML file (typically large data) and need to do so many calculations on that data and finally load into our data warehouse. Data will be very large(every etl run, 200000 rows). What is the best way to ETL this?

How to convert relational database to dimensional model?
LVL 11
rajvjaAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Reza RadConsultant, TrainerCommented:
Hi,
200,000 rows of data is not very large actually, and you can do any transformation as you need in data flow.
what to you mean by convert OLTP to OLAP? did you designed your data warehouse? did you created dimensions and fact tables?
if yes, you just need to transfer data, you can do this simply by any transformation as your requirements.
of course there are some special transformation for DataWarehouse ( like SCD - Slowly Change Dimension ) which can help you much in dimensions.

let me know what you mean exactly here? and what is the problem you faced in ETL from OLTP to OLAP? I think this way is better to pass this problem.

0
rajvjaAuthor Commented:
HI,

  Thanks for the reply.
 
There is already a relational database. The data coming from external source is growing large and large.
It will be 2m rows per day. It is a bus ticket information.

The requirement is we need to create a dimensional model coz there are many reports to be developed(KPIs).

What are the steps to follow to create a dimensional model from existing relational model.

If the data is 2m rows, what is the best way to do ETL with rich performance?
0
Reza RadConsultant, TrainerCommented:
OK,
First and foremost, you need to read books and articles about designing a Data Warehouse. In fact there is no wizard tools to create an OLAP database from OLTP model.
for start I suggest these links:
Kimball books about data warehouse is great resource:
http://www.ralphkimball.com/html/books.html

also microsoft links can help you in this area too:
http://msdn.microsoft.com/en-us/library/aa902672%28SQL.80%29.aspx

Note that designing a Data warehouse is most important step for your OLAP solution.

and in the next step, there are ways to transfer huge amount of data. but this depends on you business.
for example this is one way:
http://siddhumehta.blogspot.com/2009/03/improve-performance-of-etl-ssis-load.html
also using SCD is one of steps you probably need in your ETL, you can deal with performance there too( because SCD is quite slow ).
all these ways can be helpful when your data warehouse designed appropriate. So go on Data Warehouse first.


0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SQL Server

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.