Data flow, Data warehousing and Technologies

can you say hadoop?  That is the big data solution of the year.

before you go all crazy, "big" data is really really big. Sqlexpress (the free version) handles up to 10gb databases. The real mssql server handles 10 times that, I'm not sure if it even has a limit.

with mssql start looking at ssrs sql server reporting services (which is now ssdt sql server data tools for mssql2012)
http://www.microsoft.com/en-us/sqlserver/solutions-technologies/business-intelligence.aspx

http://www.microsoft.com/en-us/bi/default.aspx

Can you give us some more information about your scope?

I have about 10 systems with over 5 PB worth of data from different databases.

I like to identify the best of breed BI technologies and have an overview to wield them, is there any certification I can go for?

Well, size alone isn't the big differentiator for "big data"

E.g. if all 5 PB is accounts data with a ton of history, then it can aggregate into fairly tight analytics.

If the 5 PB contains a wide diversity of data and relationships, then yes, big data methodologies are going to help.

As for courses / certifications...

I am a member of TWDI and refer to it  fairly often. It has training and certifications specific to data warehousing and related : http://tdwi.org/Home.aspx

Some certifications are solutions specific such as : http://www.microsoft.com/learning/en/us/mcse-sql-business-intelligence.aspx and http://www.redbooks.ibm.com/redbooks/SG245747.html

The single most important aspect for certifications is being able to practise and do the exercises so access to machines and data is important (without compromising any live site and performance).

Good thread.

Can I safely say if i have many systems and each has a DB of its own.

I can connect them to my DW and run a BI tool from the DW to analyse the data?

So the DW should contain only the summary of the data in these underlying databases.

Correct?

Hi,
that depends on the DW/Reporting tool e.g.
SQL Server Tabular mode can connect directly with various sources (from CSV to Oracle...) and you then build your data model.
But normally you have a dedicated consolidated database where you import data from the source systems, do your logical mappings (eg in source 1 you have a country table having an int identity column and ISO Code 2 field, source 2 uses ISO Code 3 as identity column and source 3 has just long names - but your final DW db should have just one country table and all related data have to be remapped).
For performance reasons you should avoid to build up your DW directly on the transactional systems.
So your assumption is partly correct, but instead of simply connecting there will be a whole bunch of ETL processed in between.
Just my 2ct
Rainer

Good stuff.

How do I do all the ETL transformation from the raw data with all the ISOs?

That all depends on specifics of your source and DW. All the BI options have etl tools, or you can use external apps, even google refine for example (not that it's made for enterprise etl)

Ahh okay, good info. Thanks.