asked on

Data warehouse archiving

Hi,

We are planning to make a Datawarehouse. For this the RAW data can imported from Flat Files, OLTP, XML data files etc etc.....and processed using ETL and then inserted onto DW.
Now suppose I need to archive datawarewouse every 2 months. that means on archiving, data for the last two months will be MOVED from the original location to some other location.

Then how can we determine while ETL process that the particular data has been archived and should not be imported again.?

I think this is a general issue and should have a standard solution.

Please guide...!

Thx!
Swaps...

Mark Brady

If the data is being archived and saved as files, you could add to the filename with something like

$filename = "yourfilename.txt"; // whatever you name your file(s)
$archive_name = "a_".$filename;

now you can archive that file(s) and when you do your imports, don't import any file that begins with an "a_"

something along those lines would work.

SOLUTION

Raja Jegan R

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

ASKER CERTIFIED SOLUTION

Tim Humphries

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

k_swapnil

ASKER

ok...
Now suppose I archive 2008 data to one file (db) and 2009 to another and 2010 is in current data.

now if i want to generate a report in which I need data for last 3 years, how can i do this if I have only 2010 year data in current warehouse.?

Thx!
Swaps....

dportas

Typically you would keep an aggregated version of the data online for a longer period of time so that you could produce reports against it even after the base data had been archived.

SOLUTION

Raja Jegan R

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

k_swapnil

ASKER

Thx!