Advertisement

01.28.2008 at 08:56AM PST, ID: 23116771
[x]
Attachment Details
[x]
The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

  • The Grade of the Solution
  • The Zone Rank of the Expert Providing the Solution
  • The Number of Author and Expert Comments
  • The Number of Experts Contributing
  • The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

8.0

Error Handling in ETL, using SQL LOADER and PL/SQL

Asked by mpaladugu in OLAP, Oracle 10.x

Tags: ,

Hi,

I need to design an error handling framework for an ETL engine which uses SQL Loader to load data to staging tables, and uses PL/SQL procedures to load data into the warehouse, and java interaces which will call sql loader scripts and stored Procedures.

I load only deltas into the warehouse every day, by comparing the record check sums against the check sums which i saved in the previous version of load,and i use MULTI TABLE insert statements to load data into warehouse schema.

Now what i want to know is, what would be the best approach for handling, DATA ERRORS and Business Rule Validations.
i am also attaching a sample procedure i use to load data....
Help arreciated..
Start Free Trial
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
reate or replace PROCEDURE                   SP_LOAD_Financial_Consultants (in_version IN NUMBER,   in_source_id IN VARCHAR2) AS
 
/*
  Functions :
    1. Check for deltas using Checksum
    2. Insert into Advisor History tables all the new advisors and changes.
    3. Insert into event table changed record id, along with old and new versions
 
  Pending issues
  -- Include Exception handling
*/
 
  v_fc_prefix constant VARCHAR2(20) := mas.advisor_prefix;
  v_fc_version_old NUMBER;
  v_fc_version_new NUMBER := in_version;
  v_fc_source VARCHAR2(20) := in_source_id;
BEGIN
 
/* LOGIC:
    Outer join Staging and Materialized tables to retrive 
    all the records that mathed External IDs but did not match check sums 
    along with their META IDs and also the records which did not match the
    External IDS with META IDs Null, and generate META IDs at the time of insertion.
    Records with Meta ID value Null are considered New records and the rest are changed ones. 
*/
  INSERT FIRST
    WHEN meta_fc_id IS NULL THEN
-- conditional insert if the advior is a new advisor also generate new meta ID
      INTO financial_consultant(version,   source_id,   meta_fc_id,   ext_fc_id,   fc_number,   fc_name,   fc_firm_name,   fc_branch_code,   fc_status,   phone_no,   addr1,   addr2,   addr3,   addr4,   city,   state,   country,   zip_code,   email,   fax_no,   checksum)
      VALUES(v_fc_version_new,   v_fc_source, v_fc_prefix||advisor_seq.nextval,   ext_fc_id,   fc_num,   fc_name,   firm_name,   branch_code,   fc_status,   phone_no,   addr1,   addr2,   addr3,   addr4,   city,   state,   country,   zip_code,   email,   fax_no,   stg_cksum)
 
    WHEN meta_fc_id IS NOT NULL THEN
-- conditional insert for changed advisors into advsor and event table.
      INTO financial_consultant (version,   source_id,   meta_fc_id,   ext_fc_id,   fc_id,   fc_name,   fc_firm_name,   fc_branch_code,   fc_status,   phone_no,   addr1,   addr2,   addr3,   addr4,   city,   state,   country,   zip_code,   email,   fax_no,   checksum)
      VALUES(v_fc_version_new,   v_fc_source, meta_fc_id,   ext_fc_id,   fc_num,   fc_name,   firm_name,   branch_code,   fc_status,   phone_no,   addr1,   addr2,   addr3,   addr4,   city,   state,   country,   zip_code,   email,   fax_no,   stg_cksum)    
      
      INTO fc_event(old_version,   new_version,   meta_fc_id,   entity)
      VALUES(old_version,  v_fc_version_new,   meta_fc_id,   'FIN_CONSULTANT') 
-- query to fetch the deltas required to insert (only the changed and new ones from staging)
      WITH fc AS
        (
          SELECT mv.version old_version, mv.meta_fc_id, stg.ext_fc_id, stg.fc_status, stg.firm_name,
             stg.fc_name, stg.phone_no, stg.addr1, stg.addr2,   stg.addr3,
             stg.addr4, stg.city, stg.state, stg.country,  stg.zip_code,
             stg.branch_code, stg.fc_num, stg.email, stg.fax_no,
             checksum(stg.ext_fc_id || stg.fc_status || stg.firm_name || stg.fc_name || stg.phone_no || stg.addr1 || stg.addr2 || stg.addr3 || stg.addr4 || stg.city || stg.state || stg.country || stg.zip_code || stg.branch_code || stg.fc_num || stg.email || stg.fax_no) stg_cksum,
             nvl(mv.checksum,0) fd_cksum
           FROM fc_stg stg,
             financial_consultant_mv mv
           WHERE stg.ext_fc_id = mv.ext_fc_id(+)
         )
      SELECT *
      FROM FC
      WHERE fc.stg_cksum <> fc.fd_cksum;
-- inline FA table calculates the staging check sums, also retrives MV checkums into its columns.
EXCEPTION
WHEN no_data_found THEN
NULL;
 
END sp_load_financial_consultants;
 
Keywords: Error Handling in ETL, using SQL LOA…
 
Loading Advertisement...
 
[+][-]01.28.2008 at 10:14PM PST, ID: 20765764

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 7-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]01.29.2008 at 02:12PM PST, ID: 20772885

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 7-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]01.29.2008 at 06:46PM PST, ID: 20774243

View this solution now by starting your 7-day free trial. Setting up your free trial is quick, easy, and secure. We will return you to this solution, unlocked, when you're done.

 

About this solution

Zones: OLAP, Oracle 10.x
Tags: oracle, 10.2
Sign Up Now!
Solution Provided By: sujith80
Participating Experts: 1
Solution Grade: A
 
 
 
Loading Advertisement...
20080716-EE-VQP-32 / EE_QW_2_20070628