Solved

SSIS between SQL and Oracle

Posted on 2010-11-18
4
650 Views
Last Modified: 2012-05-10
Hi,

I need to extract rows from a MSSQL 2008 table and insert them into an Oracle database (SAP). I tried using Linked Server, but each transactions took between 4-16 seconds which was, of course, way too much. So I'm creating a SSIS.

I retrieve rows using a OLE DB Source on SQL Server and mapped the colums on a OLE DB Destination on Oracle (using Microsoft OLE DB Provider for Oracle). So far, it's fine. But I need to update an insert status in SQL, either fail or success. How can I do that?

Each rows inserted successfully in Oracle need to have its status updated in SQL. SSIS will be schedule to run at every 2 minutes, so new rows will be added in the SQL table between time recordset is retreive by the OLE DB Source query and the insertion in Oracle.

Anyone can help me please?
thanks for your time and help
0
Comment
Question by:Dominic34
  • 2
  • 2
4 Comments
 
LVL 7

Expert Comment

by:rmm2001
Comment Utility
Assuming that you are loading all of the rows at once and that they all succeed,..

Drag a sequence container to the canvas. Inside it put a data flow task. And then in there put your sql-> oracle logic. At success of the sequence container, drag an execute sql task (green line connecting the two) with an "UPDATE table SET Processed = 1 WHERE Processed IS NULL".

Doing this though you'll run into the problem if there are records being inserted into your source table at the same time you're trying to update - so potentially you could get records not being processed on the overlap.

To get around that I'd recommend creating two variables ("Now" and "lastupdated") and a ProcessedDate in your table.

At the end of your package, put the direct sql entry as "SELECT MAX(ProcessedDate) FROM yourtable" Selet "single row" as your output. On result set tab, select 0 as result set name and then on the drop down pick your LastUpdated variable name.

Then have the default value of "Now" be set to System::ContainerStartTime.

Go through your sequence container and send the rows from sql to oracle. Like before..At success of the sequence container, drag an execute sql task (green line connecting the two) with an "UPDATE table SET Processed = 1 WHERE ProcessedDate BETWEEN ? AND ?"

Set the first ? = User;:LastUpdated and the 2nd to User::Now

That will get your records between the time slot and allow you to process everything w/o overlaps.

One thing I'd recommend though - on your sql part - add in the "Processed" flag as part of your source query so you only grab the rows that aren't processed yet. so "SELECT stuff FROM table WHERE Processed IS NULL"
0
 

Author Comment

by:Dominic34
Comment Utility
thanks a lot for your detailed reply.

Actually, in my Data Flow task, I have a OLE DB Source with the query:

SELECT '031' AS MANDT, NoBatch, NoLigne,
CAST(DATEPART(YEAR, GETDATE()) AS VARCHAR) + REPLACE(STR(DATEPART(MONTH, GETDATE()), 2, 0), ' ', '0') + REPLACE(STR(DATEPART(DAY, GETDATE()), 2, 0), ' ', '0') AS [Date],
REPLACE(STR(DATEPART(HOUR, GETDATE()), 2, 0), ' ', '0') + REPLACE(STR(DATEPART(MINUTE, GETDATE()), 2, 0), ' ', '0') + REPLACE(STR(DATEPART(SECOND, GETDATE()), 2, 0), ' ', '0') AS [Time],
' ' AS Statut
FROM dbo.TB_psapProductionSAP
WHERE Treated = 0

This query will return 0 to a few hundreds rows.

I have a OLE DB Destination on my Oracle connection and I mapped columns accordingly. Then I need to update a status on the source table in the SQL Server (TB_psapProductionSAP) wether the insert worked or failed, for each row. A row can be successfully inserted and the following row can fail. How can I manage that ?

thanks for your precious time
0
 
LVL 7

Accepted Solution

by:
rmm2001 earned 500 total points
Comment Utility
I'm sure there's a better way to do this..but the easy way would be to make the oledb destionation be a table or view access mode (that way it inserts the rows separately and not in bulk). Then on "Error Output" select "Redirect Row". This will redirect the entire row to the next transformation. Drag an oledb command under the oldeb destination and draw the red arrow to it. Then write in your update statement. "UPDATE TB_psapProductionSAP SET Status = 'Errored' WHERE ID  = ?" <-- That would require you to pass in your ID field to the source query ( it doesn't have to be mapped). You could also join on NoBatch & NoLinge  if those are unique fields. If they are it'd be "UPDATE TB_psapProductionSAP SET Status = 'Errored' where NoBatch = ? and NoLinge = ?" and you'd have to map the ?'s in the "column mapping" tab.

Now to record the success rows - I'd do what I posted first. Just update them to "Processed" where they're null. Which is taking the assumption that if they didn't error, they succeeded.

Have you taken into consideration with what would happen if the entire thing  catastrophically errored (yes I've gotten that exact error before) in the middle of execution and half of your rows are inserted and half aren't? If you restarted that it would lead to potential dupe entries. It's a rare case but one to think about.
0
 

Author Closing Comment

by:Dominic34
Comment Utility
thanks a lot for your help, I'll try it immediatly!
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Suggested Solutions

There have been several questions about Large Transaction Log Files in SQL Server 2008, and how to get rid of them when disk space has become critical. This article will explain how to disable full recovery and implement simple recovery that carries…
Use this article to create a batch file to backup a Microsoft SQL Server database to a Windows folder.  The folder can be on the local hard drive or on a network share.  This batch file will query the SQL server to get the current date & time and wi…
When you create an app prototype with Adobe XD, you can insert system screens -- sharing or Control Center, for example -- with just a few clicks. This video shows you how. You can take the full course on Experts Exchange at http://bit.ly/XDcourse.
This video explains how to create simple products associated to Magento configurable product and offers fast way of their generation with Store Manager for Magento tool.

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now