Solved

SSIS between SQL and Oracle

Posted on 2010-11-18
4
653 Views
Last Modified: 2012-05-10
Hi,

I need to extract rows from a MSSQL 2008 table and insert them into an Oracle database (SAP). I tried using Linked Server, but each transactions took between 4-16 seconds which was, of course, way too much. So I'm creating a SSIS.

I retrieve rows using a OLE DB Source on SQL Server and mapped the colums on a OLE DB Destination on Oracle (using Microsoft OLE DB Provider for Oracle). So far, it's fine. But I need to update an insert status in SQL, either fail or success. How can I do that?

Each rows inserted successfully in Oracle need to have its status updated in SQL. SSIS will be schedule to run at every 2 minutes, so new rows will be added in the SQL table between time recordset is retreive by the OLE DB Source query and the insertion in Oracle.

Anyone can help me please?
thanks for your time and help
0
Comment
Question by:Dominic34
  • 2
  • 2
4 Comments
 
LVL 7

Expert Comment

by:rmm2001
ID: 34166623
Assuming that you are loading all of the rows at once and that they all succeed,..

Drag a sequence container to the canvas. Inside it put a data flow task. And then in there put your sql-> oracle logic. At success of the sequence container, drag an execute sql task (green line connecting the two) with an "UPDATE table SET Processed = 1 WHERE Processed IS NULL".

Doing this though you'll run into the problem if there are records being inserted into your source table at the same time you're trying to update - so potentially you could get records not being processed on the overlap.

To get around that I'd recommend creating two variables ("Now" and "lastupdated") and a ProcessedDate in your table.

At the end of your package, put the direct sql entry as "SELECT MAX(ProcessedDate) FROM yourtable" Selet "single row" as your output. On result set tab, select 0 as result set name and then on the drop down pick your LastUpdated variable name.

Then have the default value of "Now" be set to System::ContainerStartTime.

Go through your sequence container and send the rows from sql to oracle. Like before..At success of the sequence container, drag an execute sql task (green line connecting the two) with an "UPDATE table SET Processed = 1 WHERE ProcessedDate BETWEEN ? AND ?"

Set the first ? = User;:LastUpdated and the 2nd to User::Now

That will get your records between the time slot and allow you to process everything w/o overlaps.

One thing I'd recommend though - on your sql part - add in the "Processed" flag as part of your source query so you only grab the rows that aren't processed yet. so "SELECT stuff FROM table WHERE Processed IS NULL"
0
 

Author Comment

by:Dominic34
ID: 34166875
thanks a lot for your detailed reply.

Actually, in my Data Flow task, I have a OLE DB Source with the query:

SELECT '031' AS MANDT, NoBatch, NoLigne,
CAST(DATEPART(YEAR, GETDATE()) AS VARCHAR) + REPLACE(STR(DATEPART(MONTH, GETDATE()), 2, 0), ' ', '0') + REPLACE(STR(DATEPART(DAY, GETDATE()), 2, 0), ' ', '0') AS [Date],
REPLACE(STR(DATEPART(HOUR, GETDATE()), 2, 0), ' ', '0') + REPLACE(STR(DATEPART(MINUTE, GETDATE()), 2, 0), ' ', '0') + REPLACE(STR(DATEPART(SECOND, GETDATE()), 2, 0), ' ', '0') AS [Time],
' ' AS Statut
FROM dbo.TB_psapProductionSAP
WHERE Treated = 0

This query will return 0 to a few hundreds rows.

I have a OLE DB Destination on my Oracle connection and I mapped columns accordingly. Then I need to update a status on the source table in the SQL Server (TB_psapProductionSAP) wether the insert worked or failed, for each row. A row can be successfully inserted and the following row can fail. How can I manage that ?

thanks for your precious time
0
 
LVL 7

Accepted Solution

by:
rmm2001 earned 500 total points
ID: 34167049
I'm sure there's a better way to do this..but the easy way would be to make the oledb destionation be a table or view access mode (that way it inserts the rows separately and not in bulk). Then on "Error Output" select "Redirect Row". This will redirect the entire row to the next transformation. Drag an oledb command under the oldeb destination and draw the red arrow to it. Then write in your update statement. "UPDATE TB_psapProductionSAP SET Status = 'Errored' WHERE ID  = ?" <-- That would require you to pass in your ID field to the source query ( it doesn't have to be mapped). You could also join on NoBatch & NoLinge  if those are unique fields. If they are it'd be "UPDATE TB_psapProductionSAP SET Status = 'Errored' where NoBatch = ? and NoLinge = ?" and you'd have to map the ?'s in the "column mapping" tab.

Now to record the success rows - I'd do what I posted first. Just update them to "Processed" where they're null. Which is taking the assumption that if they didn't error, they succeeded.

Have you taken into consideration with what would happen if the entire thing  catastrophically errored (yes I've gotten that exact error before) in the middle of execution and half of your rows are inserted and half aren't? If you restarted that it would lead to potential dupe entries. It's a rare case but one to think about.
0
 

Author Closing Comment

by:Dominic34
ID: 34167373
thanks a lot for your help, I'll try it immediatly!
0

Featured Post

NAS Cloud Backup Strategies

This article explains backup scenarios when using network storage. We review the so-called “3-2-1 strategy” and summarize the methods you can use to send NAS data to the cloud

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this article I will describe the Detach & Attach method as one possible migration process and I will add the extra tasks needed for an upgrade when and where is applied so it will cover all.
This article explains how to reset the password of the sa account on a Microsoft SQL Server.  The steps in this article work in SQL 2005, 2008, 2008 R2, 2012, 2014 and 2016.
This Micro Tutorial hows how you can integrate  Mac OSX to a Windows Active Directory Domain. Apple has made it easy to allow users to bind their macs to a windows domain with relative ease. The following video show how to bind OSX Mavericks to …
Windows 10 is mostly good. However the one thing that annoys me is how many clicks you have to do to dial a VPN connection. You have to go to settings from the start menu, (2 clicks), Network and Internet (1 click), Click VPN (another click) then fi…

863 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now