Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

SSIS between SQL and Oracle

Posted on 2010-11-18
4
Medium Priority
?
678 Views
Last Modified: 2012-05-10
Hi,

I need to extract rows from a MSSQL 2008 table and insert them into an Oracle database (SAP). I tried using Linked Server, but each transactions took between 4-16 seconds which was, of course, way too much. So I'm creating a SSIS.

I retrieve rows using a OLE DB Source on SQL Server and mapped the colums on a OLE DB Destination on Oracle (using Microsoft OLE DB Provider for Oracle). So far, it's fine. But I need to update an insert status in SQL, either fail or success. How can I do that?

Each rows inserted successfully in Oracle need to have its status updated in SQL. SSIS will be schedule to run at every 2 minutes, so new rows will be added in the SQL table between time recordset is retreive by the OLE DB Source query and the insertion in Oracle.

Anyone can help me please?
thanks for your time and help
0
Comment
Question by:Dominic34
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
4 Comments
 
LVL 7

Expert Comment

by:rmm2001
ID: 34166623
Assuming that you are loading all of the rows at once and that they all succeed,..

Drag a sequence container to the canvas. Inside it put a data flow task. And then in there put your sql-> oracle logic. At success of the sequence container, drag an execute sql task (green line connecting the two) with an "UPDATE table SET Processed = 1 WHERE Processed IS NULL".

Doing this though you'll run into the problem if there are records being inserted into your source table at the same time you're trying to update - so potentially you could get records not being processed on the overlap.

To get around that I'd recommend creating two variables ("Now" and "lastupdated") and a ProcessedDate in your table.

At the end of your package, put the direct sql entry as "SELECT MAX(ProcessedDate) FROM yourtable" Selet "single row" as your output. On result set tab, select 0 as result set name and then on the drop down pick your LastUpdated variable name.

Then have the default value of "Now" be set to System::ContainerStartTime.

Go through your sequence container and send the rows from sql to oracle. Like before..At success of the sequence container, drag an execute sql task (green line connecting the two) with an "UPDATE table SET Processed = 1 WHERE ProcessedDate BETWEEN ? AND ?"

Set the first ? = User;:LastUpdated and the 2nd to User::Now

That will get your records between the time slot and allow you to process everything w/o overlaps.

One thing I'd recommend though - on your sql part - add in the "Processed" flag as part of your source query so you only grab the rows that aren't processed yet. so "SELECT stuff FROM table WHERE Processed IS NULL"
0
 

Author Comment

by:Dominic34
ID: 34166875
thanks a lot for your detailed reply.

Actually, in my Data Flow task, I have a OLE DB Source with the query:

SELECT '031' AS MANDT, NoBatch, NoLigne,
CAST(DATEPART(YEAR, GETDATE()) AS VARCHAR) + REPLACE(STR(DATEPART(MONTH, GETDATE()), 2, 0), ' ', '0') + REPLACE(STR(DATEPART(DAY, GETDATE()), 2, 0), ' ', '0') AS [Date],
REPLACE(STR(DATEPART(HOUR, GETDATE()), 2, 0), ' ', '0') + REPLACE(STR(DATEPART(MINUTE, GETDATE()), 2, 0), ' ', '0') + REPLACE(STR(DATEPART(SECOND, GETDATE()), 2, 0), ' ', '0') AS [Time],
' ' AS Statut
FROM dbo.TB_psapProductionSAP
WHERE Treated = 0

This query will return 0 to a few hundreds rows.

I have a OLE DB Destination on my Oracle connection and I mapped columns accordingly. Then I need to update a status on the source table in the SQL Server (TB_psapProductionSAP) wether the insert worked or failed, for each row. A row can be successfully inserted and the following row can fail. How can I manage that ?

thanks for your precious time
0
 
LVL 7

Accepted Solution

by:
rmm2001 earned 2000 total points
ID: 34167049
I'm sure there's a better way to do this..but the easy way would be to make the oledb destionation be a table or view access mode (that way it inserts the rows separately and not in bulk). Then on "Error Output" select "Redirect Row". This will redirect the entire row to the next transformation. Drag an oledb command under the oldeb destination and draw the red arrow to it. Then write in your update statement. "UPDATE TB_psapProductionSAP SET Status = 'Errored' WHERE ID  = ?" <-- That would require you to pass in your ID field to the source query ( it doesn't have to be mapped). You could also join on NoBatch & NoLinge  if those are unique fields. If they are it'd be "UPDATE TB_psapProductionSAP SET Status = 'Errored' where NoBatch = ? and NoLinge = ?" and you'd have to map the ?'s in the "column mapping" tab.

Now to record the success rows - I'd do what I posted first. Just update them to "Processed" where they're null. Which is taking the assumption that if they didn't error, they succeeded.

Have you taken into consideration with what would happen if the entire thing  catastrophically errored (yes I've gotten that exact error before) in the middle of execution and half of your rows are inserted and half aren't? If you restarted that it would lead to potential dupe entries. It's a rare case but one to think about.
0
 

Author Closing Comment

by:Dominic34
ID: 34167373
thanks a lot for your help, I'll try it immediatly!
0

Featured Post

Enroll in October's Free Course of the Month

Do you work with and analyze data? Enroll in October's Course of the Month for 7+ hours of SQL training, allowing you to quickly and efficiently store or retrieve data. It's free for Premium Members, Team Accounts, and Qualified Experts!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In SQL Server, when rows are selected from a table, does it retrieve data in the order in which it is inserted?  Many believe this is the case. Let us try to examine for ourselves with an example. To get started, use the following script, wh…
In this article I will describe the Detach & Attach method as one possible migration process and I will add the extra tasks needed for an upgrade when and where is applied so it will cover all.
Video by: ITPro.TV
In this episode Don builds upon the troubleshooting techniques by demonstrating how to properly monitor a vSphere deployment to detect problems before they occur. He begins the show using tools found within the vSphere suite as ends the show demonst…
In this video, Percona Solutions Engineer Barrett Chambers discusses some of the basic syntax differences between MySQL and MongoDB. To learn more check out our webinar on MongoDB administration for MySQL DBA: https://www.percona.com/resources/we…

636 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question