?
Solved

SSIS between SQL and Oracle

Posted on 2010-11-18
4
Medium Priority
?
673 Views
Last Modified: 2012-05-10
Hi,

I need to extract rows from a MSSQL 2008 table and insert them into an Oracle database (SAP). I tried using Linked Server, but each transactions took between 4-16 seconds which was, of course, way too much. So I'm creating a SSIS.

I retrieve rows using a OLE DB Source on SQL Server and mapped the colums on a OLE DB Destination on Oracle (using Microsoft OLE DB Provider for Oracle). So far, it's fine. But I need to update an insert status in SQL, either fail or success. How can I do that?

Each rows inserted successfully in Oracle need to have its status updated in SQL. SSIS will be schedule to run at every 2 minutes, so new rows will be added in the SQL table between time recordset is retreive by the OLE DB Source query and the insertion in Oracle.

Anyone can help me please?
thanks for your time and help
0
Comment
Question by:Dominic34
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
4 Comments
 
LVL 7

Expert Comment

by:rmm2001
ID: 34166623
Assuming that you are loading all of the rows at once and that they all succeed,..

Drag a sequence container to the canvas. Inside it put a data flow task. And then in there put your sql-> oracle logic. At success of the sequence container, drag an execute sql task (green line connecting the two) with an "UPDATE table SET Processed = 1 WHERE Processed IS NULL".

Doing this though you'll run into the problem if there are records being inserted into your source table at the same time you're trying to update - so potentially you could get records not being processed on the overlap.

To get around that I'd recommend creating two variables ("Now" and "lastupdated") and a ProcessedDate in your table.

At the end of your package, put the direct sql entry as "SELECT MAX(ProcessedDate) FROM yourtable" Selet "single row" as your output. On result set tab, select 0 as result set name and then on the drop down pick your LastUpdated variable name.

Then have the default value of "Now" be set to System::ContainerStartTime.

Go through your sequence container and send the rows from sql to oracle. Like before..At success of the sequence container, drag an execute sql task (green line connecting the two) with an "UPDATE table SET Processed = 1 WHERE ProcessedDate BETWEEN ? AND ?"

Set the first ? = User;:LastUpdated and the 2nd to User::Now

That will get your records between the time slot and allow you to process everything w/o overlaps.

One thing I'd recommend though - on your sql part - add in the "Processed" flag as part of your source query so you only grab the rows that aren't processed yet. so "SELECT stuff FROM table WHERE Processed IS NULL"
0
 

Author Comment

by:Dominic34
ID: 34166875
thanks a lot for your detailed reply.

Actually, in my Data Flow task, I have a OLE DB Source with the query:

SELECT '031' AS MANDT, NoBatch, NoLigne,
CAST(DATEPART(YEAR, GETDATE()) AS VARCHAR) + REPLACE(STR(DATEPART(MONTH, GETDATE()), 2, 0), ' ', '0') + REPLACE(STR(DATEPART(DAY, GETDATE()), 2, 0), ' ', '0') AS [Date],
REPLACE(STR(DATEPART(HOUR, GETDATE()), 2, 0), ' ', '0') + REPLACE(STR(DATEPART(MINUTE, GETDATE()), 2, 0), ' ', '0') + REPLACE(STR(DATEPART(SECOND, GETDATE()), 2, 0), ' ', '0') AS [Time],
' ' AS Statut
FROM dbo.TB_psapProductionSAP
WHERE Treated = 0

This query will return 0 to a few hundreds rows.

I have a OLE DB Destination on my Oracle connection and I mapped columns accordingly. Then I need to update a status on the source table in the SQL Server (TB_psapProductionSAP) wether the insert worked or failed, for each row. A row can be successfully inserted and the following row can fail. How can I manage that ?

thanks for your precious time
0
 
LVL 7

Accepted Solution

by:
rmm2001 earned 2000 total points
ID: 34167049
I'm sure there's a better way to do this..but the easy way would be to make the oledb destionation be a table or view access mode (that way it inserts the rows separately and not in bulk). Then on "Error Output" select "Redirect Row". This will redirect the entire row to the next transformation. Drag an oledb command under the oldeb destination and draw the red arrow to it. Then write in your update statement. "UPDATE TB_psapProductionSAP SET Status = 'Errored' WHERE ID  = ?" <-- That would require you to pass in your ID field to the source query ( it doesn't have to be mapped). You could also join on NoBatch & NoLinge  if those are unique fields. If they are it'd be "UPDATE TB_psapProductionSAP SET Status = 'Errored' where NoBatch = ? and NoLinge = ?" and you'd have to map the ?'s in the "column mapping" tab.

Now to record the success rows - I'd do what I posted first. Just update them to "Processed" where they're null. Which is taking the assumption that if they didn't error, they succeeded.

Have you taken into consideration with what would happen if the entire thing  catastrophically errored (yes I've gotten that exact error before) in the middle of execution and half of your rows are inserted and half aren't? If you restarted that it would lead to potential dupe entries. It's a rare case but one to think about.
0
 

Author Closing Comment

by:Dominic34
ID: 34167373
thanks a lot for your help, I'll try it immediatly!
0

Featured Post

Get real performance insights from real users

Key features:
- Total Pages Views and Load times
- Top Pages Viewed and Load Times
- Real Time Site Page Build Performance
- Users’ Browser and Platform Performance
- Geographic User Breakdown
- And more

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Data architecture is an important aspect in Software as a Service (SaaS) delivery model. This article is a study on the database of a single-tenant application that could be extended to support multiple tenants. The application is web-based develope…
How to leverage one TLS certificate to encrypt Microsoft SQL traffic and Remote Desktop Services, versus creating multiple tickets for the same server.
This is my first video review of Microsoft Bookings, I will be doing a part two with a bit more information, but wanted to get this out to you folks.
Add bar graphs to Access queries using Unicode block characters. Graphs appear on every record in the color you want. Give life to numbers. Hopes this gives you ideas on visualizing your data in new ways ~ Create a calculated field in a query: …

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question