?
Solved

Nightly Imports using DTS or Incremental Updating

Posted on 2004-08-11
6
Medium Priority
?
442 Views
Last Modified: 2008-03-06
I am trying to update a table using DTS.  Here is the setup

1. AS/400 Machine with DB/2
2. MS SQL 2000

Constraints.

I am actually updating invoice history, since querying against the as/400 takes too long, so I decided to move the history files over to the SQL Server to improve performance.  The history dates back quite a bit.

There are errors in the dates, ie. something could have been invoiced July 2006, (the year is wrong) ... As my original idea was just to select the max date on the SQL Server, then select all invoices past that date on the as400.

The invoice numbers are also erronous.

So, I am left with two choices.

1. Complete import of the table during off hours ( 2:00 am)
2. figure out some way of incremental updates. The best solution.

Can someone walk me through how to create/deploy a DTS package that can do this?  Or, if there is a better way than DTS please let me know.
0
Comment
Question by:2xhelix
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
6 Comments
 
LVL 34

Expert Comment

by:arbert
ID: 11772768
How much information are you talking about?  You said the invoice date can be wrong--is there an add date or a change date on the records?
0
 

Author Comment

by:2xhelix
ID: 11773947
Hi Arbert!

Once again, thanks for your help.
From what I can tell , the as400 always appends the data to the end of the table.  
There is no add date, change date, .. only invoice date =(

The information is approxmately 3 years worth of data .. approx 700,000 rows.
A complete import takes approx ~6 mins

I tried to
SELECT *
FROM DB01.TAITOR.TFITORDATA.ORIDPAB1
Where SCINV# NOT IN (SELECT SCINV# from ORIDPAB1)

Which to me, is selecting all invoice numbers that are inside the as/400 which are not inside the SQL Server.  This query took ~15 mins?

So I am not sure which is the best solution.  
0
 
LVL 34

Expert Comment

by:arbert
ID: 11774116
Well, to get the 6minutes down, you're probably better off to insert EVERY record into a local sql table and then apply your NOT EXISTS logic (don't use not in--query plan isn't quite as good).

If you can't guarantee the accuracy of the invoice number and invoice date, I really don't see a way around it.  Will the 700,000 grow much larger, or do they keep only a certain time period in this table?
0
Get real performance insights from real users

Key features:
- Total Pages Views and Load times
- Top Pages Viewed and Load Times
- Real Time Site Page Build Performance
- Users’ Browser and Platform Performance
- Geographic User Breakdown
- And more

 

Author Comment

by:2xhelix
ID: 11774613
700,000 will continue to grow over time.

To be quite honest, I am not really familiar with DTS and what it can provide.  
If I did a nightly import, that would require me to drop the table, and then import the table from as400?

since I believe the invoice history on the as/400 appends to the end of the table, and there are never any updates to any rows, is there some way I can just rowcount (as/400) - rowcount(sqlserver) .. then somehow start a cursor at that location and copy?

0
 
LVL 34

Accepted Solution

by:
arbert earned 1000 total points
ID: 11774763
You could, but you CAN'T be guaranteed the order of the rows when you get them.  Only by adding an ORDER BY to your select statement are you guaranteed order--might not match the original AS400 table...

I don't think I would drop the entire table--transfer the entire as400 table into a sql server temp table (only transfer columns that you need) then, use the code (like you have above) to figure out what records don't exist and do an insert from the temp sql table....You'll find it's probably faster to transfer every row and put the criteria on SQL Server after the fact--linked servers aren't that efficient with criteria.
0
 

Author Comment

by:2xhelix
ID: 11783480
Great, I just put together everything, and started to import/update several tables last nite, and it is working perfectly.  And the execution times are acceptable as well.

Thank's for all your help .. don't know where I would be in this project without you =)
0

Featured Post

Simplifying Server Workload Migrations

This use case outlines the migration challenges that organizations face and how the Acronis AnyData Engine supports physical-to-physical (P2P), physical-to-virtual (P2V), virtual to physical (V2P), and cross-virtual (V2V) migration scenarios to address these challenges.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

JSON is being used more and more, besides XML, and you surely wanted to parse the data out into SQL instead of doing it in some Javascript. The below function in SQL Server can do the job for you, returning a quick table with the parsed data.
Ever wondered why sometimes your SQL Server is slow or unresponsive with connections spiking up but by the time you go in, all is well? The following article will show you how to install and configure a SQL job that will send you email alerts includ…
Using examples as well as descriptions, and references to Books Online, show the documentation available for date manipulation functions and by using a select few of these functions, show how date based data can be manipulated with these functions.
Via a live example, show how to set up a backup for SQL Server using a Maintenance Plan and how to schedule the job into SQL Server Agent.
Suggested Courses

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question