Solved

Synchronizing tables between AS/400 and MS SQL 2000

Posted on 2004-10-06
6
459 Views
Last Modified: 2012-06-27
Here's my dilemma.  I have a live production database running on an AS/400.  We use an MS SQL 2000 database as a warehouse for that same data so that we can run our reports off of the SQL2000 instead of taxing our aging AS/400.  We probably tax the AS/400 more the way we do it than if we could somehow just run our reports straight from the horse's mouth.

Our vendor has a solution that uses triggers to automatically sync each transaction from the AS/400 to the MSSQL server so that corresponding tables on both servers are always in sync.  It doesn't work anymore for reasons I won't get into.  Their secondary solution (the overtaxing one) is to use DTS packages to copy the tables from AS/400 to SQL.

The tables currently work as such:  Drop table from SQL database, create new table, copy all data from corresponding AS/400 table.

I would love a way to synchronize rather than drop and recreate.  We're talking about 1.5 million rows twice a day here.  I have it all scheduled so it's pretty hands off...but the data is never realtime in this method.  Our vendor's solution requires lots of triggers to be placed on the AS/400 and a client application to run on the SQL server.  We no longer have the AS/400 expertise to do this kind of AS/400 database stuff.

Appending tables won't work, because rows are not just added, the are also changed, and not always in sequence...so if you just added the new rows you might miss changes made to previous stuff.  Is there a way to compare the two tables and only copy data from changed entries?  I can't think of one and I don't think it's possible without the AS/400 triggers...but you people are the experts so I thought I'd pose my question to you.

Sorry this got so wordy...I just don't like having to go back and clarify things a lot.
0
Comment
Question by:mslunecka
  • 3
  • 3
6 Comments
 
LVL 34

Expert Comment

by:arbert
Comment Utility
Ya, avoid the triggers--talk about overhead.

Do you have a natural key on the AS400 tables you can use to identify a row?  Are there any change dates on the AS400 data?
0
 
LVL 6

Author Comment

by:mslunecka
Comment Utility
Sadly the data on the AS400 is some of the most horrible I've ever worked with.  No primary keys, no natural keys...hell half the tables aren't even in first normal form.  You can see why I don't want to deal with the vendor on this.  They're the ones that designed this rather digusting database.  I inherited this sytem and I'm trying to make the most of it.  Automated DTS packages get the job done...but they just aren't elegant.
0
 
LVL 34

Accepted Solution

by:
arbert earned 500 total points
Comment Utility
Well, if you don't have a natural key, I don't see a way to not truncate and reload every time.  Are there fields that you can form a key out of once it reaches the SQL Server side so you can determine whether to do an update/insert?
0
PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

 
LVL 6

Author Comment

by:mslunecka
Comment Utility
I've worked on that before...but it seems like it would just be a huge waste of comparisons since I can only have the natural key on the SQL Server side.  And if I'm not comparing the entire database (or at least anything dating back a month or so to be safe) I could miss something.

Our AS/400 runs slow during peak usage...so I just can't justify doubling the number of database transactions on it with those triggers, plus network traffic on an already overtaxed network.  I think it's better to just inconvenience people for 5 minutes while I sync tables than to degrade performance all day long.

Maybe someone has a better solution, but I can't see one just now.
0
 
LVL 34

Expert Comment

by:arbert
Comment Utility
I would ditch the triggers, download the entire set of data at night into a staging table.  Use the "derived natural key" to perform Updates/Inserts.

It doesn't sound like, with the current application, it's going to get much better than that...
0
 
LVL 6

Author Comment

by:mslunecka
Comment Utility
I'm just going to accept this as a "you can't do that" answer and call it correct.  I can't change the tables on the AS/400...so I can't add a natural key.  I think causing a few minutes of slow down once or twice a day is ultimately a better solution than constantly increasing system overhead.  Especially since we're about to do an OS upgrade that will hurt system performance by a significant amount.
0

Featured Post

Control application downtime with dependency maps

Visualize the interdependencies between application components better with Applications Manager's automated application discovery and dependency mapping feature. Resolve performance issues faster by quickly isolating problematic components.

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
How to use SQL Transaction log to analyze high usage. 5 37
Full Text Search string 5 32
Need Counts 11 40
.htaccess file settings 4 31
I wrote this interesting script that really help me find jobs or procedures when working in a huge environment. I could I have written it as a Procedure but then I would have to have it on each machine or have a link to a server-related search that …
This article explains how to reset the password of the sa account on a Microsoft SQL Server.  The steps in this article work in SQL 2005, 2008, 2008 R2, 2012, 2014 and 2016.
Viewers will learn how to use the INSERT statement to insert data into their tables. It will also introduce the NULL statement, to show them what happens when no value is giving for any given column.
Viewers will learn how to use the SELECT statement in SQL to return specific rows and columns, with various degrees of sorting and limits in place.

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now