Transferring data from one system to another

Posted on 2014-07-22
Last Modified: 2014-08-22
I have recently been thrown into a situation where I'm dealing with financial data (batch processing environment) that is transferred on a daily basis from a Mainframe COBOL system to (after some intermediate steps including ETL) an Oracle system.  As I am not in the IT department, all I really have direct access to is the dollars the Mainframe systems people say they transferred and the dollars the Oracle systems people say they received.  Those dollars may include multiple batches.  As ridiculous as this may seem, whenever there is a problem (for example, there is a multi-day delay in receiving a particular day's work in Oracle), the way the Oracle people are going about trying to find the data (that came into Oracle late)  is by searching Oracle batch reports for a batch or group of batches whose dollars add up to (or closely add up to) the amount the Mainframe people say they transferred. The batches (batch numbers) I am referring to are on the Oracle side only.  They are not batch numbers that are transferred by the Mainframe and captured by Oracle.  So they are of little if any use in cross-referencing between the two systems.

My question is:    What identifying data should be being transferred by the mainframe to Oracle to enable the Oracle IT department to identify which mainframe data was attempted to be transferred, what data actually got into Oracle and what data didn't get in?   The entire problem is currently the inability to match data that was sent with data that was received (and also what data was attempted to be sent but failed).  Thank you.
Question by:dbfromnewjersey
LVL 16

Assisted Solution

by:Wasim Akram Shaik
Wasim Akram Shaik earned 125 total points
ID: 40211873
There is a broad question.. Overall you are transferring data between two systems.. but how the transmission takes place?? this does really matter to answer your question better.

Is it a file based mechanism... I don't know much of mainframes.. but oracle has all its data in only tables...!!

so if any thing is not there in table means, it has got stuck somewhere in transmission(failed or missed)

To overcome this type of problem probably you need to get the batch numbers from oracle itself, overall it should be like this

1. Mainframe System will ask for oracle to send out the batch number(oracle sequence may be, for suppose), oracle will generate batch number and will update it status as pending
2. Once you receive batch number from oracle , then use it as an acknowledgement and start sending data
3. Once Batch is Sent, Oracle will automatically see the batch number which has been sent and updates corresponding flag as received in Oracle

The steps which were posted above may seem to over burden the process a little bit, but it will help out in such cases where in you have dilemma/difference between systems.
LVL 37

Assisted Solution

by:Gerwin Jansen
Gerwin Jansen earned 125 total points
ID: 40211919
I would say transaction ID's - for each financial data 'record' to be transferred, a unique ID from the COBOL system should be transferred into the Oracle system, as a (source) reference.
LVL 29

Accepted Solution

MikeOM_DBA earned 125 total points
ID: 40214949
As a minimum, you need to control the batches with following numbers:

- Batch number
- Timestamp
- transaction (row) sequence# and/or identification
- Record (row) count per batch
- Sum total of amounts (DB/CR) per batch

At each step (stage) transferred, received, loaded, etc... you should verify the above numbers and reject batch if there is a difference.

Good luck!
LVL 48

Assisted Solution

PortletPaul earned 125 total points
ID: 40215634
There is also a need to understand the method of file handling (I'd be pretty confident this is file based, most M/F interfaces to non-M/F system work that way).

The initial file is probably placed in a fixed location. Where is that?
(nb: What happens if that location isn't available? there should be procedures for this)

Will the M/F produce a file even if there are no transactions to process?
What is the cycle? daily? is the file produced at the same time of day for each cycle?

What is the naming convention of the files? (this can be very important)

What is the file "format"? (e.g. csv?)

What does the recipient system do with the file? Does it get "moved"? to where?
(nb: What happens if that location isn't available? there should be procedures for this)

If the file gets "moved" is it moved according to status?
(e.g. to a "done" folder? to "reject" folder?)

How long are files retained in these locations?
What are the backup regimes?
Are the files archived?

Operations staff should have all of this documented. Where is that documentation?
(although getting operations staff to share documents can be a rare event)

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

How to Unravel a Tricky Query Introduction If you browse through the Oracle zones or any of the other database-related zones you'll come across some complicated solutions and sometimes you'll just have to wonder how anyone came up with them.  …
Using SQL Scripts we can save all the SQL queries as files that we use very frequently on our database later point of time. This is one of the feature present under SQL Workshop in Oracle Application Express.
Via a live example, show how to take different types of Oracle backups using RMAN.
This video shows how to Export data from an Oracle database using the Datapump Export Utility.  The corresponding Datapump Import utility is also discussed and demonstrated.

867 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

23 Experts available now in Live!

Get 1:1 Help Now