Better iSeries throughput - CALLs or SBMJOB to Single Thread JO Queue?
When processing thousands of IFS Text Files into DB tables, could there be a significant performance improvement if the process is changed from using two SBMJOB requests for each IFS Text file operation, to using two program CALLs in a MQ Get App job to complete processing of each IFS Text files? We have a process designed to process SAP iDoc that were converted to "|" delimited text files and placed in the IBMi IFS file system via SSH/sFTP sessions. MQ is used to send new IFS Text file header information to the IBMi server. The MQ Queue Get App currently get one file header, and submits a job that evaluates the header name and determine the correct program for that file. Then submits another job into a single thread job queue and this last job actual moves the file from one IFS directory to another IFS directoty, then finally copies the data it to a DB work table for the final process to generate the appropriate DB record(s). Two new jobs for each IDoc Text file to be processed, sequential processing being almost ensure via single thread job queue in the last step. We did a 200,000 iDoc benchmark, and I found that the 100th iDoc completed this process with 36 seconds of wait time in the single thread job queue. 1,000th iDoc - 6 minutes job queue wait time, 20,000th iDoc - 49 minutes job queue wait time, and at 200,000th iDoc's job queue wait time was 13 hours and 19 minutes or so. I am believing that Job Queue (work) management overhead is the problem here, and if we let the MQ Get app perform the complete process in the IBMi server we will see better throughput. WHAT DO YOU THINK?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Drop files to IFS, send MQ notice.
Create a pool of one or more prestart jobs that, in a single job:
Do a MQGet
Select correct processing program
Call processing program, passing file to process.
Skip unnecessary file move if possible, or move if needed for archival purposes.
Modify processing programs to NOT output to work table to avoid unnecessary IO. Instead, pass as parameters to final program that writes to DB.
If all file types write to the same set of tables, you may run into contention if you have multiple parallel processing jobs. Try to segregate so that each job (or two) just inserts to one set of tables. Monitor db contention on these jobs - until you see it climb, you can add more processing jobs until throughput is optimized. Optimal mix depends on system size, competing workload, job volume, and processing overhead of applications involved, so experimentation and measurement are required to optimize (and make sure overall process is well behaved with otherworkloads).
Also, if sequence matters, as it sometimes does with iDocs like the SHPMNT06 and DLVRY05s I work with every day, you may need do take some additional steps to protect sequencing.
I'd need to know more about your specifics to offer more detailed advice, but this is familiar territory and I'm happy to help if I can.
- Gary