Avatar of Michael
MichaelFlag for United States of America

asked on 

Better iSeries throughput - CALLs or SBMJOB to Single Thread JO Queue?

When processing thousands of IFS Text Files into DB tables, could there be a significant performance improvement if the process is changed from using two SBMJOB requests for each IFS Text file operation, to using two program CALLs in a MQ Get App job to complete processing of each IFS Text files?  We have a process designed to process SAP iDoc that were converted to "|" delimited text files and placed in the IBMi IFS file system via SSH/sFTP sessions.  MQ is used to send new IFS Text file header information to the IBMi server.  The MQ Queue Get App currently get one file header, and submits a job that evaluates the header name and determine the correct program for that file.  Then submits another job into a single thread job queue and this last job actual moves the file from one IFS directory to another IFS directoty, then finally copies the data it to a DB work table for the final process to generate the appropriate DB record(s).  Two new jobs for each IDoc Text file to be processed, sequential processing being almost ensure via single thread job queue in the last step.  We did a 200,000 iDoc benchmark, and I found that the 100th iDoc completed this process with 36 seconds of wait time in the single thread job queue.  1,000th iDoc - 6 minutes job queue wait time, 20,000th iDoc - 49 minutes job queue wait time, and at 200,000th iDoc's job queue wait time was 13 hours and 19 minutes or so.  I am believing that Job Queue (work) management overhead is the problem here, and if we let the MQ Get app perform the complete process in the IBMi server we will see better throughput.  WHAT DO YOU THINK?
* iSeriesSQL* AS400IBM System i

Avatar of undefined
Last Comment
Gary Patterson, CISSP
Avatar of Gary Patterson, CISSP
Gary Patterson, CISSP
Flag of United States of America image

Blurred text
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
Existing architecture is terribly inefficient.  If you gave this to me to implement, I'd so something like this:

Drop files to IFS, send MQ notice.
Create a pool of one or more prestart jobs that, in a single job:
Do a MQGet
Select correct processing program
Call processing program, passing file to process.  
Skip unnecessary file move if possible, or move if needed for archival purposes.
Modify processing programs to NOT output to work table to avoid unnecessary IO.  Instead, pass as parameters to final program that writes to DB.

If all file types write to the same set of tables, you may run into contention if you have multiple parallel processing jobs.  Try to segregate so that each job (or two) just inserts to one set of tables.  Monitor db contention on these jobs - until you see it climb, you can add more processing jobs until throughput is optimized.  Optimal mix depends on system size, competing workload, job volume, and processing overhead of applications involved, so experimentation and measurement are required to optimize (and make sure overall process is well behaved with otherworkloads).

Also, if sequence matters, as it sometimes does with iDocs like the SHPMNT06 and DLVRY05s I work with every day, you may need do take some additional steps to protect sequencing.

I'd need to know more about your specifics to offer more detailed advice, but this is familiar territory and I'm happy to help if I can.

- Gary

IBM System i
IBM System i

IBM i, formerly known as AS/400, is a proprietary, midrange, object-based operating and application system. It includes a relational database, a menu-driven interface, support for multiple users, block-oriented terminal support and printers, and supports security, communications, and web-based application servers including Websphere, Apache and Tomcat.

Top Experts
Get a personalized solution from industry experts
Ask the experts
Read over 600 more reviews


IBM logoIntel logoMicrosoft logoUbisoft logoSAP logo
Qualcomm logoCitrix Systems logoWorkday logoErnst & Young logo
High performer badgeUsers love us badge
LinkedIn logoFacebook logoX logoInstagram logoTikTok logoYouTube logo