Making records available to other RPG programs without using force ratio 1

Hi

Records are entered in one program and they need to be available to other programs before the first program ends. Programmer has used OVRDBF with force ratio 1 and RPGLE block(*no) in the past to accomplish this.

Environment is v7r1 RPG/400 and RPGLE. Many programs involved in legacy application, so complete rewrite or redesign is not practical. No commitment control or journalling in use.

For performance, we want to remove the force ratio 1 but still need the app to work.

Thoughts?
LVL 13
_b_hAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Gary PattersonVP Technology / Senior Consultant Commented:
RPG automatically attempts to block writes when it can in order to improve performance.  Setting the Force Ratio on the file disables record blocking.  Records are forced to disk after every single WRITE, in this case - making them immediately available to other programs, but at the cost of reduced overall performance - and in high-volume programs, this can dramatically reduce performance.

For interactive programs, this can create a problem like the one you describe - records can be buffered for a long time.  

Best practice there is to either do the override described, or use the FEOD opcode in the program to force a write to disk when desired (maybe before throwing a screen, for example, to keep rows from being help in the write buffer while the user is on a lunch break).  

For well-written batch programs, it usually isn't a problem, and batch processes should generally be allowed to do blocked writes, as long as there aren't long periods where they pause to wait for other processes to complete, or something like that.

So, if this is an interactive program, and it is possible to do a few WRITE operations before you really need to flush them to disk, maybe you can remove the override and just strategically position the FEOD operation in the program.  Should generally only matter in an interactive program that has a very large number of high-volume users.

If this is a batch program that has long pauses in it, or a service-style (BCI / never-ending program, whatever you want to call it) program, then setting a force ratio of 1 is often a bad idea.  Instead, just add a FEOD outside the end of loops where you write to this file (or anywhere there might be a long pause in execution):

DOW not done
prepareData
WRITE FILE1
ENDDO
FEOD FILE1

This allows you to buffer writes while in the loop, but then force any remaining rows in the buffer to disk once you drop out of the loop and go on to do other things.  

Tell us more about the specific program and why this is creating a performance issue and we can give you better advice.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
_b_hAuthor Commented:
Thanks for the quick reply, Gary!

It is from a high volume order entry system, so many interactive programs and users are involved in the entering and picking of orders.
We want to use journalling for replication but are concerned about the performance impact of force ratio 1 and journalling together.

Do records have to be forced to auxiliary storage before they are visible to other programs?
Gary PattersonVP Technology / Senior Consultant Commented:
In this case, yes.  Un-blocked RPG WRITE, FEOD, or Force Ratio on file = 1.

Not sure I understand the concern.  Journaling performance and force ratio don't have much impact on each other (especially true if you enable journal caching).  

Do you have a performance problem now?  

If so, please provide details during peak usage time for this application.  Not usually that difficult to asses journaling impact, even if you aren't set up to load test.  Just start journaling small groups of files and monitor performance of your critical applications as you increase the number of files journaled.

OS is very smart about when it does journaling work, and even smarter if you enable journal caching (when appropriate for your replication SLA's):

CRTJRN JRNCACHE(*YES)

If you plan to journal a lot of files for replication purposes that aren't currently journaled, there is definitely some planning that needs to be done.  More than I can reasonable cover here, though, since some considerations are specific to your environment.  More info you can provide - for example, do you have current performance data you can share here? - the better.

I do occasionally see problems with journaling performance, but usually that is in applications that perform commitment control;;

http://garyrpatterson.net/ibm-i-performance/
Become a CompTIA Certified Healthcare IT Tech

This course will help prep you to earn the CompTIA Healthcare IT Technician certification showing that you have the knowledge and skills needed to succeed in installing, managing, and troubleshooting IT systems in medical and clinical settings.

Gary PattersonVP Technology / Senior Consultant Commented:
In general, journaling one file shouldn't have any significant impact on system performance.  Adds a small amount of CPU overhead, and some disk I/O overhead, depending on volume of journaled data and journal setup.  

Enabling general-purpose journaling for the purposes of replication is a different matter, and typically involves doing a little capacity analysis before flipping the switch.  In a large, busy system, IO overhead can be significant, though system handles it well in even the largest busiest systems I work on.  That means looking at your current system performance characteristics, estimating the additional workload, and verifying that you have adequate headroom for the new work.  

Disk arm utilization is a key metric to watch when talking about journaling.  Look WRKDSKSTS during your peak utilization times.  Any arms consistently over 40%?  If so, large-scale journaling is more likely to have a noticeable impact on performance, and you'll probably need to consider adding disk arms to support the additional disk IO workload.

Lots of ways to minimize the performance impact of journaling:

Journal caching
Minimized journal entries
Journal after images instead of *BOTH.
Don't journal busy work files not required for recovery
Enable soft commit when using commitment control.

Sometimes we also modify high-volume programs in order to minimize journaling impact, for example by consolidating multiple update operations into one, or other IO-saving changes.
tliottaCommented:
...we want to remove the force ratio 1...

Why? It's what ensures the action that you are trying to accomplish.

If you want a record to appear in the file visible to other jobs when a program writes a record, it is by definition a "force ratio 1". I mean, there could be any number of alternatives that go by various method names, but they would all end up doing the same things under the covers. If one record is written, it either stays in a

The biggest factor is probably the cache in your disk controllers, since writes don't really go to "disk" nowadays.

...but are concerned about the performance impact of force ratio 1 and journalling together.

Open in new window


When journaling is activated, the force-ratio of the journal is (1), but the "1" write might include a big chunk of related data. There is no alternative other than perhaps obtaining/using journal caching. But again, the number of controllers (and/or arms) and the amount (size) of their caches is a major concern.

But is this an issue mostly of immediate (or quick) access by other jobs? Should that be a higher concern than overall performance? Or perhaps, should it be a higher concern than lost data if a system/hardware failure occurs?

Tom
_b_hAuthor Commented:
Thank you for your guidance! I will open a new question regarding performance if needed.
Barry
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
IBM System i

From novice to tech pro — start learning today.