as400 slowdown

Posted on 2012-08-16
Last Modified: 2012-09-21
I am pretty novice at as400 problems but need some help on this.  We have an iseries 520 and about 3 weeks ago when we run reports it is taking much longer than normal.  Our payroll manager has a payroll edit register that she runs and it usually takes about 7 minutes to complete.  It compiles all of the employees info for the payroll check run.  Now it is taking 50 minutes to run.  This just happened 3 weeks ago on a Wednesday when we run this job.  Before that it was fine for as long as I can remember.  ANything I can check?  I looked at wrkactjob and I can see that the job is taking up 20% of the cpu. There is also another job with the status of run called QRWTRSVR type is PJ and it is at 5.7% cpu.  It keeps going back and forth with a status of TIMA and RUN.
Question by:mkramer777
    LVL 34

    Expert Comment

    by:Gary Patterson
    Dramatically increased runtimes can have a LOT of causes, and it probably isn't going to be possible to diagnose this at a distance.

    First question:  What changed in the system 3 weeks ago?  

    Did you load a bunch of new data onto the system for some reason?  Are the new or updated programs running?  Was this program changed?  Were there any system tuning changes made?  Did you apply PTFs or vendor updates?

    You mention QRWTSVR.  This is a system prestart job that services IBM DRDA coming in over TCP/IP.  It shows usage when a remote system established a DRDA database connection to the AS/400

    Look at the job log from the long-running job.  Are there errors?  

    Are you running Collection Services to capture system performance data? (GO PERFORM, Option 2 - Status=Started?).  If so, and you have performance data from before the problem, it would be possible to compare and see what is different.

    Jobs run longer than usual for the following reasons:

    Errors or contention with other jobs (maybe something else is locking a file this one needs).

    Increased file sizes (maybe as the result of new business or maybe just the result of a program error).

    Increased competition for CPU, memory, I/O or other limited resources by new or changed programs.

    System tuning changes - someone moved memory to a different pool, or turned off auto-tune.

    Are any other users complaining of problems, or any other processes experiencing runtime issues?

    - Gary Patterson

    Author Comment

    It is only the payroll manager that is experiencing these problems.  Also I have noticed that when users sign on to the system (which defaults to send them to the our accounting software main menu) this is taking longer than it has before.  A month or so ago when a user would sign on it would bring them to the main menu of our software in about 2 seconds.  Now it takes about 10-15.
    LVL 34

    Accepted Solution

    No answer to "what changed" questions, so I can't give much guidance.
    I have a client that is a small AS/400 shop that starts to experience slowdowns about this time every year - actually it starts to get noticeable in September, and gets worse until the end of the year.  

    Running their year-end process fixes the problem and they are good for another 8-9 months.  They've decided to live with it since they are involved in a search for a new system.

    Anyway, in their case the problem is just due to fact that the application they use doesn't purge or archive transactional data until year-end, and in the last two quarters, the file sizes get a little big for optimum performance on their small system.  This particular application doesn't perform well when the transactional tables are too big (not uncommon).  

    File grown performance issues tend to sneak up on you - gradual degradation each day or week until one day a user starts to complain, and then you get a cascade of similar "slow" complaints.  Unless you do regular performance monitoring and compare back to a baseline, you probably won't know until it has become an "issue".

    Maybe that is what is happening here if nothing else has changed.

    Anyway, diagnosing performance problems can be complicated.  In general, you start wide and go narrow, using the following tools.  Some of these tools are not part of the base OS, so you may not have them.  

    In that case, you might want to engage a consultant that can take your data and analyze it for you.  You might want to do that anyway.  I've seen a lot of performance and database disasters that resulted from inexperienced technicians trying to "fix" performance problems by changing something they saw on Google.

    1) Performance Collector / Performance Tools allow you to look at overall system performance and trends, and identify timeframes with poor performance.  There are also Navigator and iDoctor interface that allows you to view performance data in a variety of graphical formats.

    2) iDoctor: Job Watcher allows you to zero in on individual jobs and determine how those jobs are spending their time, and held determine which specific programs are responsible for delays

    3) Performance Explorer: PEX is a program profiling and trace tool that allows you to profile the entire system, individual jobs, or individual programs of modules and determine down to the line level of code where they are spending their time.

    I'll warn you, there is a significant learning curve on all of these tools, and this is a big topic.  Here are some useful links:

    Lots of additional version-specific resources in the Information Center for your OS release:

    Search for the tools above, or just "performance".

    - Gary Patterson

    Check out my EE profile for more AS/400 Performance resources:
    LVL 26

    Expert Comment

    I don't suppose you have both a current joblog and an old joblog from before the slowdown? Timestamps in joblogs can help narrow things down significantly sometimes. (That question is a real shot-in-the-dark.)

    The QRWTSRVR job is interesting. There should be no reason for it to take up any significant CPU at all unless there are active DRDA/DDM requests being serviced. At 5.7%, I'd be noticing.

    The 20% for your job is also interesting. That's a major percentage, at least double what I'd expect for a "payroll" related job.

    The point is in how those figures were obtained.

    When you look next time, run WRKACTJOB at least a few seconds after your payroll job starts. Then press <F10> to ensure that all statistics are specifically from the time WRKACTJOB started. Then wait 5 or 10  minutes and press <F5>.

    Then check the percentages.

    You probably should not see 20% unless the interval was too short. I'd like to see a more certain figure. That much CPU would be strange.


    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    What Should I Do With This Threat Intelligence?

    Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

    These days socially coordinated efforts have turned into a critical requirement for enterprises.
    Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
    Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…
    Here's a very brief overview of the methods PRTG Network Monitor ( offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…

    761 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    11 Experts available now in Live!

    Get 1:1 Help Now