Speed problem with AIX 5.3 and BBX Pro5 Application

I have a customer running a BBX Pro5 application that seems to run too slow for the machine they have.  The machine is an IBM P630 running AIX 5.3 with 4 Gig RAM and lots of disk.  The problem is that it seems to run slow with respect to disk access from the BBX application, yet seems super fast from the OS level.

Here are some stats:

# svmon -G
               size      inuse       free        pin    virtual
memory      1048576    1062012     -13436     121158     224251
pg space    2752512        474

               work       pers       clnt
pin          121158          0          0
in use       224251      42024     795737

Now the disk access (attached as a code snippet to display the columns better)  Note the Kbs/s for BBX access compared to OS access:

Output of sar -d (only a snippet of whenthe disks are acually busy), the first one is when the BBX pro5 application is accessing the disks. The second (at 10:56) is me doing some file copy opperations from the OS itself.   
09:18:22     device    %busy    avque    r+w/s    Kbs/s   avwait   avserv
09:18:48     hdisk2    100     57.3      287     1362    594.5     10.5
             hdisk0     97      0.4      241     1194      5.1      9.5
             hdisk3     84     23.7      228     1136    509.1     11.0
             hdisk1    100     58.4      332     1635    622.8      9.0
10:56:13     hdisk2    100      8.3      544    34673     65.8      5.9
             hdisk0      1      0.0        2       11      0.0      6.6
             hdisk3     82      0.0      253    16102      0.4      5.2
             hdisk1     97      2.1      491    31370     16.5      5.9

Open in new window

LVL 14
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

General: Has this application ever run acceptably on thjis system? If so, what changed? If not, do you have any expectations from other systems?

Specific: Is the BBX Pro/5 application running on top of Oracle? Is the data access to files or to raw devices? How many CPUs in the 630? ("prtconf"). Please post the output of "lsvg", "lsvg -o", "lsvg -o | lsvg -i -l".
mikelfritzAuthor Commented:
The application never ran as fast as it should have.  No oracle or raw device access.  

The application was moved to the P630 from an old F50 two years ago.  At that time the BBX needed to be upgraded to handle AIX 5.3,  the upgrade doubled the size of all of the data files.  The new P630 and the old F50 seem to run the application about the same - although since the BBX was upgraded it's not really an apples to apples comparison.

 From the OS level it is like night and day, the P630 runs circles around the F50.  I ran some tests using awk on some really big files on both systems back when we switched and the new server was orders of magnitude faster.

# prtconf
System Model: IBM,7028-6C4
Machine Serial Number: 0000000
Processor Type: PowerPC_POWER4
Number Of Processors: 4
Processor Clock Speed: 1000 MHz
CPU Type: 64-bit
Kernel Type: 32-bit
Memory Size: 4096 MB
Good Memory Size: 4096 MB
Platform Firmware level: RR021114
Firmware Version: IBM,RG021113_GA3_ALL
Console Login: enable
Auto Restart: true
Full Core: false
# lsvg -o
# lsvg -o | lsvg -i -l
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
loglv00             jfs2log    1     1     1    open/syncd    N/A
fslv00              jfs2       239   239   1    closed/syncd  /u1.old
fslv01              jfs2       239   239   1    closed/syncd  /u2.old
u1str               jfs2       240   240   3    open/syncd    /u1
u2str               jfs2       240   240   3    open/syncd    /u2
u3str               jfs2       240   240   3    open/syncd    /u3
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
hd5                 boot       1     1     1    closed/syncd  N/A
hd6                 paging     21    21    1    open/syncd    N/A
hd8                 jfslog     1     1     1    open/syncd    N/A
hd4                 jfs        1     1     1    open/syncd    /
hd2                 jfs        3     3     1    open/syncd    /usr
hd9var              jfs        2     2     1    open/syncd    /var
hd3                 jfs        1     1     1    open/syncd    /tmp
loglv01             jfs2log    1     1     1    open/syncd    N/A
hd10opt             jfs        1     1     1    open/syncd    /opt
lg_dumplv           sysdump    2     2     1    open/syncd    N/A
fslv02              jfs2       239   239   1    open/syncd    /home

Open in new window

Oops! Forgot to ask for a "lspv" but I think we can guess that rootvg is on hdisk0. The first thing I notice about the "sar" output is that when doing file copy work the vg2 disks are busy but rootvg is quiet whereas in the application example rootvg is also busy. This is unusual - is that paging?

Very busy disks with low data rate looks like very random i/o - AIX will spot if you are doing stuff with sequential files and is usually very good at reading ahead.

What is also interesting is that /home has been changed to a jfs2 and it's big - much bigger than a usual /home. Is there application activity there also?

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Cloud Class® Course: Python 3 Fundamentals

This course will teach participants about installing and configuring Python, syntax, importing, statements, types, strings, booleans, files, lists, tuples, comprehensions, functions, and classes.

mikelfritzAuthor Commented:
When I was doing the testing from the OS I was making a copy from /u2 to /u3, so the vg2 being busy would be expected.  I get similar results when I copy between vg's.

/home is where the application resides along with some of the data.  /u1 /u2 and /u3 each contain data files and they are very large.  /u2 has only 27 files that range from 200 Meg to 4.5 Gig.  I believe you are correct with the very random i/o.  
# lspv
hdisk0          00000000d3cb1293                    rootvg          active
hdisk1          00000000e7f4d4c4                    vg2             active
hdisk2          00000000e7fac13c                    vg2             active
hdisk3          00000000e7fac234                    vg2             active

Open in new window

mikelfritzAuthor Commented:
I'm going to leave this open for a little while longer to see if there is any other advice.
mikelfritzAuthor Commented:
Thanks for the input.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Unix OS

From novice to tech pro — start learning today.