Speed problem with AIX 5.3 and BBX Pro5 Application

I have a customer running a BBX Pro5 application that seems to run too slow for the machine they have.  The machine is an IBM P630 running AIX 5.3 with 4 Gig RAM and lots of disk.  The problem is that it seems to run slow with respect to disk access from the BBX application, yet seems super fast from the OS level.

Here are some stats:

# svmon -G
               size      inuse       free        pin    virtual
memory      1048576    1062012     -13436     121158     224251
pg space    2752512        474

               work       pers       clnt
pin          121158          0          0
in use       224251      42024     795737

Now the disk access (attached as a code snippet to display the columns better)  Note the Kbs/s for BBX access compared to OS access:

Output of sar -d (only a snippet of whenthe disks are acually busy), the first one is when the BBX pro5 application is accessing the disks. The second (at 10:56) is me doing some file copy opperations from the OS itself.   
09:18:22     device    %busy    avque    r+w/s    Kbs/s   avwait   avserv
09:18:48     hdisk2    100     57.3      287     1362    594.5     10.5
             hdisk0     97      0.4      241     1194      5.1      9.5
             hdisk3     84     23.7      228     1136    509.1     11.0
             hdisk1    100     58.4      332     1635    622.8      9.0
10:56:13     hdisk2    100      8.3      544    34673     65.8      5.9
             hdisk0      1      0.0        2       11      0.0      6.6
             hdisk3     82      0.0      253    16102      0.4      5.2
             hdisk1     97      2.1      491    31370     16.5      5.9

Open in new window

LVL 14
Who is Participating?
sjm_eeConnect With a Mentor Commented:
Oops! Forgot to ask for a "lspv" but I think we can guess that rootvg is on hdisk0. The first thing I notice about the "sar" output is that when doing file copy work the vg2 disks are busy but rootvg is quiet whereas in the application example rootvg is also busy. This is unusual - is that paging?

Very busy disks with low data rate looks like very random i/o - AIX will spot if you are doing stuff with sequential files and is usually very good at reading ahead.

What is also interesting is that /home has been changed to a jfs2 and it's big - much bigger than a usual /home. Is there application activity there also?
General: Has this application ever run acceptably on thjis system? If so, what changed? If not, do you have any expectations from other systems?

Specific: Is the BBX Pro/5 application running on top of Oracle? Is the data access to files or to raw devices? How many CPUs in the 630? ("prtconf"). Please post the output of "lsvg", "lsvg -o", "lsvg -o | lsvg -i -l".
mikelfritzAuthor Commented:
The application never ran as fast as it should have.  No oracle or raw device access.  

The application was moved to the P630 from an old F50 two years ago.  At that time the BBX needed to be upgraded to handle AIX 5.3,  the upgrade doubled the size of all of the data files.  The new P630 and the old F50 seem to run the application about the same - although since the BBX was upgraded it's not really an apples to apples comparison.

 From the OS level it is like night and day, the P630 runs circles around the F50.  I ran some tests using awk on some really big files on both systems back when we switched and the new server was orders of magnitude faster.

# prtconf
System Model: IBM,7028-6C4
Machine Serial Number: 0000000
Processor Type: PowerPC_POWER4
Number Of Processors: 4
Processor Clock Speed: 1000 MHz
CPU Type: 64-bit
Kernel Type: 32-bit
Memory Size: 4096 MB
Good Memory Size: 4096 MB
Platform Firmware level: RR021114
Firmware Version: IBM,RG021113_GA3_ALL
Console Login: enable
Auto Restart: true
Full Core: false
# lsvg -o
# lsvg -o | lsvg -i -l
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
loglv00             jfs2log    1     1     1    open/syncd    N/A
fslv00              jfs2       239   239   1    closed/syncd  /u1.old
fslv01              jfs2       239   239   1    closed/syncd  /u2.old
u1str               jfs2       240   240   3    open/syncd    /u1
u2str               jfs2       240   240   3    open/syncd    /u2
u3str               jfs2       240   240   3    open/syncd    /u3
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
hd5                 boot       1     1     1    closed/syncd  N/A
hd6                 paging     21    21    1    open/syncd    N/A
hd8                 jfslog     1     1     1    open/syncd    N/A
hd4                 jfs        1     1     1    open/syncd    /
hd2                 jfs        3     3     1    open/syncd    /usr
hd9var              jfs        2     2     1    open/syncd    /var
hd3                 jfs        1     1     1    open/syncd    /tmp
loglv01             jfs2log    1     1     1    open/syncd    N/A
hd10opt             jfs        1     1     1    open/syncd    /opt
lg_dumplv           sysdump    2     2     1    open/syncd    N/A
fslv02              jfs2       239   239   1    open/syncd    /home

Open in new window

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

mikelfritzAuthor Commented:
When I was doing the testing from the OS I was making a copy from /u2 to /u3, so the vg2 being busy would be expected.  I get similar results when I copy between vg's.

/home is where the application resides along with some of the data.  /u1 /u2 and /u3 each contain data files and they are very large.  /u2 has only 27 files that range from 200 Meg to 4.5 Gig.  I believe you are correct with the very random i/o.  
# lspv
hdisk0          00000000d3cb1293                    rootvg          active
hdisk1          00000000e7f4d4c4                    vg2             active
hdisk2          00000000e7fac13c                    vg2             active
hdisk3          00000000e7fac234                    vg2             active

Open in new window

mikelfritzAuthor Commented:
I'm going to leave this open for a little while longer to see if there is any other advice.
mikelfritzAuthor Commented:
Thanks for the input.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.