Speed problem with AIX 5.3 and BBX Pro5 Application

Posted on 2008-06-20
Last Modified: 2013-11-17
I have a customer running a BBX Pro5 application that seems to run too slow for the machine they have.  The machine is an IBM P630 running AIX 5.3 with 4 Gig RAM and lots of disk.  The problem is that it seems to run slow with respect to disk access from the BBX application, yet seems super fast from the OS level.

Here are some stats:

# svmon -G
               size      inuse       free        pin    virtual
memory      1048576    1062012     -13436     121158     224251
pg space    2752512        474

               work       pers       clnt
pin          121158          0          0
in use       224251      42024     795737

Now the disk access (attached as a code snippet to display the columns better)  Note the Kbs/s for BBX access compared to OS access:

Output of sar -d (only a snippet of whenthe disks are acually busy), the first one is when the BBX pro5 application is accessing the disks. The second (at 10:56) is me doing some file copy opperations from the OS itself.   
09:18:22     device    %busy    avque    r+w/s    Kbs/s   avwait   avserv
09:18:48     hdisk2    100     57.3      287     1362    594.5     10.5
             hdisk0     97      0.4      241     1194      5.1      9.5
             hdisk3     84     23.7      228     1136    509.1     11.0
             hdisk1    100     58.4      332     1635    622.8      9.0
10:56:13     hdisk2    100      8.3      544    34673     65.8      5.9
             hdisk0      1      0.0        2       11      0.0      6.6
             hdisk3     82      0.0      253    16102      0.4      5.2
             hdisk1     97      2.1      491    31370     16.5      5.9

Open in new window

Question by:mikelfritz
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 2
LVL 14

Expert Comment

ID: 21832543
General: Has this application ever run acceptably on thjis system? If so, what changed? If not, do you have any expectations from other systems?

Specific: Is the BBX Pro/5 application running on top of Oracle? Is the data access to files or to raw devices? How many CPUs in the 630? ("prtconf"). Please post the output of "lsvg", "lsvg -o", "lsvg -o | lsvg -i -l".
LVL 14

Author Comment

ID: 21832672
The application never ran as fast as it should have.  No oracle or raw device access.  

The application was moved to the P630 from an old F50 two years ago.  At that time the BBX needed to be upgraded to handle AIX 5.3,  the upgrade doubled the size of all of the data files.  The new P630 and the old F50 seem to run the application about the same - although since the BBX was upgraded it's not really an apples to apples comparison.

 From the OS level it is like night and day, the P630 runs circles around the F50.  I ran some tests using awk on some really big files on both systems back when we switched and the new server was orders of magnitude faster.

# prtconf
System Model: IBM,7028-6C4
Machine Serial Number: 0000000
Processor Type: PowerPC_POWER4
Number Of Processors: 4
Processor Clock Speed: 1000 MHz
CPU Type: 64-bit
Kernel Type: 32-bit
Memory Size: 4096 MB
Good Memory Size: 4096 MB
Platform Firmware level: RR021114
Firmware Version: IBM,RG021113_GA3_ALL
Console Login: enable
Auto Restart: true
Full Core: false
# lsvg -o
# lsvg -o | lsvg -i -l
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
loglv00             jfs2log    1     1     1    open/syncd    N/A
fslv00              jfs2       239   239   1    closed/syncd  /u1.old
fslv01              jfs2       239   239   1    closed/syncd  /u2.old
u1str               jfs2       240   240   3    open/syncd    /u1
u2str               jfs2       240   240   3    open/syncd    /u2
u3str               jfs2       240   240   3    open/syncd    /u3
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
hd5                 boot       1     1     1    closed/syncd  N/A
hd6                 paging     21    21    1    open/syncd    N/A
hd8                 jfslog     1     1     1    open/syncd    N/A
hd4                 jfs        1     1     1    open/syncd    /
hd2                 jfs        3     3     1    open/syncd    /usr
hd9var              jfs        2     2     1    open/syncd    /var
hd3                 jfs        1     1     1    open/syncd    /tmp
loglv01             jfs2log    1     1     1    open/syncd    N/A
hd10opt             jfs        1     1     1    open/syncd    /opt
lg_dumplv           sysdump    2     2     1    open/syncd    N/A
fslv02              jfs2       239   239   1    open/syncd    /home

Open in new window

LVL 14

Accepted Solution

sjm_ee earned 500 total points
ID: 21837007
Oops! Forgot to ask for a "lspv" but I think we can guess that rootvg is on hdisk0. The first thing I notice about the "sar" output is that when doing file copy work the vg2 disks are busy but rootvg is quiet whereas in the application example rootvg is also busy. This is unusual - is that paging?

Very busy disks with low data rate looks like very random i/o - AIX will spot if you are doing stuff with sequential files and is usually very good at reading ahead.

What is also interesting is that /home has been changed to a jfs2 and it's big - much bigger than a usual /home. Is there application activity there also?
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

LVL 14

Author Comment

ID: 21841030
When I was doing the testing from the OS I was making a copy from /u2 to /u3, so the vg2 being busy would be expected.  I get similar results when I copy between vg's.

/home is where the application resides along with some of the data.  /u1 /u2 and /u3 each contain data files and they are very large.  /u2 has only 27 files that range from 200 Meg to 4.5 Gig.  I believe you are correct with the very random i/o.  
# lspv
hdisk0          00000000d3cb1293                    rootvg          active
hdisk1          00000000e7f4d4c4                    vg2             active
hdisk2          00000000e7fac13c                    vg2             active
hdisk3          00000000e7fac234                    vg2             active

Open in new window

LVL 14

Author Comment

ID: 21892898
I'm going to leave this open for a little while longer to see if there is any other advice.
LVL 14

Author Closing Comment

ID: 31469214
Thanks for the input.

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This tech tip describes how to install the Solaris Operating System from a tape backup that was created using the Solaris flash archive utility. I have used this procedure on the Solaris 8 and 9 OS, and it shoudl also work well on the Solaris 10 rel…
Every server (virtual or physical) needs a console: and the console can be provided through hardware directly connected, software for remote connections, local connections, through a KVM, etc. This document explains the different types of consol…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.

738 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question