Solved

Speed problem with AIX 5.3 and BBX Pro5 Application

Posted on 2008-06-20
6
1,037 Views
Last Modified: 2013-11-17
I have a customer running a BBX Pro5 application that seems to run too slow for the machine they have.  The machine is an IBM P630 running AIX 5.3 with 4 Gig RAM and lots of disk.  The problem is that it seems to run slow with respect to disk access from the BBX application, yet seems super fast from the OS level.

Here are some stats:

# svmon -G
               size      inuse       free        pin    virtual
memory      1048576    1062012     -13436     121158     224251
pg space    2752512        474

               work       pers       clnt
pin          121158          0          0
in use       224251      42024     795737

Now the disk access (attached as a code snippet to display the columns better)  Note the Kbs/s for BBX access compared to OS access:


Output of sar -d (only a snippet of whenthe disks are acually busy), the first one is when the BBX pro5 application is accessing the disks. The second (at 10:56) is me doing some file copy opperations from the OS itself.   
 
 

09:18:22     device    %busy    avque    r+w/s    Kbs/s   avwait   avserv
 

09:18:48     hdisk2    100     57.3      287     1362    594.5     10.5

             hdisk0     97      0.4      241     1194      5.1      9.5

             hdisk3     84     23.7      228     1136    509.1     11.0

             hdisk1    100     58.4      332     1635    622.8      9.0
 
 

10:56:13     hdisk2    100      8.3      544    34673     65.8      5.9

             hdisk0      1      0.0        2       11      0.0      6.6

             hdisk3     82      0.0      253    16102      0.4      5.2

             hdisk1     97      2.1      491    31370     16.5      5.9

Open in new window

0
Comment
Question by:mikelfritz
  • 4
  • 2
6 Comments
 
LVL 14

Expert Comment

by:sjm_ee
ID: 21832543
General: Has this application ever run acceptably on thjis system? If so, what changed? If not, do you have any expectations from other systems?

Specific: Is the BBX Pro/5 application running on top of Oracle? Is the data access to files or to raw devices? How many CPUs in the 630? ("prtconf"). Please post the output of "lsvg", "lsvg -o", "lsvg -o | lsvg -i -l".
0
 
LVL 14

Author Comment

by:mikelfritz
ID: 21832672
The application never ran as fast as it should have.  No oracle or raw device access.  

The application was moved to the P630 from an old F50 two years ago.  At that time the BBX needed to be upgraded to handle AIX 5.3,  the upgrade doubled the size of all of the data files.  The new P630 and the old F50 seem to run the application about the same - although since the BBX was upgraded it's not really an apples to apples comparison.

 From the OS level it is like night and day, the P630 runs circles around the F50.  I ran some tests using awk on some really big files on both systems back when we switched and the new server was orders of magnitude faster.


# prtconf

System Model: IBM,7028-6C4

Machine Serial Number: 0000000

Processor Type: PowerPC_POWER4

Number Of Processors: 4

Processor Clock Speed: 1000 MHz

CPU Type: 64-bit

Kernel Type: 32-bit

LPAR Info: 1 NULL

Memory Size: 4096 MB

Good Memory Size: 4096 MB

Platform Firmware level: RR021114

Firmware Version: IBM,RG021113_GA3_ALL

Console Login: enable

Auto Restart: true

Full Core: false
 

# lsvg -o

vg2

rootvg

# lsvg -o | lsvg -i -l

vg2:

LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT

loglv00             jfs2log    1     1     1    open/syncd    N/A

fslv00              jfs2       239   239   1    closed/syncd  /u1.old

fslv01              jfs2       239   239   1    closed/syncd  /u2.old

u1str               jfs2       240   240   3    open/syncd    /u1

u2str               jfs2       240   240   3    open/syncd    /u2

u3str               jfs2       240   240   3    open/syncd    /u3

rootvg:

LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT

hd5                 boot       1     1     1    closed/syncd  N/A

hd6                 paging     21    21    1    open/syncd    N/A

hd8                 jfslog     1     1     1    open/syncd    N/A

hd4                 jfs        1     1     1    open/syncd    /

hd2                 jfs        3     3     1    open/syncd    /usr

hd9var              jfs        2     2     1    open/syncd    /var

hd3                 jfs        1     1     1    open/syncd    /tmp

loglv01             jfs2log    1     1     1    open/syncd    N/A

hd10opt             jfs        1     1     1    open/syncd    /opt

lg_dumplv           sysdump    2     2     1    open/syncd    N/A

fslv02              jfs2       239   239   1    open/syncd    /home

Open in new window

0
 
LVL 14

Accepted Solution

by:
sjm_ee earned 500 total points
ID: 21837007
Oops! Forgot to ask for a "lspv" but I think we can guess that rootvg is on hdisk0. The first thing I notice about the "sar" output is that when doing file copy work the vg2 disks are busy but rootvg is quiet whereas in the application example rootvg is also busy. This is unusual - is that paging?

Very busy disks with low data rate looks like very random i/o - AIX will spot if you are doing stuff with sequential files and is usually very good at reading ahead.

What is also interesting is that /home has been changed to a jfs2 and it's big - much bigger than a usual /home. Is there application activity there also?
0
What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

 
LVL 14

Author Comment

by:mikelfritz
ID: 21841030
When I was doing the testing from the OS I was making a copy from /u2 to /u3, so the vg2 being busy would be expected.  I get similar results when I copy between vg's.

/home is where the application resides along with some of the data.  /u1 /u2 and /u3 each contain data files and they are very large.  /u2 has only 27 files that range from 200 Meg to 4.5 Gig.  I believe you are correct with the very random i/o.  
# lspv

hdisk0          00000000d3cb1293                    rootvg          active

hdisk1          00000000e7f4d4c4                    vg2             active

hdisk2          00000000e7fac13c                    vg2             active

hdisk3          00000000e7fac234                    vg2             active

Open in new window

0
 
LVL 14

Author Comment

by:mikelfritz
ID: 21892898
I'm going to leave this open for a little while longer to see if there is any other advice.
0
 
LVL 14

Author Closing Comment

by:mikelfritz
ID: 31469214
Thanks for the input.
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

Hello fellow BSD lovers, I've created a patch process for patching openjdk6 for BSD (FreeBSD specifically), although I tried to keep all BSD versions in mind when creating my patch. Welcome to OpenJDK6 on BSD First let me start with a little …
Attention: This article will no longer be maintained. If you have any questions, please feel free to mail me. jgh@FreeBSD.org Please see http://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-update-server/ for the updated article. It is avail…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now