• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1295
  • Last Modified:

Performance Issues - Troubleshooting Solaris


Hey Experts,
I"m more of a Windows admin but have adopted some Solaris 5.8 servers and learning as i go..
The situation is we've been having some java app issues on the box referencing memory. I am not seeing any specific processes not terminating properly, so now was looking at the hardware itself. Below are some quick snapshots of some stats for cpu & memory. I have an idea what i'm looking at but would like someone with more knowledge to confirm. Would like to know if these stats under the memory are indicating that I may need to purchase additional mem for the box.... can you confirm / deny please.. and if there's any other utilities i can run to give me additonal useful stats..
thanks in advance

CPU STATS
iostat 5 10
   tty        md0           md1           md2           md3            cpu
 tin tout kps tps serv  kps tps serv  kps tps serv  kps tps serv   us sy wt id
  16   41   2   0   27    1   0   27    1   0   33    0   0   44   111 40  2 281
  11   84   0   0    0    0   0    0    0   0    0    0   0    0    7  2  0 91
   0   16   0   0    0    0   0    0    0   0    0    0   0    0   20  8  0 71
   0   16   0   0    0    0   0    0    0   0    0    0   0    0   27  3  0 70
   0   16   0   0    0    0   0    0    0   0    0    0   0    0   15  3  0 82
   0   16   0   0    0    0   0    0    0   0    0    0   0    0   42  4  0 55
   0   16   0   0    0    0   0    0    0   0    0    0   0    0   60  7  0 34
   0   16   0   0    0    0   0    0    0   0    0    0   0    0   52  5  0 42
   0   16   0   0    0    0   0    0    0   0    0    0   0    0   51 10  0 39
   0   16   0   0    0    0   0    0    0   0    0    0   0    0   50  4  0 46

MEMORY STATS
vmstat 5 10
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 m0 m1 m2   in   sy   cs us sy id
 0 0 0 1900384 678056 69 160 424 6 6  0  0  0  0  0  0  236  459  380 110 40 282
 0 0 36 1132448 903376 0  1  0  0  0  0  0  0  0  0  0  775 1863 1488 45  5 50
 0 0 36 1131040 902816 0 49  0  0  0  0  0  0  0  0  0 1191 4012 2225 63  8 29
 0 0 36 1130528 902632 0 43  0  0  0  0  0  0  0  0  0 1454 4151 2565 70 15 15
 0 0 36 1130568 902656 0  7  0  0  0  0  0  0  0  0  0 1317 3124 2573 74  7 19
 0 0 36 1132320 903472 0  1  0  0  0  0  0  0  0  0  0 1245 2602 2497 55  7 38
 0 0 36 1133104 903856 1  3  0  0  0  0  0  0  0  0  0  753 2197 1519 37  6 57
 0 0 36 1134096 904320 0  1  0  0  0  0  0  0  0  0  0  428  603  778  2  1 97
 0 0 36 1134832 904696 0  0  0  0  0  0  0  0  0  0  0  487  950  905 10  2 88
 0 0 36 1134832 904696 0  0  0  0  0  0  0  0  0  0  0  498  776  831  5  7 88

if you need me to pull more stats (processes , etc.) let me know and i'll post them
thanks again









0
SflahertyEE
Asked:
SflahertyEE
4 Solutions
 
bpeterseCommented:
Please post the following:
prtdiag
mpstat 5 5  (if multi proc)
prstat 5 5 (if not)
swap -l

What's your java app and do you have a certain amt of memory allocated to it?
0
 
NukfrorCommented:
As bpeterse rightfully suggested, more data please :)

But of interest in the data you sent:

>  r b w   swap  free  re  mf pi po fr de sr f0 m0 m1 m2   in   sy   cs us sy id
> 0 0 36 1131040 902816 0 49  0  0  0  0  0  0  0  0  0 1191 4012 2225 63  8 29
> 0 0 36 1130528 902632 0 43  0  0  0  0  0  0  0  0  0 1454 4151 2565 70 15 15
> 0 0 36 1130568 902656 0  7  0  0  0  0  0  0  0  0  0 1317 3124 2573 74  7 19

You have 36 swapped out processes - which processes who knows and it can be hard to determine but this means at a some point in the past free memory dropped below DESFREE e.g. you have an EXTREME shortage of memory.  With the data you sent, you've currently got right at 900MB.  You've got about 1.1GB of available swap.  Can't say whether either is bad or good without more data.

Swapping processes out *once* and assuming they stay swapped out - no big deal.

Running into a cycle of processes getting swapped out and then swapped back in later then they wake up for some reason - you've got a problem.
0
 
arthurjbCommented:
The output of swap -l would help a lot.

During the Solaris 8 time there were a few folks who suggested too little swap space during the set up process.  

The /tmp uses the tempfs file services which depend on swap space, so if you are using a service/program that uses a lot of temporary files, you may have a preformance problem related to swap, even if you have what seems like the proper amount of swap space.

The biggest problem I have seen with this is msql , it caused our server to slow to a crawl, until I doubled swap space, then it worked fine and the server preformance was fine...

Another way to test for this is during the slow periods do a;
df -lk
look at the /tmp and swap areas, and if you see very little space available, lack of swap space is likely your problem.
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
SflahertyEEAuthor Commented:

Hi Guys,
Thanks for the quick response. Was off at a client site today so will post that additional info tomorrow when can get back to the server.

bpeterse, as for your question, not sure exactly what the 'java' app is (was created by 3rd party developers), so a proprietary app. As for your memory question, I know they had just upped the memory parameters on the java side, but not sure what the upped it to.. i'll try and find out and post that as well tomorrow.

0
 
bpeterseCommented:
OK - is the custom app hosted through tomcat, jboss or [since this is the Solaris section] Sun's App server?  Try to also find out what the total amount of memory vs. amount allocated to java.  If you're going to price memory, be sure what the hardware will handle - that's where the prtdiag -v will come in handy.  I don't have the link to Sun's hardware site right now but can post that later unless someone else can post it before.

 - man vmstat: A couple of items to look at like Nukfror suggests is paging - 'vmstat -p' will give more details.  Swapping - 'vmstat -S'

Yet another item that comes to mind is how the application/app server is tuned, e.g. what's the time-out for each connection to the app/database, if that information is being cached and for how long.

More data would be very helpful. Post what you can.
0
 
cpc2004Commented:
The most useful indicator for memory short is sr
 
/sr = The number of pages scanned by the page daemon as it looks for
pages to steal from the processes that are not using them often. This
number is the key memory shortage indicator if it stays above 200 pages
per second for long periods of time. This is the most important value /

From your vmstat, all the value of SR is zero. Your system does not have memory shortage. Your system has high CPU utlilization at user mode (ie from 63 to 74) and it is main problem. Your Java application uses a lot of CPU.
0
 
SflahertyEEAuthor Commented:
Hey Guys, sorry for the delay, been real busy with other client's up until now...
here's the additional info you were asking for::

# swap -l
swapfile             dev  swaplo blocks   free
/dev/dsk/c0t0d0s1   32,1      16 4195664 4140784                              

 prstat 5 5
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 28663 lsadmin   253M  234M sleep   10    0   0:00.22  13% java/44
 11687 lsadmin  1195M  817M sleep   50    0   0:00.18 2.9% jrun/72
 12763 lsadmin   109M   67M sleep   31    0   0:00.09 0.4% java/26
 17700 nobody   5976K 3360K sleep   58    0   0:00.00 0.1% httpd/3
 28725 nobody   6848K 4368K sleep   58    0   0:02.52 0.1% httpd/3
 28648 nobody   6840K 4360K sleep   58    0   0:02.58 0.1% httpd/3
 28719 nobody   6848K 4368K sleep   58    0   0:02.45 0.1% httpd/3
 28675 nobody   6904K 4424K sleep   58    0   0:02.49 0.1% httpd/3
 28666 nobody   6912K 4432K sleep   58    0   0:02.45 0.1% httpd/3
 17699 root     1536K 1288K cpu1    58    0   0:00.00 0.1% prstat/1
 11684 lsadmin   103M   42M sleep   20    0   0:00.07 0.1% jrun/22
 28664 nobody   6840K 4360K sleep   58    0   0:02.51 0.0% httpd/3
 28724 nobody   6872K 4392K sleep   58    0   0:02.50 0.0% httpd/3
 28732 nobody   6856K 4376K sleep   58    0   0:02.47 0.0% httpd/3
 28665 nobody   6904K 4424K sleep   58    0   0:02.59 0.0% httpd/3
Total: 67 processes, 348 lwps, load averages: 0.52, 0.58, 0.68      

df -lk
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d0       4127406 2281770 1804362    56%    /
/proc                      0       0       0     0%    /proc
fd                         0       0       0     0%    /dev/fd
mnttab                     0       0       0     0%    /etc/mnttab
swap                 2136696      16 2136680     1%    /var/run
swap                 2137064     384 2136680     1%    /tmp
/dev/md/dsk/d6       5161590 2748013 2413577    54%    /opt
/dev/md/dsk/d2       9377102 1888982 7488120    21%    /export
/dev/md/dsk/d12      19796037 14634710 5161327    74%    /com
/dev/md/dsk/d8       10322592  352725 9969867     4%    /com/logs
/vol/dev/dsk/c0t6d0/s8_maintenance_update_7_sparc
                      533566  533566       0   100%    /cdrom/s8_maintenance_up

vmstat -p
     memory           page          executable      anonymous      filesystem
   swap  free  re  mf  fr  de  sr  epi  epo  epf  api  apo  apf  fpi  fpo  fpf
 1900720 679064 64 148  5   0   0    0    0    0    0    0    1  394    4    4

vmstat -s
      253 swap ins
       51 swap outs
      506 pages swapped in
     1321 pages swapped out
1038389972 total address trans. faults taken
 46121855 page ins
  1270563 page outs
345453781 pages paged in
  4922661 pages paged out
449891970 total reclaims
449667504 reclaims from free list
        0 micro (hat) faults
1038389972 minor (as) faults
 36607110 major faults
197559979 copy-on-write faults
161767816 zero fill page faults
  3431121 pages examined by the clock daemon
       13 revolutions of the clock hand
  4937420 pages freed by the clock daemon
  5847068 forks
   274344 vforks
  2778470 execs
2921305232 cpu context switches
2496380308 device interrupts
3324887828 traps
3379437271 system calls
972839145 total name lookups (cache hits 94%)
1438227717 user   cpu
519251955 system cpu
3714671965 idle   cpu
 19588147 wait   cpu              

*** By what you all were saying, and the outputs I was receiving (and posted above), I don't believe more memory is the issue. I'm starting to lean towards the web app programming. As I mentioned, I don't know the exact parameters regarding how much memory was allocated in java, as well as the time out setting for the connections but am going to see if I can track that data down now.....let me know if anyone sees anything in the additional info I posted..
thx
0
 
bpeterseCommented:
SflahertyEE,

When you get a chance, please run the following command for the java and jrun processes and post the results:

/usr/ucb/ps -auxwww <pid# of java/jrun process>

This will give you a complete list of all the params passed in the startup command.

0
 
SflahertyEEAuthor Commented:
Hi Folks,
The developers got back to me and apparently it was an issue on their end. Thanks to everyone for their help.

I'll splice out the points between everyone...

thanks again..
0

Featured Post

Upgrade your Question Security!

Add Premium security features to your question to ensure its privacy or anonymity. Learn more about your ability to control Question Security today.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now