Memory analysis

How do you determine if your system (Sun Solaris, version 6) is running out of memory? The only way that I know is using the "top" command and to see if a lot of swap space is being used.
Any suggestions are appreciated.
Who is Participating?
NukfrorConnect With a Mentor Commented:
top gives a a limited view of what's going on.  What's important to understand is if your system is page scanning.  The command that will tell you the "scan rate" is vmstat.  In vmstat, the "sr" column shows the number of pages scanned per second by the page daemon.  First a suggestion ... to really understand how virtual memory works on Solaris.  There are several good discussions on the Net - just Google for "solaris virtual memory".   Princeton has a nice discussion:

Anyway, the hardest part for people to understand about Solaris is something called the FS Page Cache and how and why it consumes.  It is *the* largest consumer of memory.  It will consume as much memory as it needs to up to a point called LOTSFREE.  

LOTSFREE is simply a threshold.  Basically, if the number of free memory pages is at or below LOTSFREE, the page daemon wakes up "scanning" for pages to steal and return to the free page list.  The "art" part of your question is determining when page scanning becomes a performance problem.  A little isn't a bad thing and is perfectly normal.  Excessive page scanning causes any number of performance issues but the usual one is swap area performance as stack/heap pages that are stolen are physically written out to the swap partition.

The old rule is 200 pages / sec is considered a problem.  For older systems and/or smaller systems this rule is probably still valid.  
yuzhConnect With a Mentor Commented:
You can also use "vmstat" to check it out.
man vmstat
to learn more details.

Please have a look at "Memory and swapping":

"Sizing up memory in Solaris":

"Solaris 8 Memory Architecture":
anumalasConnect With a Mentor Commented:
Hi rlburris

Im going to give lot of information related to performance ... please go through bellow mentioned. Im not giving answer to your question but im giving complete senario and steps to follow to avoid such errors and steps to improve your sever performance . allocatr some time to read bellow

Generally the system performance is obtained and improved with several tasks like

Monitoring disk space with du and df

Monitoring system memory utilization, cpu utilization  with free and top commands

Checking the processes in queue with atq and at –l

Finding out the diagnostics information about process memory swap i/o system and  cpu utilization with vmstat, iostat, mpstat

Monitoring swap space

Monitoring system with syslog. And solve the problems depends on messages alerts and warnings.

Generally syslog messages are stored in /var/adm/messages file:

Observing all the log files:

Checking the log file sizes.

Important log files are

/var/adm/messages – The message file holds information that prints to the console.  These might include root logins and su attempts.

/var/adm/lastlog – this file holds most recent login time for each user in the system

/var/adm/utmp and /var/adm/utmpx – The utmp database file contains user access and accounting information for commands such as who, write and login.  The utmp file is where the information such as the terminal line and login time are stored for access by the who command.

/var/adm/wtmp and /var/adm/wtmpx – The wtmp file contains the history of user access and accounting information for the utmp database.  The wtmp file keeps track of logins and logouts since reboot.  The last command reads that file and process information.

/var/adm/acct – This is the system accounting file.  If enabled the accounting file records a record for every process listing the following information. The name of the user who ran the command the name of the command the cpu time used the completion time stamp of the process and a flag indicating completion status.  Accounting information can be very useful in monitoring who is doing what on your system.

/var/adm/sulog – This sulog file holds records for everyone who has used the su command on the sytem.

/var/cron/log  -- This file keeps a record of all cron activity.

As a administrator keeping the close watch on system messages and processes, you will become familiar with what is normal and what is abnormal.  Watch system messages daily and create shell scripts to watch processes for you and to look for irregularities in the system log files.

Displays an overview of CPU utilization, along with I/O statistics for one or more disk drives.
Displays more in-depth CPU statistics.
Sysstat also contains tools that collect system resource utilization data and create daily reports based
on that data. These tools are:
Known as the system activity data collector, sadc collects system resource utilization information
and writes it to a _le.
Producing reports from the _les created by sadc, sar reports can be generated interactively or
written to a _le for more intensive analysis.
The following sections explore each of these tools in more detail.

The iostat command at its most basic provides an overview of CPU and disk I/O statistics:

 iostat displays an overview of the system's average CPU utilization since the last reboot. The
CPU utilization report includes the following percentages:
ï Percentage of time spent in user mode (running applications, etc.)
ï Percentage of time spent in user mode (for processes that have altered their scheduling priority
using nice(2))
ï Percentage of time spent in kernel mode
ï Percentage of time spent idle
Below the CPU utilization report is the device utilization report. This report contains one line for each
active disk device on the system and includes the following information:

The mpstat command
The mpstat command at _rst appears no different from the CPU utilization report produced by iostat:
07:09:26 PM CPU %user %nice %system %idle intr/s
07:09:26 PM all 6.40 5.84 3.29 84.47 542.47
With the exception of an additional column showing the interrupts per second being handled by the
CPU, there is no real difference. However, the situation changes if mpstat's -P ALL option is used:

07:13:03 PM CPU %user %nice %system %idle intr/s
07:13:03 PM all 6.40 5.84 3.29 84.47 542.47
07:13:03 PM 0 6.36 5.80 3.29 84.54 542.47
07:13:03 PM 1 6.43 5.87 3.29 84.40 542.47
On multiprocessor systems, mpstat allows the utilization for each CPU to be viewed individually,
making it possible to determine how effectively each CPU is being used.

The sar command
The sar command produces system utilization reports based on the data collected by sadc. As con-
_gured in Red Hat Linux, sar is automatically run to process the _les automatically collected by
sadc. The report _les are written to /var/log/sa/ and are named sar
, where
is the
two-digit representations of the previous day's two-digit date.
sar is normally run by the sa2 script. This script is periodically invoked by cron via the _le sysstat,
which is located in /etc/crond.d. By default, cron runs sa2 once a day at 23:53, allowing
it to produce a report for the entire day's data. Reading sar Reports
The format of a sar report produced by the default Red Hat Linux con_guration consists of multiple
sections, with each section containing a speci_c type of data, ordered by the time of day that the
data was collected. Since sadc is con_gured to perform a one-second measurement interval every ten
minutes, the default sar reports contain data in ten-minute increments, from 00:00 to 23:504.
Each section of the report starts with a heading that illustrates the data contained in the section. The
heading is repeated at regular intervals throughout the section, making it easier to interpret the data
while paging through the report. Each section ends with a line containing the average of the data
reported in that section.
Here is a sample section sar report, with the data from 00:30 through 23:40 removed to save space:
00:00:01 CPU %user %nice %system %idle
00:10:00 all 6.39 1.96 0.66 90.98
00:20:01 all 1.61 3.16 1.09 94.14
23:50:01 all 44.07 0.02 0.77 55.14
Average: all 5.80 4.99 2.87 86.34
In this section, CPU utilization information is displayed. This is very similar to the data displayed by
Other sections may have more than one line's worth of data per time, as shown by this section generated
from CPU utilization data collected on a dual-processor system:
00:00:01 CPU %user %nice %system %idle
00:10:00 0 4.19 1.75 0.70 93.37
00:10:00 1 8.59 2.18 0.63 88.60
00:20:01 0 1.87 3.21 1.14 93.78
00:20:01 1 1.35 3.12 1.04 94.49
23:50:01 0 42.84 0.03 0.80 56.33
23:50:01 1 45.29 0.01 0.74 53.95
Average: 0 6.00 5.01 2.74 86.25
Average: 1 5.61 4.97 2.99 86.43
There are a total of seventeen different sections present in reports generated by the default Red Hat
Linux sar con_guration; many are discussing in upcoming chapters. For more information about the
data contained in each section, see the sar(1) man page.

By using vmstat, it is possible to determine if overall device activity is excessive by viewing the bi
and bo _elds; in addition, taking note of the si and so _elds give you a bit more insight into how
much disk activity is due to swap-related I/O:
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
1 0 0 0 248088 158636 480804 0 0 2 6 120 120 10 3 87

Based on the abvoe info you can tune your system in a beautiful way :)
Hope this will help :)

rlburrisAuthor Commented:
All... Thank you for your input!
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.