Solved

Memory analysis

Posted on 2006-07-02
4
2,177 Views
Last Modified: 2013-12-27
How do you determine if your system (Sun Solaris, version 6) is running out of memory? The only way that I know is using the "top" command and to see if a lot of swap space is being used.
Any suggestions are appreciated.
0
Comment
Question by:rlburris
4 Comments
 
LVL 10

Accepted Solution

by:
Nukfror earned 200 total points
Comment Utility
top gives a a limited view of what's going on.  What's important to understand is if your system is page scanning.  The command that will tell you the "scan rate" is vmstat.  In vmstat, the "sr" column shows the number of pages scanned per second by the page daemon.  First a suggestion ... to really understand how virtual memory works on Solaris.  There are several good discussions on the Net - just Google for "solaris virtual memory".   Princeton has a nice discussion:

http://www.princeton.edu/~psg/unix/Solaris/troubleshoot/ram.html

Anyway, the hardest part for people to understand about Solaris is something called the FS Page Cache and how and why it consumes.  It is *the* largest consumer of memory.  It will consume as much memory as it needs to up to a point called LOTSFREE.  

LOTSFREE is simply a threshold.  Basically, if the number of free memory pages is at or below LOTSFREE, the page daemon wakes up "scanning" for pages to steal and return to the free page list.  The "art" part of your question is determining when page scanning becomes a performance problem.  A little isn't a bad thing and is perfectly normal.  Excessive page scanning causes any number of performance issues but the usual one is swap area performance as stack/heap pages that are stolen are physically written out to the swap partition.

The old rule is 200 pages / sec is considered a problem.  For older systems and/or smaller systems this rule is probably still valid.  
0
 
LVL 38

Assisted Solution

by:yuzh
yuzh earned 200 total points
Comment Utility
You can also use "vmstat" to check it out.
man vmstat
to learn more details.

Please have a look at "Memory and swapping":
http://www.princeton.edu/~psg/unix/Solaris/troubleshoot/ram.html

"Sizing up memory in Solaris":
http://sunsite.uakom.sk/sunworldonline/swol-03-1998/swol-03-perf.html

"Solaris 8 Memory Architecture":
http://developers.sun.com/solaris/articles/sol8memory.html
0
 
LVL 3

Assisted Solution

by:anumalas
anumalas earned 100 total points
Comment Utility
Hi rlburris

Im going to give lot of information related to performance ... please go through bellow mentioned. Im not giving answer to your question but im giving complete senario and steps to follow to avoid such errors and steps to improve your sever performance . allocatr some time to read bellow

Generally the system performance is obtained and improved with several tasks like

Monitoring disk space with du and df

Monitoring system memory utilization, cpu utilization  with free and top commands

Checking the processes in queue with atq and at –l

Finding out the diagnostics information about process memory swap i/o system and  cpu utilization with vmstat, iostat, mpstat

Monitoring swap space

Monitoring system with syslog. And solve the problems depends on messages alerts and warnings.

Generally syslog messages are stored in /var/adm/messages file:

Observing all the log files:

Checking the log file sizes.



Important log files are

/var/adm/messages – The message file holds information that prints to the console.  These might include root logins and su attempts.

/var/adm/lastlog – this file holds most recent login time for each user in the system

/var/adm/utmp and /var/adm/utmpx – The utmp database file contains user access and accounting information for commands such as who, write and login.  The utmp file is where the information such as the terminal line and login time are stored for access by the who command.

/var/adm/wtmp and /var/adm/wtmpx – The wtmp file contains the history of user access and accounting information for the utmp database.  The wtmp file keeps track of logins and logouts since reboot.  The last command reads that file and process information.

/var/adm/acct – This is the system accounting file.  If enabled the accounting file records a record for every process listing the following information. The name of the user who ran the command the name of the command the cpu time used the completion time stamp of the process and a flag indicating completion status.  Accounting information can be very useful in monitoring who is doing what on your system.

/var/adm/sulog – This sulog file holds records for everyone who has used the su command on the sytem.

/var/cron/log  -- This file keeps a record of all cron activity.

As a administrator keeping the close watch on system messages and processes, you will become familiar with what is normal and what is abnormal.  Watch system messages daily and create shell scripts to watch processes for you and to look for irregularities in the system log files.


iostat
Displays an overview of CPU utilization, along with I/O statistics for one or more disk drives.
mpstat
Displays more in-depth CPU statistics.
Sysstat also contains tools that collect system resource utilization data and create daily reports based
on that data. These tools are:
sadc
Known as the system activity data collector, sadc collects system resource utilization information
and writes it to a _le.
sar
Producing reports from the _les created by sadc, sar reports can be generated interactively or
written to a _le for more intensive analysis.
The following sections explore each of these tools in more detail.

The iostat command at its most basic provides an overview of CPU and disk I/O statistics:

 iostat displays an overview of the system's average CPU utilization since the last reboot. The
CPU utilization report includes the following percentages:
ï Percentage of time spent in user mode (running applications, etc.)
ï Percentage of time spent in user mode (for processes that have altered their scheduling priority
using nice(2))
ï Percentage of time spent in kernel mode
ï Percentage of time spent idle
Below the CPU utilization report is the device utilization report. This report contains one line for each
active disk device on the system and includes the following information:

The mpstat command
The mpstat command at _rst appears no different from the CPU utilization report produced by iostat:
07:09:26 PM CPU %user %nice %system %idle intr/s
07:09:26 PM all 6.40 5.84 3.29 84.47 542.47
With the exception of an additional column showing the interrupts per second being handled by the
CPU, there is no real difference. However, the situation changes if mpstat's -P ALL option is used:

07:13:03 PM CPU %user %nice %system %idle intr/s
07:13:03 PM all 6.40 5.84 3.29 84.47 542.47
07:13:03 PM 0 6.36 5.80 3.29 84.54 542.47
07:13:03 PM 1 6.43 5.87 3.29 84.40 542.47
On multiprocessor systems, mpstat allows the utilization for each CPU to be viewed individually,
making it possible to determine how effectively each CPU is being used.

The sar command
The sar command produces system utilization reports based on the data collected by sadc. As con-
_gured in Red Hat Linux, sar is automatically run to process the _les automatically collected by
sadc. The report _les are written to /var/log/sa/ and are named sar
_
dd
      
, where
_
dd
      
is the
two-digit representations of the previous day's two-digit date.
sar is normally run by the sa2 script. This script is periodically invoked by cron via the _le sysstat,
which is located in /etc/crond.d. By default, cron runs sa2 once a day at 23:53, allowing
it to produce a report for the entire day's data.
2.5.4.4.1. Reading sar Reports
The format of a sar report produced by the default Red Hat Linux con_guration consists of multiple
sections, with each section containing a speci_c type of data, ordered by the time of day that the
data was collected. Since sadc is con_gured to perform a one-second measurement interval every ten
minutes, the default sar reports contain data in ten-minute increments, from 00:00 to 23:504.
Each section of the report starts with a heading that illustrates the data contained in the section. The
heading is repeated at regular intervals throughout the section, making it easier to interpret the data
while paging through the report. Each section ends with a line containing the average of the data
reported in that section.
Here is a sample section sar report, with the data from 00:30 through 23:40 removed to save space:
00:00:01 CPU %user %nice %system %idle
00:10:00 all 6.39 1.96 0.66 90.98
00:20:01 all 1.61 3.16 1.09 94.14
...
23:50:01 all 44.07 0.02 0.77 55.14
Average: all 5.80 4.99 2.87 86.34
In this section, CPU utilization information is displayed. This is very similar to the data displayed by
iostat.
Other sections may have more than one line's worth of data per time, as shown by this section generated
from CPU utilization data collected on a dual-processor system:
00:00:01 CPU %user %nice %system %idle
00:10:00 0 4.19 1.75 0.70 93.37
00:10:00 1 8.59 2.18 0.63 88.60
00:20:01 0 1.87 3.21 1.14 93.78
00:20:01 1 1.35 3.12 1.04 94.49
...
23:50:01 0 42.84 0.03 0.80 56.33
23:50:01 1 45.29 0.01 0.74 53.95
Average: 0 6.00 5.01 2.74 86.25
Average: 1 5.61 4.97 2.99 86.43
There are a total of seventeen different sections present in reports generated by the default Red Hat
Linux sar con_guration; many are discussing in upcoming chapters. For more information about the
data contained in each section, see the sar(1) man page.

By using vmstat, it is possible to determine if overall device activity is excessive by viewing the bi
and bo _elds; in addition, taking note of the si and so _elds give you a bit more insight into how
much disk activity is due to swap-related I/O:
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
1 0 0 0 248088 158636 480804 0 0 2 6 120 120 10 3 87


Based on the abvoe info you can tune your system in a beautiful way :)
Hope this will help :)


0
 

Author Comment

by:rlburris
Comment Utility
All... Thank you for your input!
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

Why Shell Scripting? Shell scripting is a powerful method of accessing UNIX systems and it is very flexible. Shell scripts are required when we want to execute a sequence of commands in Unix flavored operating systems. “Shell” is the command line i…
Every server (virtual or physical) needs a console: and the console can be provided through hardware directly connected, software for remote connections, local connections, through a KVM, etc. This document explains the different types of consol…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now