Swap space issue in Solaris

Posted on 2008-11-10
Last Modified: 2013-12-27

Sometimes I'm really going mad with Swap space in Solaris? Please look at the below output, based on this i have some questions about Swap space?

root@hotsun# uname -a
SunOS 5.8 Generic_117350-14 sun4u sparc SUNW,Sun-Fire-880
root@hotsun# prtconf | grep Mem
Memory size: 16384 Megabytes
root@hotsun# swap -s
total: 13790024k bytes allocated + 99080k reserved = 13889104k used, 10667080k available
root@hotsun# swap -l
swapfile             dev  swaplo blocks   free
/dev/md/dsk/d1      85,1      16 23163984 22928464 df -k
Filesystem            kbytes    used        avail      capacity  Mounted on
/dev/md/dsk/d0   4691952   2882534 1762499    63%    /
/proc                       0           0       0                      0%        /proc
fd                            0           0       0                      0%    /dev/fd
mnttab                     0           0       0                      0%    /etc/mnttab
/dev/md/dsk/d3  14219619 5033441 9043982     36%    /var
swap                 10662024 32           10661992    1%    /var/run
swap                 10784344 122352   10661992    2%    /tmp
root@hotsun# cat /var/adm/messages.0 | grep 'Nov  9'

Nov  9 03:00:17 sendmail[10822]: [ID 801593 mail.alert] mA990F210822: SYSERR(root): savemail: cannot save rejected email anywhere
Nov  9 03:01:41 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded                                 <---------------------- top
last pid: 27680;  load averages:  3.62,  3.59, 3.70  
135 processes: 131 sleeping, 4 on cpu
CPU states: 13.1% idle, 39.1% user,  4.4% kernel, 43.3% iowait,  0.0% swap
Memory: 16G real, 14G swap in use, 10G swap free                   <-----------------------------------

25462 oracle     1  20    0   10G   10G cpu4   48:35 12.45% oracle
24883 oracle     1  20    0 9182M 9136M cpu3   57:07 12.39% oracle
21364 oracle    11  20    0 8406M 8358M cpu1   38:09  3.61% oracle
21362 oracle    11  51    0 8406M 8358M sleep  39:32  3.23% oracle
21366 oracle    11  41    0 8406M 8358M sleep  37:18  2.98% oracle
21360 oracle    11  52    0 8406M 8358M sleep  37:06  2.91% oracle
25423 root      11  60    0   78M   67M sleep   4:32  1.91% bpbkar
27607 oracle    11  58    0 8393M 8345M sleep   0:22  0.38% oracle
27611 oracle    11  58    0 8393M 8345M sleep   0:21  0.38% oracle
27605 oracle    11  58    0 8392M 8347M sleep   0:07  0.21% oracle
27609 oracle    11  58    0 8392M 8344M sleep   0:07  0.20% oracle
27668 root       1  58    0 3168K 2272K cpu6    0:00  0.11% top
  148 root      14  59    0   11M 7504K sleep  75:25  0.02% picld
27646 oracle    11  58    0 8391M 8343M sleep   0:00  0.02% oracle
27642 oracle    11  58    0 8391M 8343M sleep   0:00  0.02% oracle

1. Now what is the Total Swap swap space available in the system? From 'swap -l',  I could see it is 11 GB. If that is the case, why the 'top' output showing 14G swap in use, 10G swap free?

2. 'top' output shows 14G swap in use, 10G swap free. Please explain this.

3. Due to the following error " /tmp: File system full, swap space limit exceeded ", the Oracle DB running in the server got hung and we were forced to restart the Oracle database. Can you explain why it is and how to prevent it?

4. I persume /tmp filesystem is the Total size of Physical and Swap memory. If that is the case, it should show as 16 GB physical memory + Swap space. But 'df' output just shows approx. around 10 GB.  Can you clarify me this?


Question by:rdashokraj
    LVL 38

    Assisted Solution

    It might have something to to with your kernel settings, you need to configure parameter related to semaphores or shared memory need to increase. You need to configure the kernel parametrers to run Oracle.
    have a look at: "Solaris kernel has parameters " settings in:
    It is better to check the installation guide for your version of Oracle.
    To learn details about the Solaris swap space, please read:
    http: Q_21630919.html
    PS: "top" is 3rd command, for Solaris 8 or newer, you should run "prstat", it is better than "top"
    man prstat
    to learn more
    LVL 22

    Accepted Solution

    Questions 1, 2 and 4 all really have the same explaination. Swap is accounted as the total of the
    available memory plus the space on the swap devices. As you noted, the swap device is 11GB. However, not all of memory is available as swap. First, the memory used by the Kernel is subtracted. So according to the outputs, you have a total of 24GB of swap total.  At the time these commands were run, all the commands are in agreement that you have 10GB of swap space free.

    >3. Due to the following error " /tmp: File system full, swap space limit exceeded ",

    Ah, now this one is trickier. This says that there was a write which returned ENOSPC. Either /tmp was actually full or it has a quota placed on it. If it does have a quota, I would think that the df would indicate it in the available column, but it might not, I am not sure.

    It is certainly possible to fill up /tmp in a transitory manner. It is often the case that a program will open a temporary file and then delete it while it is still open. This means that when the program terminates, the OS will automatically delete the file. This used to be a common practice, but it is now considered a bit antisocial, since it makes it impossible for an adminstrator to deal with errant programs. So, it would be trivial to create such a temporary file, and then keep writing to it until /tmp was full and then exit, freeing up the space again. In fact, the write that failed might easily have been just such an occurance.

    Also notice, that it wasn't actually a program that is reporting that /tmp if full, it is the kernel. Another thing to note, is that sendmail does not save rejected email in /tmp, it should be writing that to the /var file system.
    I hope that helps.  If you are using Solaris 10, then you might be able touse Dtrace to hel pyou find out where your swap space goes, if it happens again.

    Author Comment

    Before I accept your solutions, I just to want confirm onething:

    So if someone creates files of huge size in /tmp filesystem, they can fill up the Swap space and it would potentially can crash the OS itself right? (due to insufficient swap space).  If yes, don't you think it is a vulnerability in Solaris OS, that anyone normal user can bring down the application or OS just by creatiing some huge files in /tmp ?
    LVL 22

    Expert Comment

    As I noted, you can set a quota on the size of /tmp, enforcing a buffer that will allow the system to keep running without running out of swap space. Of course applications that need free space in /tmp will start to fail, but it is up to the developers of the applications to handle error conditions gracefully.

    See the mount_tmpfs man page for details.

    Author Comment

    Thankyou so much for your explanation.

    Featured Post

    Gigs: Get Your Project Delivered by an Expert

    Select from freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely and get projects done right.

    Join & Write a Comment

    Suggested Solutions

    Let's say you need to move the data of a file system from one partition to another. This generally involves dismounting the file system, backing it up to tapes, and restoring it to a new partition. You may also copy the file system from one place to…
    This tech tip describes how to install the Solaris Operating System from a tape backup that was created using the Solaris flash archive utility. I have used this procedure on the Solaris 8 and 9 OS, and it shoudl also work well on the Solaris 10 rel…
    Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
    Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

    745 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    15 Experts available now in Live!

    Get 1:1 Help Now