Swap space issue in Solaris


Sometimes I'm really going mad with Swap space in Solaris? Please look at the below output, based on this i have some questions about Swap space?

root@hotsun# uname -a
SunOS hotsun.travelclick.net 5.8 Generic_117350-14 sun4u sparc SUNW,Sun-Fire-880
root@hotsun# prtconf | grep Mem
Memory size: 16384 Megabytes
root@hotsun# swap -s
total: 13790024k bytes allocated + 99080k reserved = 13889104k used, 10667080k available
root@hotsun# swap -l
swapfile             dev  swaplo blocks   free
/dev/md/dsk/d1      85,1      16 23163984 22928464
root@hotsun.travelclick.net# df -k
Filesystem            kbytes    used        avail      capacity  Mounted on
/dev/md/dsk/d0   4691952   2882534 1762499    63%    /
/proc                       0           0       0                      0%        /proc
fd                            0           0       0                      0%    /dev/fd
mnttab                     0           0       0                      0%    /etc/mnttab
/dev/md/dsk/d3  14219619 5033441 9043982     36%    /var
swap                 10662024 32           10661992    1%    /var/run
swap                 10784344 122352   10661992    2%    /tmp
root@hotsun# cat /var/adm/messages.0 | grep 'Nov  9'

Nov  9 03:00:17 hotsun.travelclick.net sendmail[10822]: [ID 801593 mail.alert] mA990F210822: SYSERR(root): savemail: cannot save rejected email anywhere
Nov  9 03:01:41 hotsun.travelclick.net tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded                                 <----------------------

root@hotsun.travelclick.net# top
last pid: 27680;  load averages:  3.62,  3.59, 3.70  
135 processes: 131 sleeping, 4 on cpu
CPU states: 13.1% idle, 39.1% user,  4.4% kernel, 43.3% iowait,  0.0% swap
Memory: 16G real, 14G swap in use, 10G swap free                   <-----------------------------------

25462 oracle     1  20    0   10G   10G cpu4   48:35 12.45% oracle
24883 oracle     1  20    0 9182M 9136M cpu3   57:07 12.39% oracle
21364 oracle    11  20    0 8406M 8358M cpu1   38:09  3.61% oracle
21362 oracle    11  51    0 8406M 8358M sleep  39:32  3.23% oracle
21366 oracle    11  41    0 8406M 8358M sleep  37:18  2.98% oracle
21360 oracle    11  52    0 8406M 8358M sleep  37:06  2.91% oracle
25423 root      11  60    0   78M   67M sleep   4:32  1.91% bpbkar
27607 oracle    11  58    0 8393M 8345M sleep   0:22  0.38% oracle
27611 oracle    11  58    0 8393M 8345M sleep   0:21  0.38% oracle
27605 oracle    11  58    0 8392M 8347M sleep   0:07  0.21% oracle
27609 oracle    11  58    0 8392M 8344M sleep   0:07  0.20% oracle
27668 root       1  58    0 3168K 2272K cpu6    0:00  0.11% top
  148 root      14  59    0   11M 7504K sleep  75:25  0.02% picld
27646 oracle    11  58    0 8391M 8343M sleep   0:00  0.02% oracle
27642 oracle    11  58    0 8391M 8343M sleep   0:00  0.02% oracle

1. Now what is the Total Swap swap space available in the system? From 'swap -l',  I could see it is 11 GB. If that is the case, why the 'top' output showing 14G swap in use, 10G swap free?

2. 'top' output shows 14G swap in use, 10G swap free. Please explain this.

3. Due to the following error " /tmp: File system full, swap space limit exceeded ", the Oracle DB running in the server got hung and we were forced to restart the Oracle database. Can you explain why it is and how to prevent it?

4. I persume /tmp filesystem is the Total size of Physical and Swap memory. If that is the case, it should show as 16 GB physical memory + Swap space. But 'df' output just shows approx. around 10 GB.  Can you clarify me this?


Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

It might have something to to with your kernel settings, you need to configure parameter related to semaphores or shared memory need to increase. You need to configure the kernel parametrers to run Oracle.
have a look at: "Solaris kernel has parameters " settings in:
It is better to check the installation guide for your version of Oracle.
To learn details about the Solaris swap space, please read:
http: Q_21630919.html
PS: "top" is 3rd command, for Solaris 8 or newer, you should run "prstat", it is better than "top"
man prstat
to learn more
Brian UtterbackPrinciple Software EngineerCommented:
Questions 1, 2 and 4 all really have the same explaination. Swap is accounted as the total of the
available memory plus the space on the swap devices. As you noted, the swap device is 11GB. However, not all of memory is available as swap. First, the memory used by the Kernel is subtracted. So according to the outputs, you have a total of 24GB of swap total.  At the time these commands were run, all the commands are in agreement that you have 10GB of swap space free.

>3. Due to the following error " /tmp: File system full, swap space limit exceeded ",

Ah, now this one is trickier. This says that there was a write which returned ENOSPC. Either /tmp was actually full or it has a quota placed on it. If it does have a quota, I would think that the df would indicate it in the available column, but it might not, I am not sure.

It is certainly possible to fill up /tmp in a transitory manner. It is often the case that a program will open a temporary file and then delete it while it is still open. This means that when the program terminates, the OS will automatically delete the file. This used to be a common practice, but it is now considered a bit antisocial, since it makes it impossible for an adminstrator to deal with errant programs. So, it would be trivial to create such a temporary file, and then keep writing to it until /tmp was full and then exit, freeing up the space again. In fact, the write that failed might easily have been just such an occurance.

Also notice, that it wasn't actually a program that is reporting that /tmp if full, it is the kernel. Another thing to note, is that sendmail does not save rejected email in /tmp, it should be writing that to the /var file system.
I hope that helps.  If you are using Solaris 10, then you might be able touse Dtrace to hel pyou find out where your swap space goes, if it happens again.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
rdashokrajAuthor Commented:
Before I accept your solutions, I just to want confirm onething:

So if someone creates files of huge size in /tmp filesystem, they can fill up the Swap space and it would potentially can crash the OS itself right? (due to insufficient swap space).  If yes, don't you think it is a vulnerability in Solaris OS, that anyone normal user can bring down the application or OS just by creatiing some huge files in /tmp ?
Brian UtterbackPrinciple Software EngineerCommented:
As I noted, you can set a quota on the size of /tmp, enforcing a buffer that will allow the system to keep running without running out of swap space. Of course applications that need free space in /tmp will start to fail, but it is up to the developers of the applications to handle error conditions gracefully.

See the mount_tmpfs man page for details.
rdashokrajAuthor Commented:
Thankyou so much for your explanation.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Unix OS

From novice to tech pro — start learning today.