Solved

semget: No space left on device

Posted on 2004-08-27
14
394 Views
Last Modified: 2010-08-05
I have Apache+mod_ssl, and recently started getting the following error in the apache error log when trying to run "apachectl startssl":
    semget: No space left on device
"apachectl start" works fine however.

None of my partitions are near full, nor did I run out of inodes.

ideas?
0
Comment
Question by:jtrunek
  • 7
  • 4
  • 3
14 Comments
 
LVL 19

Expert Comment

by:ramazanyich
ID: 11918427
The problem occurs because working files can't be created.
It can happen if you don't have space left. You can check it using df -k
or there is no enough inodes left. You can check it using df -i.
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11918748
I didn't think I was out of space and/or inodes, but I checked that prior to posting using both df -k  and df -i as I stated in the original post.  Max on any partition was 7% disk usage, and 4% inode usage.
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11918804
I just tried to restart the server again, and it works fine this time.  I didn't do anything since my original post, so I'm not sure why it suddenly works.  Explanations in case it happens again? I don't like mystery fixes like this.
0
 
LVL 19

Expert Comment

by:ramazanyich
ID: 11919700
found similiar problem on the net. Hope this will help you
http://www.forum.psoft.net/showthread.php?t=8428
0
 
LVL 2

Expert Comment

by:brozzis
ID: 11920457

it's not the physical disk space that is missing here,
it's the space reserved by the kernel to your Semaphores,
to managed your shared memory

You can see (on a linux box) quotes reserved to your semaphores with the command:

$ cat /proc/sys/kernel/sem
250     32000   32      128

(if support for the /proc is complied in)

you could add some space with the command (resetted after a boot) as root:

$ echo 64 65536 1024 1024 > /proc/sys/kernel/sem

but you should investigate on the reasons about this behaviour.
ipcs could help on check which process is taking so much resources...
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11921237
That link is unresponsive for me.  I can't even resolve the domain via dns.  Sure thats the right link?
I'll again later too
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11921244
here is my /proc/sys/kernel/sem:
   512     256000  128     1024
Shouldn't that have likely been sufficient? (this is basically a brand new barebones system, was just starting to set up the web services on it)

When the error occurred, i tried using ipcs to track down the resource hog, but it failed to turn up anything.  output that i got:

$ ipcs -s
------ Semaphore Arrays --------
key        semid      owner      perms      nsems      status

Any further ideas?
0
Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

 
LVL 2

Expert Comment

by:brozzis
ID: 11921333

uh, big values there! this is getting interesting...
please post also /proc/sys/kernel/shmmax.

versione of apache ?
have you compiled apache by yourself ? if yes, which mpm used ? could you post your "configure" string ?
have you modified your httpd.conf in the mpm sections (number of processes and so on?)


0
 
LVL 1

Author Comment

by:jtrunek
ID: 11922505
I realize those are big values.  Thats the default that came setup on this User Mode Linux VM, and I haven't bothered tweaking it yet.  /proc/sys/kernel/shmmax is 33554432.

I compiled Apache 1.3.31 and used the default mpm (prefork).  Unfortunately, I don't seem to have the configure string anymore, but I think it was:
./configure \
--prefix=/usr/local/apache \
--enable-module=ssl \
--enable-shared=ssl \
--enable-shared=max \
--enable-module=so

default httpd.conf in mpm section
0
 
LVL 19

Expert Comment

by:ramazanyich
ID: 11923033
I found that link using www.google.com. put your error response which you have and you wil get a lot of links with such a problem
0
 
LVL 2

Expert Comment

by:brozzis
ID: 11923513

no strange third part modules ?
no other processes on this machine ?

shmmax is on a reasonable value. small in respect to other values.
Personally, I'd try to reduce other values, to something like:

echo 250 32000 100 128 > /proc/sys/kernel/sem
echo 536870912 > /proc/sys/kernel/shmmax
echo 4096 > /proc/sys/kernel/shmmni
echo 2097152 > /proc/sys/kernel/shmall

that is a quite reasonable configuration, just to be sure that error isn't in there.
Try to post also a ipcs -l, perhaps we are missing something.
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11924460
no strange3rd party modules.  only ones were mod_ssl and php5.

no other oddball processes either. just sendmail and the typical system services like syslog and cron.

$ ipcs -l

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1

------ Semaphore Limits --------
max number of arrays = 2048
max semaphores per array = 1024
max semaphores system wide = 512000
max ops per semop call = 256
semaphore max value = 32767

------ Messages: Limits --------
max queues system wide = 16
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384
0
 
LVL 2

Accepted Solution

by:
brozzis earned 250 total points
ID: 11924667

well, I don't like these values:

------ Semaphore Limits --------
max number of arrays = 2048
max semaphores per array = 1024
max semaphores system wide = 512000
max ops per semop call = 256
semaphore max value = 32767

I think that "max semaphores system wide " should be equal to "max number of arrays" * "max semaphores per array"
2048 * 1024 = 2097152

please change values and retry.
I'd reduce values, alternatively set "max semaphores system wide" to 2097152

echo 250     32000   32      128 > /proc/sys/kernel/sem


For your reference, I have these values:

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11945908
I'll do that, but as hard as I try, I can't seem to reproduce this problem.  Since its nearly impossible to troubleshoot a problem that cant be reproduced, I'm going to award the points to brozzis for his help so far.
0

Featured Post

Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
tomcat LDAP authentication issue 3 147
Running Guacamole on port 80 5 152
Missing Apache mod_DBD packages in Centos 7 2 65
NGINX/Apache redirect on 403 and 404 5 68
Over the last year I have answered a couple of basic URL rewriting questions several times so I thought I might as well have a stab at: explaining the basics, providing a few useful links and consolidating some of the most common queries into a sing…
If you are a web developer, you would be aware of the <iframe> tag in HTML. The <iframe> stands for inline frame and is used to embed another document within the current HTML document. The embedded document could be even another website.
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
A short film showing how OnPage and Connectwise integration works.

929 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now