Solved

semget: No space left on device

Posted on 2004-08-27
14
405 Views
Last Modified: 2010-08-05
I have Apache+mod_ssl, and recently started getting the following error in the apache error log when trying to run "apachectl startssl":
    semget: No space left on device
"apachectl start" works fine however.

None of my partitions are near full, nor did I run out of inodes.

ideas?
0
Comment
Question by:jtrunek
  • 7
  • 4
  • 3
14 Comments
 
LVL 19

Expert Comment

by:ramazanyich
ID: 11918427
The problem occurs because working files can't be created.
It can happen if you don't have space left. You can check it using df -k
or there is no enough inodes left. You can check it using df -i.
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11918748
I didn't think I was out of space and/or inodes, but I checked that prior to posting using both df -k  and df -i as I stated in the original post.  Max on any partition was 7% disk usage, and 4% inode usage.
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11918804
I just tried to restart the server again, and it works fine this time.  I didn't do anything since my original post, so I'm not sure why it suddenly works.  Explanations in case it happens again? I don't like mystery fixes like this.
0
Use Case: Protecting a Hybrid Cloud Infrastructure

Microsoft Azure is rapidly becoming the norm in dynamic IT environments. This document describes the challenges that organizations face when protecting data in a hybrid cloud IT environment and presents a use case to demonstrate how Acronis Backup protects all data.

 
LVL 19

Expert Comment

by:ramazanyich
ID: 11919700
found similiar problem on the net. Hope this will help you
http://www.forum.psoft.net/showthread.php?t=8428
0
 
LVL 2

Expert Comment

by:brozzis
ID: 11920457

it's not the physical disk space that is missing here,
it's the space reserved by the kernel to your Semaphores,
to managed your shared memory

You can see (on a linux box) quotes reserved to your semaphores with the command:

$ cat /proc/sys/kernel/sem
250     32000   32      128

(if support for the /proc is complied in)

you could add some space with the command (resetted after a boot) as root:

$ echo 64 65536 1024 1024 > /proc/sys/kernel/sem

but you should investigate on the reasons about this behaviour.
ipcs could help on check which process is taking so much resources...
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11921237
That link is unresponsive for me.  I can't even resolve the domain via dns.  Sure thats the right link?
I'll again later too
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11921244
here is my /proc/sys/kernel/sem:
   512     256000  128     1024
Shouldn't that have likely been sufficient? (this is basically a brand new barebones system, was just starting to set up the web services on it)

When the error occurred, i tried using ipcs to track down the resource hog, but it failed to turn up anything.  output that i got:

$ ipcs -s
------ Semaphore Arrays --------
key        semid      owner      perms      nsems      status

Any further ideas?
0
 
LVL 2

Expert Comment

by:brozzis
ID: 11921333

uh, big values there! this is getting interesting...
please post also /proc/sys/kernel/shmmax.

versione of apache ?
have you compiled apache by yourself ? if yes, which mpm used ? could you post your "configure" string ?
have you modified your httpd.conf in the mpm sections (number of processes and so on?)


0
 
LVL 1

Author Comment

by:jtrunek
ID: 11922505
I realize those are big values.  Thats the default that came setup on this User Mode Linux VM, and I haven't bothered tweaking it yet.  /proc/sys/kernel/shmmax is 33554432.

I compiled Apache 1.3.31 and used the default mpm (prefork).  Unfortunately, I don't seem to have the configure string anymore, but I think it was:
./configure \
--prefix=/usr/local/apache \
--enable-module=ssl \
--enable-shared=ssl \
--enable-shared=max \
--enable-module=so

default httpd.conf in mpm section
0
 
LVL 19

Expert Comment

by:ramazanyich
ID: 11923033
I found that link using www.google.com. put your error response which you have and you wil get a lot of links with such a problem
0
 
LVL 2

Expert Comment

by:brozzis
ID: 11923513

no strange third part modules ?
no other processes on this machine ?

shmmax is on a reasonable value. small in respect to other values.
Personally, I'd try to reduce other values, to something like:

echo 250 32000 100 128 > /proc/sys/kernel/sem
echo 536870912 > /proc/sys/kernel/shmmax
echo 4096 > /proc/sys/kernel/shmmni
echo 2097152 > /proc/sys/kernel/shmall

that is a quite reasonable configuration, just to be sure that error isn't in there.
Try to post also a ipcs -l, perhaps we are missing something.
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11924460
no strange3rd party modules.  only ones were mod_ssl and php5.

no other oddball processes either. just sendmail and the typical system services like syslog and cron.

$ ipcs -l

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1

------ Semaphore Limits --------
max number of arrays = 2048
max semaphores per array = 1024
max semaphores system wide = 512000
max ops per semop call = 256
semaphore max value = 32767

------ Messages: Limits --------
max queues system wide = 16
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384
0
 
LVL 2

Accepted Solution

by:
brozzis earned 250 total points
ID: 11924667

well, I don't like these values:

------ Semaphore Limits --------
max number of arrays = 2048
max semaphores per array = 1024
max semaphores system wide = 512000
max ops per semop call = 256
semaphore max value = 32767

I think that "max semaphores system wide " should be equal to "max number of arrays" * "max semaphores per array"
2048 * 1024 = 2097152

please change values and retry.
I'd reduce values, alternatively set "max semaphores system wide" to 2097152

echo 250     32000   32      128 > /proc/sys/kernel/sem


For your reference, I have these values:

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11945908
I'll do that, but as hard as I try, I can't seem to reproduce this problem.  Since its nearly impossible to troubleshoot a problem that cant be reproduced, I'm going to award the points to brozzis for his help so far.
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In my time as an SEO for the last 2 years and in the questions I have assisted with on here I have always seen the need to redirect from non-www urls to their www versions. For instance redirecting http://domain.com (http://domain.com) to http…
Introduction As you’re probably aware the HTTP protocol offers basic / weak authentication, which in combination with the relevant configuration on your web server, provides the ability to password protect all or part of your host.  If you were not…
Microsoft Active Directory, the widely used IT infrastructure, is known for its high risk of credential theft. The best way to test your Active Directory’s vulnerabilities to pass-the-ticket, pass-the-hash, privilege escalation, and malware attacks …
Although Jacob Bernoulli (1654-1705) has been credited as the creator of "Binomial Distribution Table", Gottfried Leibniz (1646-1716) did his dissertation on the subject in 1666; Leibniz you may recall is the co-inventor of "Calculus" and beat Isaac…

803 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question