Solved

semget: No space left on device

Posted on 2004-08-27
14
387 Views
Last Modified: 2010-08-05
I have Apache+mod_ssl, and recently started getting the following error in the apache error log when trying to run "apachectl startssl":
    semget: No space left on device
"apachectl start" works fine however.

None of my partitions are near full, nor did I run out of inodes.

ideas?
0
Comment
Question by:jtrunek
  • 7
  • 4
  • 3
14 Comments
 
LVL 19

Expert Comment

by:ramazanyich
ID: 11918427
The problem occurs because working files can't be created.
It can happen if you don't have space left. You can check it using df -k
or there is no enough inodes left. You can check it using df -i.
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11918748
I didn't think I was out of space and/or inodes, but I checked that prior to posting using both df -k  and df -i as I stated in the original post.  Max on any partition was 7% disk usage, and 4% inode usage.
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11918804
I just tried to restart the server again, and it works fine this time.  I didn't do anything since my original post, so I'm not sure why it suddenly works.  Explanations in case it happens again? I don't like mystery fixes like this.
0
 
LVL 19

Expert Comment

by:ramazanyich
ID: 11919700
found similiar problem on the net. Hope this will help you
http://www.forum.psoft.net/showthread.php?t=8428
0
 
LVL 2

Expert Comment

by:brozzis
ID: 11920457

it's not the physical disk space that is missing here,
it's the space reserved by the kernel to your Semaphores,
to managed your shared memory

You can see (on a linux box) quotes reserved to your semaphores with the command:

$ cat /proc/sys/kernel/sem
250     32000   32      128

(if support for the /proc is complied in)

you could add some space with the command (resetted after a boot) as root:

$ echo 64 65536 1024 1024 > /proc/sys/kernel/sem

but you should investigate on the reasons about this behaviour.
ipcs could help on check which process is taking so much resources...
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11921237
That link is unresponsive for me.  I can't even resolve the domain via dns.  Sure thats the right link?
I'll again later too
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11921244
here is my /proc/sys/kernel/sem:
   512     256000  128     1024
Shouldn't that have likely been sufficient? (this is basically a brand new barebones system, was just starting to set up the web services on it)

When the error occurred, i tried using ipcs to track down the resource hog, but it failed to turn up anything.  output that i got:

$ ipcs -s
------ Semaphore Arrays --------
key        semid      owner      perms      nsems      status

Any further ideas?
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 2

Expert Comment

by:brozzis
ID: 11921333

uh, big values there! this is getting interesting...
please post also /proc/sys/kernel/shmmax.

versione of apache ?
have you compiled apache by yourself ? if yes, which mpm used ? could you post your "configure" string ?
have you modified your httpd.conf in the mpm sections (number of processes and so on?)


0
 
LVL 1

Author Comment

by:jtrunek
ID: 11922505
I realize those are big values.  Thats the default that came setup on this User Mode Linux VM, and I haven't bothered tweaking it yet.  /proc/sys/kernel/shmmax is 33554432.

I compiled Apache 1.3.31 and used the default mpm (prefork).  Unfortunately, I don't seem to have the configure string anymore, but I think it was:
./configure \
--prefix=/usr/local/apache \
--enable-module=ssl \
--enable-shared=ssl \
--enable-shared=max \
--enable-module=so

default httpd.conf in mpm section
0
 
LVL 19

Expert Comment

by:ramazanyich
ID: 11923033
I found that link using www.google.com. put your error response which you have and you wil get a lot of links with such a problem
0
 
LVL 2

Expert Comment

by:brozzis
ID: 11923513

no strange third part modules ?
no other processes on this machine ?

shmmax is on a reasonable value. small in respect to other values.
Personally, I'd try to reduce other values, to something like:

echo 250 32000 100 128 > /proc/sys/kernel/sem
echo 536870912 > /proc/sys/kernel/shmmax
echo 4096 > /proc/sys/kernel/shmmni
echo 2097152 > /proc/sys/kernel/shmall

that is a quite reasonable configuration, just to be sure that error isn't in there.
Try to post also a ipcs -l, perhaps we are missing something.
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11924460
no strange3rd party modules.  only ones were mod_ssl and php5.

no other oddball processes either. just sendmail and the typical system services like syslog and cron.

$ ipcs -l

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1

------ Semaphore Limits --------
max number of arrays = 2048
max semaphores per array = 1024
max semaphores system wide = 512000
max ops per semop call = 256
semaphore max value = 32767

------ Messages: Limits --------
max queues system wide = 16
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384
0
 
LVL 2

Accepted Solution

by:
brozzis earned 250 total points
ID: 11924667

well, I don't like these values:

------ Semaphore Limits --------
max number of arrays = 2048
max semaphores per array = 1024
max semaphores system wide = 512000
max ops per semop call = 256
semaphore max value = 32767

I think that "max semaphores system wide " should be equal to "max number of arrays" * "max semaphores per array"
2048 * 1024 = 2097152

please change values and retry.
I'd reduce values, alternatively set "max semaphores system wide" to 2097152

echo 250     32000   32      128 > /proc/sys/kernel/sem


For your reference, I have these values:

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767
0
 
LVL 1

Author Comment

by:jtrunek
ID: 11945908
I'll do that, but as hard as I try, I can't seem to reproduce this problem.  Since its nearly impossible to troubleshoot a problem that cant be reproduced, I'm going to award the points to brozzis for his help so far.
0

Featured Post

Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

Join & Write a Comment

As Wikipedia explains 'robots.txt' as -- the robot exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a websit…
If your site has a few sections that need to be secure when data is transmitted between the server and local computer, such as a /order/ section for ordering or /customer/ which contains customer data, etc it would of course be recommended to secure…
Internet Business Fax to Email Made Easy - With eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…
You have products, that come in variants and want to set different prices for them? Watch this micro tutorial that describes how to configure prices for Magento super attributes. Assigning simple products to configurable: We assigned simple products…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now