Avatar of seaquesttr
seaquesttr
Flag for Turkey asked on

"No space left on device" and "Too many open file" errors

Some of my appliciations gets "No space left on device" and "Too many open file" errors.

Kernel Linux 2.6.18 Centos EL5

# ulimit
unlimited

# sysctl -a |grep file
fs.file-max = 1616310
fs.file-nr = 10048      0       1616310

# lsof -n | wc -l
14186

I have increased file-max.

# sysctl -a |grep file
fs.file-max = 6465240
fs.file-nr = 8128       0       6465240

Errors are still exist.

# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/cciss/c0d1p1    2560864   53916 2506948    3% /
/dev/cciss/c0d1p5    103186432    7420 103179012    1% /var
/dev/cciss/c0d1p2    1280000      29 1279971    1% /tmp
/dev/cciss/c0d0p1      50200      35   50165    1% /boot
tmpfs                 193845       1  193844    1% /dev/shm

# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/cciss/c0d1p1      9920592   2702068   6706456  29% /
/dev/cciss/c0d1p5    399696092  84475200 294589820  23% /var
/dev/cciss/c0d1p2      4956316    142068   4558416   4% /tmp
/dev/cciss/c0d0p1       194442     11194    173209   7% /boot
tmpfs                  8316692         0   8316692   0% /dev/shm

And error is not a disk space issue actually.

Some other info:

# free
             total       used       free     shared    buffers     cached
Mem:      16633388   15818376     815012          0     107792   14617052
-/+ buffers/cache:    1093532   15539856
Swap:     11477420    2159604    9317816


What is the problem? I can provide other values also.

Linux DistributionsLinux

Avatar of undefined
Last Comment
seaquesttr

8/22/2022 - Mon
ASKER CERTIFIED SOLUTION
woolmilkporc

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
arnold

Under which user's credentials does the application in question run i.e. are you talking about a web server (apache) mysql (mysql) etc.?
woolmilkporc


>> error is not a disk space issue actually <<

Maybe disk quotas? Issue "quota" to check.

seaquesttr

ASKER
Disk quotas are not enabled.
Error is system wide. I see the error during installination of a rpm as root, during squid operation as squid and tomcat-apache operation as apache.

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 137215
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 137215
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Where can I see current used values of each of the ulimit value? (the ones with some limits)
This is the best money I have ever spent. I cannot not tell you how many times these folks have saved my bacon. I learn so much from the contributors.
rwheeler23
arnold

do you have any other error messages in /var/log/squid/access.log store.log /var/log/messages /var/log/audit/audit.log?

lsof -u <username> | wc -l
will produce a count of all open files by username (note pipes, etc are counted as files)
remove the -u <username> and you'll get a system wide total.

Which applications do you run as root when you get these messages?
seaquesttr

ASKER
Lots of application runs as root.

# lsof -n | wc -l
14186

SOLUTION
arnold

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
Pieter Jordaan


I have seen squid do that when the authenticator processes have some issue.
The squid logs will probably tell you more.
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
seaquesttr

ASKER
This was about default value of 1024 ulimit nofile value. I have written limits.conf to coreect the issue

/etc/security/limits.conf
* soft nofile 4096
* hard nofile 9192