asked on "No space left on device" and "Too many open file" errors
Some of my appliciations gets "No space left on device" and "Too many open file" errors.
Kernel Linux 2.6.18 Centos EL5
# ulimit
unlimited
# sysctl -a |grep file
fs.file-max = 1616310
fs.file-nr = 10048 0 1616310
# lsof -n | wc -l
14186
I have increased file-max.
# sysctl -a |grep file
fs.file-max = 6465240
fs.file-nr = 8128 0 6465240
Errors are still exist.
# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/cciss/c0d1p1 2560864 53916 2506948 3% /
/dev/cciss/c0d1p5 103186432 7420 103179012 1% /var
/dev/cciss/c0d1p2 1280000 29 1279971 1% /tmp
/dev/cciss/c0d0p1 50200 35 50165 1% /boot
tmpfs 193845 1 193844 1% /dev/shm
# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/cciss/c0d1p1 9920592 2702068 6706456 29% /
/dev/cciss/c0d1p5 399696092 84475200 294589820 23% /var
/dev/cciss/c0d1p2 4956316 142068 4558416 4% /tmp
/dev/cciss/c0d0p1 194442 11194 173209 7% /boot
tmpfs 8316692 0 8316692 0% /dev/shm
And error is not a disk space issue actually.
Some other info:
# free
total used free shared buffers cached
Mem: 16633388 15818376 815012 0 107792 14617052
-/+ buffers/cache: 1093532 15539856
Swap: 11477420 2159604 9317816
What is the problem? I can provide other values also.
Kernel Linux 2.6.18 Centos EL5
# ulimit
unlimited
# sysctl -a |grep file
fs.file-max = 1616310
fs.file-nr = 10048 0 1616310
# lsof -n | wc -l
14186
I have increased file-max.
# sysctl -a |grep file
fs.file-max = 6465240
fs.file-nr = 8128 0 6465240
Errors are still exist.
# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/cciss/c0d1p1 2560864 53916 2506948 3% /
/dev/cciss/c0d1p5 103186432 7420 103179012 1% /var
/dev/cciss/c0d1p2 1280000 29 1279971 1% /tmp
/dev/cciss/c0d0p1 50200 35 50165 1% /boot
tmpfs 193845 1 193844 1% /dev/shm
# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/cciss/c0d1p1 9920592 2702068 6706456 29% /
/dev/cciss/c0d1p5 399696092 84475200 294589820 23% /var
/dev/cciss/c0d1p2 4956316 142068 4558416 4% /tmp
/dev/cciss/c0d0p1 194442 11194 173209 7% /boot
tmpfs 8316692 0 8316692 0% /dev/shm
And error is not a disk space issue actually.
Some other info:
# free
total used free shared buffers cached
Mem: 16633388 15818376 815012 0 107792 14617052
-/+ buffers/cache: 1093532 15539856
Swap: 11477420 2159604 9317816
What is the problem? I can provide other values also.
Linux DistributionsLinux
Last Comment
seaquesttr
ASKER CERTIFIED SOLUTION
Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
Under which user's credentials does the application in question run i.e. are you talking about a web server (apache) mysql (mysql) etc.?
>> error is not a disk space issue actually <<
Maybe disk quotas? Issue "quota" to check.
ASKER
Disk quotas are not enabled.
Error is system wide. I see the error during installination of a rpm as root, during squid operation as squid and tomcat-apache operation as apache.
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 137215
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 137215
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Where can I see current used values of each of the ulimit value? (the ones with some limits)
Error is system wide. I see the error during installination of a rpm as root, during squid operation as squid and tomcat-apache operation as apache.
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 137215
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 137215
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Where can I see current used values of each of the ulimit value? (the ones with some limits)
This is the best money I have ever spent. I cannot not tell you how many times these folks have saved my bacon. I learn so much from the contributors.
rwheeler23
do you have any other error messages in /var/log/squid/access.log store.log /var/log/messages /var/log/audit/audit.log?
lsof -u <username> | wc -l
will produce a count of all open files by username (note pipes, etc are counted as files)
remove the -u <username> and you'll get a system wide total.
Which applications do you run as root when you get these messages?
lsof -u <username> | wc -l
will produce a count of all open files by username (note pipes, etc are counted as files)
remove the -u <username> and you'll get a system wide total.
Which applications do you run as root when you get these messages?
ASKER
Lots of application runs as root.
# lsof -n | wc -l
14186
# lsof -n | wc -l
14186
SOLUTION
Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a questionI have seen squid do that when the authenticator processes have some issue.
The squid logs will probably tell you more.
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
ASKER
This was about default value of 1024 ulimit nofile value. I have written limits.conf to coreect the issue
/etc/security/limits.conf
* soft nofile 4096
* hard nofile 9192
/etc/security/limits.conf
* soft nofile 4096
* hard nofile 9192