Link to home
Start Free TrialLog in
Avatar of seaquesttr
seaquesttrFlag for Türkiye

asked on

"No space left on device" and "Too many open file" errors

Some of my appliciations gets "No space left on device" and "Too many open file" errors.

Kernel Linux 2.6.18 Centos EL5

# ulimit
unlimited

# sysctl -a |grep file
fs.file-max = 1616310
fs.file-nr = 10048      0       1616310

# lsof -n | wc -l
14186

I have increased file-max.

# sysctl -a |grep file
fs.file-max = 6465240
fs.file-nr = 8128       0       6465240

Errors are still exist.

# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/cciss/c0d1p1    2560864   53916 2506948    3% /
/dev/cciss/c0d1p5    103186432    7420 103179012    1% /var
/dev/cciss/c0d1p2    1280000      29 1279971    1% /tmp
/dev/cciss/c0d0p1      50200      35   50165    1% /boot
tmpfs                 193845       1  193844    1% /dev/shm

# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/cciss/c0d1p1      9920592   2702068   6706456  29% /
/dev/cciss/c0d1p5    399696092  84475200 294589820  23% /var
/dev/cciss/c0d1p2      4956316    142068   4558416   4% /tmp
/dev/cciss/c0d0p1       194442     11194    173209   7% /boot
tmpfs                  8316692         0   8316692   0% /dev/shm

And error is not a disk space issue actually.

Some other info:

# free
             total       used       free     shared    buffers     cached
Mem:      16633388   15818376     815012          0     107792   14617052
-/+ buffers/cache:    1093532   15539856
Swap:     11477420    2159604    9317816


What is the problem? I can provide other values also.

ASKER CERTIFIED SOLUTION
Avatar of woolmilkporc
woolmilkporc
Flag of Germany image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Under which user's credentials does the application in question run i.e. are you talking about a web server (apache) mysql (mysql) etc.?

>> error is not a disk space issue actually <<

Maybe disk quotas? Issue "quota" to check.

Avatar of seaquesttr

ASKER

Disk quotas are not enabled.
Error is system wide. I see the error during installination of a rpm as root, during squid operation as squid and tomcat-apache operation as apache.

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 137215
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 137215
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Where can I see current used values of each of the ulimit value? (the ones with some limits)
do you have any other error messages in /var/log/squid/access.log store.log /var/log/messages /var/log/audit/audit.log?

lsof -u <username> | wc -l
will produce a count of all open files by username (note pipes, etc are counted as files)
remove the -u <username> and you'll get a system wide total.

Which applications do you run as root when you get these messages?
Lots of application runs as root.

# lsof -n | wc -l
14186

SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial

I have seen squid do that when the authenticator processes have some issue.
The squid logs will probably tell you more.
This was about default value of 1024 ulimit nofile value. I have written limits.conf to coreect the issue

/etc/security/limits.conf
* soft nofile 4096
* hard nofile 9192