Solved

how to find total number of files opened

Posted on 2010-09-24
9
949 Views
Last Modified: 2013-12-16
Hello,

Today I got an escalation stating the following on one server:

We cannot execute *ANY* commands and looks like it will require a Linux server bounce.

orabi@mialsb02-t1$ top
ksh: top: /usr/bin/top: cannot execute [Too many open files in system]
dwilliams@mialsb02-t1$ ls -l
ksh: ls: /bin/ls: cannot execute [Too many open files in system]

After seeing this, I some how login to the server and increased the "file-max" parameter by executing the following command:  echo "75536" > /proc/sys/fs/file-max
Earlier the file-max value was showing as 65536.  After increasing the file-max parameter, am able to execute the commands and its stopped giving the "Too many files opened error".
However I want to know what is the current status of opened files. I did a count of 'lsof' output (lsof | wc -l) but it shows value less than 8000. I'm not sure if thats the right value, if that would have been the case, I shouldn't be getting "Too many files opened error" since I have so many left. Please let me what is the current way to check the number of files opened in a Linux system and how would I find those.

Thanks.
0
Comment
Question by:ashsysad
9 Comments
 
LVL 7

Expert Comment

by:ziceva
Comment Utility
The command is
lsof
(list open files)

lsof | wc -l

should give you a close enough number of open files

This way you can also see witch process opens that many files ....
0
 
LVL 7

Expert Comment

by:ziceva
Comment Utility
sorry ... I didn't read all the way through ...
0
 

Author Comment

by:ashsysad
Comment Utility
Hello, I did the same but that value is far less than the file-max parameter (which was 65536).  Then why I got the "Too many files opened" error? Any idea ?
0
 
LVL 7

Expert Comment

by:ziceva
Comment Utility
try

ulimit -u

This will give you the maximum number of files the current user can open

ulimit -u [number]

Will set this param to [number]
0
Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

 
LVL 7

Expert Comment

by:namol
Comment Utility
When the issue was happening were you able to do a ps command? What does the crontab look like? are there a lot of processes setup to run in there? To find out why this happened you need to start looking over your logs and see what might have been causing the error.
0
 

Author Comment

by:ashsysad
Comment Utility
Hi Namol, Is there a way that I can find the total count of number of files opened in a system by all users?
0
 
LVL 22

Accepted Solution

by:
pjedmond earned 500 total points
Comment Utility
I suspect that we may be looking at the problem from a slightly too simplistic perspective. The issue here is that linux abstracts most things including devices, sockets and pipes as files.

lsof | wc -l will give you a count of the number of files open, but it is possible for a single file to be opened mutiple times for reading, Each each additional concurrent open will use up your file-max. Connections to ports can also eat into your file-max, although this requires sustained connections as network ports are normally configured to timeout after a specified period of time.

However the good news is that lsof can help us identify where the problem might be, as it has options to analyse specific subcategories of files. Because there are so many ways that this type of scenario can arise, either from an attempted denial of service through to a rogue application, you will need to look your system logs in order to get a better feel for what area the problem might have occurred in. You can then use lsof to get better visibility of files associated with networks/data files/devices/pipes etc. Have a look at:

http://sial.org/howto/debug/unix/lsof/
http://www.novell.com/coolsolutions/tip/18078.html

which give you a number of ways in which the lsof options enable you to break down file activity associated with network activity and file activity.

HTH:)

(   (()
(`-' _\
 ''  ''

0
 
LVL 19

Expert Comment

by:jools
Comment Utility
The last time I had this (and it was a few years ago now) it was definately ulimit related (someone mentioned this above).

check your settings, eg/
# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 8189
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 8189
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

There are hard and soft limits, check the man page for details `man ulimit`
0
 

Author Closing Comment

by:ashsysad
Comment Utility
Thanks for explaining me in detail. Now I can figure out where the issue is.
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

Introduction We as admins face situation where we need to redirect websites to another. This may be required as a part of an upgrade keeping the old URL but website should be served from new URL. This document would brief you on different ways ca…
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now