Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

how to find total number of files opened

Posted on 2010-09-24
9
Medium Priority
?
1,007 Views
Last Modified: 2013-12-16
Hello,

Today I got an escalation stating the following on one server:

We cannot execute *ANY* commands and looks like it will require a Linux server bounce.

orabi@mialsb02-t1$ top
ksh: top: /usr/bin/top: cannot execute [Too many open files in system]
dwilliams@mialsb02-t1$ ls -l
ksh: ls: /bin/ls: cannot execute [Too many open files in system]

After seeing this, I some how login to the server and increased the "file-max" parameter by executing the following command:  echo "75536" > /proc/sys/fs/file-max
Earlier the file-max value was showing as 65536.  After increasing the file-max parameter, am able to execute the commands and its stopped giving the "Too many files opened error".
However I want to know what is the current status of opened files. I did a count of 'lsof' output (lsof | wc -l) but it shows value less than 8000. I'm not sure if thats the right value, if that would have been the case, I shouldn't be getting "Too many files opened error" since I have so many left. Please let me what is the current way to check the number of files opened in a Linux system and how would I find those.

Thanks.
0
Comment
Question by:ashsysad
9 Comments
 
LVL 7

Expert Comment

by:ziceva
ID: 33755643
The command is
lsof
(list open files)

lsof | wc -l

should give you a close enough number of open files

This way you can also see witch process opens that many files ....
0
 
LVL 7

Expert Comment

by:ziceva
ID: 33755664
sorry ... I didn't read all the way through ...
0
 

Author Comment

by:ashsysad
ID: 33755667
Hello, I did the same but that value is far less than the file-max parameter (which was 65536).  Then why I got the "Too many files opened" error? Any idea ?
0
Free learning courses: Active Directory Deep Dive

Get a firm grasp on your IT environment when you learn Active Directory best practices with Veeam! Watch all, or choose any amount, of this three-part webinar series to improve your skills. From the basics to virtualization and backup, we got you covered.

 
LVL 7

Expert Comment

by:ziceva
ID: 33755698
try

ulimit -u

This will give you the maximum number of files the current user can open

ulimit -u [number]

Will set this param to [number]
0
 
LVL 7

Expert Comment

by:namol
ID: 33755878
When the issue was happening were you able to do a ps command? What does the crontab look like? are there a lot of processes setup to run in there? To find out why this happened you need to start looking over your logs and see what might have been causing the error.
0
 

Author Comment

by:ashsysad
ID: 33756319
Hi Namol, Is there a way that I can find the total count of number of files opened in a system by all users?
0
 
LVL 22

Accepted Solution

by:
pjedmond earned 2000 total points
ID: 33762100
I suspect that we may be looking at the problem from a slightly too simplistic perspective. The issue here is that linux abstracts most things including devices, sockets and pipes as files.

lsof | wc -l will give you a count of the number of files open, but it is possible for a single file to be opened mutiple times for reading, Each each additional concurrent open will use up your file-max. Connections to ports can also eat into your file-max, although this requires sustained connections as network ports are normally configured to timeout after a specified period of time.

However the good news is that lsof can help us identify where the problem might be, as it has options to analyse specific subcategories of files. Because there are so many ways that this type of scenario can arise, either from an attempted denial of service through to a rogue application, you will need to look your system logs in order to get a better feel for what area the problem might have occurred in. You can then use lsof to get better visibility of files associated with networks/data files/devices/pipes etc. Have a look at:

http://sial.org/howto/debug/unix/lsof/
http://www.novell.com/coolsolutions/tip/18078.html

which give you a number of ways in which the lsof options enable you to break down file activity associated with network activity and file activity.

HTH:)

(   (()
(`-' _\
 ''  ''

0
 
LVL 19

Expert Comment

by:jools
ID: 33774445
The last time I had this (and it was a few years ago now) it was definately ulimit related (someone mentioned this above).

check your settings, eg/
# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 8189
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 8189
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

There are hard and soft limits, check the man page for details `man ulimit`
0
 

Author Closing Comment

by:ashsysad
ID: 33775350
Thanks for explaining me in detail. Now I can figure out where the issue is.
0

Featured Post

Free learning courses: Active Directory Deep Dive

Get a firm grasp on your IT environment when you learn Active Directory best practices with Veeam! Watch all, or choose any amount, of this three-part webinar series to improve your skills. From the basics to virtualization and backup, we got you covered.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Over the last ten+ years I have seen Linux configuration tools come and go. In the early days there was the tried-and-true, all-powerful linuxconf that many thought would remain the one and only Linux configuration tool until the end of times. Well,…
If you have a server on collocation with the super-fast CPU, that doesn't mean that you get it running at full power. Here is a preamble. When doing inventory of Linux servers, that I'm administering, I've found that some of them are running on l…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Suggested Courses
Course of the Month10 days, 23 hours left to enroll

885 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question