Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

how to find total number of files opened

Posted on 2010-09-24
9
Medium Priority
?
1,001 Views
Last Modified: 2013-12-16
Hello,

Today I got an escalation stating the following on one server:

We cannot execute *ANY* commands and looks like it will require a Linux server bounce.

orabi@mialsb02-t1$ top
ksh: top: /usr/bin/top: cannot execute [Too many open files in system]
dwilliams@mialsb02-t1$ ls -l
ksh: ls: /bin/ls: cannot execute [Too many open files in system]

After seeing this, I some how login to the server and increased the "file-max" parameter by executing the following command:  echo "75536" > /proc/sys/fs/file-max
Earlier the file-max value was showing as 65536.  After increasing the file-max parameter, am able to execute the commands and its stopped giving the "Too many files opened error".
However I want to know what is the current status of opened files. I did a count of 'lsof' output (lsof | wc -l) but it shows value less than 8000. I'm not sure if thats the right value, if that would have been the case, I shouldn't be getting "Too many files opened error" since I have so many left. Please let me what is the current way to check the number of files opened in a Linux system and how would I find those.

Thanks.
0
Comment
Question by:ashsysad
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
9 Comments
 
LVL 7

Expert Comment

by:ziceva
ID: 33755643
The command is
lsof
(list open files)

lsof | wc -l

should give you a close enough number of open files

This way you can also see witch process opens that many files ....
0
 
LVL 7

Expert Comment

by:ziceva
ID: 33755664
sorry ... I didn't read all the way through ...
0
 

Author Comment

by:ashsysad
ID: 33755667
Hello, I did the same but that value is far less than the file-max parameter (which was 65536).  Then why I got the "Too many files opened" error? Any idea ?
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 7

Expert Comment

by:ziceva
ID: 33755698
try

ulimit -u

This will give you the maximum number of files the current user can open

ulimit -u [number]

Will set this param to [number]
0
 
LVL 7

Expert Comment

by:namol
ID: 33755878
When the issue was happening were you able to do a ps command? What does the crontab look like? are there a lot of processes setup to run in there? To find out why this happened you need to start looking over your logs and see what might have been causing the error.
0
 

Author Comment

by:ashsysad
ID: 33756319
Hi Namol, Is there a way that I can find the total count of number of files opened in a system by all users?
0
 
LVL 22

Accepted Solution

by:
pjedmond earned 2000 total points
ID: 33762100
I suspect that we may be looking at the problem from a slightly too simplistic perspective. The issue here is that linux abstracts most things including devices, sockets and pipes as files.

lsof | wc -l will give you a count of the number of files open, but it is possible for a single file to be opened mutiple times for reading, Each each additional concurrent open will use up your file-max. Connections to ports can also eat into your file-max, although this requires sustained connections as network ports are normally configured to timeout after a specified period of time.

However the good news is that lsof can help us identify where the problem might be, as it has options to analyse specific subcategories of files. Because there are so many ways that this type of scenario can arise, either from an attempted denial of service through to a rogue application, you will need to look your system logs in order to get a better feel for what area the problem might have occurred in. You can then use lsof to get better visibility of files associated with networks/data files/devices/pipes etc. Have a look at:

http://sial.org/howto/debug/unix/lsof/
http://www.novell.com/coolsolutions/tip/18078.html

which give you a number of ways in which the lsof options enable you to break down file activity associated with network activity and file activity.

HTH:)

(   (()
(`-' _\
 ''  ''

0
 
LVL 19

Expert Comment

by:jools
ID: 33774445
The last time I had this (and it was a few years ago now) it was definately ulimit related (someone mentioned this above).

check your settings, eg/
# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 8189
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 8189
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

There are hard and soft limits, check the man page for details `man ulimit`
0
 

Author Closing Comment

by:ashsysad
ID: 33775350
Thanks for explaining me in detail. Now I can figure out where the issue is.
0

Featured Post

Learn how to optimize MySQL for your business need

With the increasing importance of apps & networks in both business & personal interconnections, perfor. has become one of the key metrics of successful communication. This ebook is a hands-on business-case-driven guide to understanding MySQL query parameter tuning & database perf

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Network Interface Card (NIC) bonding, also known as link aggregation, NIC teaming and trunking, is an important concept to understand and implement in any environment where high availability is of concern. Using this feature, a server administrator …
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
Suggested Courses

704 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question