Solved

server would not shutdown and lots of lsmod procs.... any idea what this is?

Posted on 2016-08-10
5
75 Views
Last Modified: 2016-08-20
I was trying to shut down a Red Hat five server and it still had not shut down by the next day.  I noticed a ton of these lsmod procs.... Not sure if it's related....  It just so happens I was shutting it down to replace the operating system so there is nothing to fix I guess but I'm still curious what this might of been about had this been a server that I needed to continue using.  




t      4714  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      4800  0.0  0.0   3796   384 ?        D<   15:18   0:00 /sbin/lsmod
root      5132  0.0  0.0   3796   380 ?        D<   Aug08   0:00 /sbin/lsmod
root      5239  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root      5281  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      5348  0.0  0.0   3796   384 ?        D<   15:38   0:00 /sbin/lsmod
root      5704  0.0  0.0   3796   380 ?        D<   Aug08   0:00 /sbin/lsmod
root      5787  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root      5825  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      5903  0.0  0.0   3796   384 ?        D<   15:58   0:00 /sbin/lsmod
root      6247  0.0  0.0   3796   380 ?        D<   Aug08   0:00 /sbin/lsmod
root      6331  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      6365  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root      6485  0.0  0.0   3796   380 ?        D<   16:18   0:00 /sbin/lsmod
root      6855  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root      6982  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root      7008  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      7108  0.0  0.0   3796   384 ?        D<   16:38   0:00 /sbin/lsmod
root      7420  0.0  0.0   3796   388 ?        D<   Aug08   0:00 /sbin/lsmod
root      7524  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root      7547  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root      7653  0.0  0.0   3796   388 ?        D<   16:58   0:00 /sbin/lsmod
root      8061  0.0  0.0   3796   388 ?        D<   Aug09   0:00 /sbin/lsmod
root      8073  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      8078  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root      8224  0.0  0.0   3796   380 ?        D<   17:18   0:00 /sbin/lsmod
root      8608  0.0  0.0   5896   704 pts/4    D+   Aug09   0:00 shutdown -h 0 w
root      8613  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root      8634  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      8661  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      8762  0.0  0.0   3796   384 ?        D<   17:38   0:00 /sbin/lsmod
root      9159  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      9171  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root      9193  0.0  0.0   3796   388 ?        D<   Aug09   0:00 /sbin/lsmod
root      9290  0.0  0.0   3796   384 ?        D<   17:58   0:00 /sbin/lsmod
root      9686  0.0  0.0   3796   388 ?        D<   Aug09   0:00 /sbin/lsmod
root      9698  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root      9727  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      9844  0.0  0.0   3796   384 ?        D<   18:18   0:00 /sbin/lsmod
root     10230  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root     10290  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root     10766  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root     10799  0.0  0.0   3796   380 ?        D<   Aug08   0:00 /sbin/lsmod
root     10831  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root     11295  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root     11328  0.0  0.0   3796   380 ?        D<   Aug08   0:00 /sbin/lsmod
root     11362  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root     11855  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root     11885  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root     11947  0.0  0.0   3796   388 ?        D<   00:18   0:00 /sbin/lsmod
root     12411  0.0  0.0   3796   388 ?        D<   Aug09   0:00 /sbin/lsmod
root     12455  0.0  0.0   3796   380 ?        D<   Aug08   0:00 /sbin/lsmod
root     12485  0.0  0.0   3796   384 ?        D<   00:38   0:00 /sbin/lsmod
root     12956  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root     12979  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root     13030  0.0  0.0   3796   380 ?        D<   00:58   0:00 /sbin/lsmod
root     13486  0.0  0.0   3796   388 ?        D<   Aug09   0:00 /sbin/lsmod
root     13510  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root     13594  0.0  0.0   3796   380 ?        D<   01:18   0:00 /sbin/lsmod
root     14041  0.0  0.0   3796   388 ?        D<   Aug09   0:00 /sbin/lsmod
root     14074  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root     14130  0.0  0.0   3796   384 ?        D<   01:38   0:00 /sbin/lsmod
root     14573  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root     14610  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root     14663  0.0  0.0   3796   380 ?        D<   01:58   0:00 /sbin/lsmod
root     15101  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root     15137  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod

Open in new window

0
Comment
Question by:Xetroximyn
5 Comments
 
LVL 57

Assisted Solution

by:giltjr
giltjr earned 83 total points
Comment Utility
I would expect you have something in chron that is running the lsmod command and has an issue and is not ending successfully.

If you don't have anything scheduled in chron, I would monitor to see if they start popping up again.
0
 
LVL 37

Assisted Solution

by:Gerwin Jansen
Gerwin Jansen earned 83 total points
Comment Utility
Try killing the lsmod processes first:

killall -e lsmod

Then try shutting down the server, does it shutdown now?
0
 
LVL 16

Assisted Solution

by:Joseph Gan
Joseph Gan earned 83 total points
Comment Utility
Your server has a cron job running "lsmod" 3 times every hour, eg 15:18, 15:38 and 15:58 etc since 08th August with some issues, which stop you shutting down the server. Once you kill all the processes, the server will shutdown automatically.
0
 
LVL 34

Accepted Solution

by:
Duncan Roe earned 251 total points
Comment Utility
Do you have NFS mounts? All manner of processes can get stuck if an NFS server is shut down before clients unmount.
In the meantime, it might be helpful to examine what one of these lsmod processes is doing. Do this from its directory in /proc. Text files in /proc are null-delimited strings (no newlines), so I define a function to cat them:
pcat(){ cat $1|xargs -0 -n1 echo; }
Now, cd /proc/15137 (the last process in your list - as an example).
pcat cmdline to see how lsmod was invoked
ls -l cwd to see process's working directory (cwd is always a symbolic link)
pcat environ|sort to see environment variables
And so on. Post anything that looks at all suspicious (for example, mentions a directory that is NFS mounted)
If you can get a spare terminal, you could try invoking lsmod yourself and see what happens
0
 

Author Comment

by:Xetroximyn
Comment Utility
thanks all!  this was an old server we migrated our app off of, and I was just shutting it down for the last time.  Was just curious what this might be about.  (But can't actually do any troubleshooting, as I just hard-shutdown.
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

This document is written for Red Hat Enterprise Linux AS release 4 and ORACLE 10g.  Earlier releases can be installed using this document as well however there are some additional steps for packages to be installed see Metalink. Disclaimer: I hav…
If you use Debian 6 Squeeze and you are tired of looking at the childish graphical GDM login screen that is used by default, here's an easy way to change it. If you've already tried to change it you've probably discovered that none of the old met…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now