server would not shutdown and lots of lsmod procs.... any idea what this is?

I was trying to shut down a Red Hat five server and it still had not shut down by the next day.  I noticed a ton of these lsmod procs.... Not sure if it's related....  It just so happens I was shutting it down to replace the operating system so there is nothing to fix I guess but I'm still curious what this might of been about had this been a server that I needed to continue using.  




t      4714  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      4800  0.0  0.0   3796   384 ?        D<   15:18   0:00 /sbin/lsmod
root      5132  0.0  0.0   3796   380 ?        D<   Aug08   0:00 /sbin/lsmod
root      5239  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root      5281  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      5348  0.0  0.0   3796   384 ?        D<   15:38   0:00 /sbin/lsmod
root      5704  0.0  0.0   3796   380 ?        D<   Aug08   0:00 /sbin/lsmod
root      5787  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root      5825  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      5903  0.0  0.0   3796   384 ?        D<   15:58   0:00 /sbin/lsmod
root      6247  0.0  0.0   3796   380 ?        D<   Aug08   0:00 /sbin/lsmod
root      6331  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      6365  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root      6485  0.0  0.0   3796   380 ?        D<   16:18   0:00 /sbin/lsmod
root      6855  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root      6982  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root      7008  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      7108  0.0  0.0   3796   384 ?        D<   16:38   0:00 /sbin/lsmod
root      7420  0.0  0.0   3796   388 ?        D<   Aug08   0:00 /sbin/lsmod
root      7524  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root      7547  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root      7653  0.0  0.0   3796   388 ?        D<   16:58   0:00 /sbin/lsmod
root      8061  0.0  0.0   3796   388 ?        D<   Aug09   0:00 /sbin/lsmod
root      8073  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      8078  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root      8224  0.0  0.0   3796   380 ?        D<   17:18   0:00 /sbin/lsmod
root      8608  0.0  0.0   5896   704 pts/4    D+   Aug09   0:00 shutdown -h 0 w
root      8613  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root      8634  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      8661  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      8762  0.0  0.0   3796   384 ?        D<   17:38   0:00 /sbin/lsmod
root      9159  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      9171  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root      9193  0.0  0.0   3796   388 ?        D<   Aug09   0:00 /sbin/lsmod
root      9290  0.0  0.0   3796   384 ?        D<   17:58   0:00 /sbin/lsmod
root      9686  0.0  0.0   3796   388 ?        D<   Aug09   0:00 /sbin/lsmod
root      9698  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root      9727  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root      9844  0.0  0.0   3796   384 ?        D<   18:18   0:00 /sbin/lsmod
root     10230  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root     10290  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root     10766  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root     10799  0.0  0.0   3796   380 ?        D<   Aug08   0:00 /sbin/lsmod
root     10831  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root     11295  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root     11328  0.0  0.0   3796   380 ?        D<   Aug08   0:00 /sbin/lsmod
root     11362  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root     11855  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root     11885  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root     11947  0.0  0.0   3796   388 ?        D<   00:18   0:00 /sbin/lsmod
root     12411  0.0  0.0   3796   388 ?        D<   Aug09   0:00 /sbin/lsmod
root     12455  0.0  0.0   3796   380 ?        D<   Aug08   0:00 /sbin/lsmod
root     12485  0.0  0.0   3796   384 ?        D<   00:38   0:00 /sbin/lsmod
root     12956  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root     12979  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root     13030  0.0  0.0   3796   380 ?        D<   00:58   0:00 /sbin/lsmod
root     13486  0.0  0.0   3796   388 ?        D<   Aug09   0:00 /sbin/lsmod
root     13510  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root     13594  0.0  0.0   3796   380 ?        D<   01:18   0:00 /sbin/lsmod
root     14041  0.0  0.0   3796   388 ?        D<   Aug09   0:00 /sbin/lsmod
root     14074  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root     14130  0.0  0.0   3796   384 ?        D<   01:38   0:00 /sbin/lsmod
root     14573  0.0  0.0   3796   380 ?        D<   Aug09   0:00 /sbin/lsmod
root     14610  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod
root     14663  0.0  0.0   3796   380 ?        D<   01:58   0:00 /sbin/lsmod
root     15101  0.0  0.0   3796   384 ?        D<   Aug09   0:00 /sbin/lsmod
root     15137  0.0  0.0   3796   384 ?        D<   Aug08   0:00 /sbin/lsmod

Open in new window

XetroximynAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

giltjrCommented:
I would expect you have something in chron that is running the lsmod command and has an issue and is not ending successfully.

If you don't have anything scheduled in chron, I would monitor to see if they start popping up again.
0
Gerwin Jansen, EE MVETopic Advisor Commented:
Try killing the lsmod processes first:

killall -e lsmod

Then try shutting down the server, does it shutdown now?
0
Joseph GanSystem AdminCommented:
Your server has a cron job running "lsmod" 3 times every hour, eg 15:18, 15:38 and 15:58 etc since 08th August with some issues, which stop you shutting down the server. Once you kill all the processes, the server will shutdown automatically.
0
Duncan RoeSoftware DeveloperCommented:
Do you have NFS mounts? All manner of processes can get stuck if an NFS server is shut down before clients unmount.
In the meantime, it might be helpful to examine what one of these lsmod processes is doing. Do this from its directory in /proc. Text files in /proc are null-delimited strings (no newlines), so I define a function to cat them:
pcat(){ cat $1|xargs -0 -n1 echo; }
Now, cd /proc/15137 (the last process in your list - as an example).
pcat cmdline to see how lsmod was invoked
ls -l cwd to see process's working directory (cwd is always a symbolic link)
pcat environ|sort to see environment variables
And so on. Post anything that looks at all suspicious (for example, mentions a directory that is NFS mounted)
If you can get a spare terminal, you could try invoking lsmod yourself and see what happens
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
XetroximynAuthor Commented:
thanks all!  this was an old server we migrated our app off of, and I was just shutting it down for the last time.  Was just curious what this might be about.  (But can't actually do any troubleshooting, as I just hard-shutdown.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.