Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

attack of the killer zombies.

Posted on 2000-03-10
4
Medium Priority
?
555 Views
Last Modified: 2013-12-15
here's a nutkicker. in my five years with linux, i've never had this problem before.

this is a plain jane RedHat 6.1 (no upgrades except for PHP3 and mysql)

i come to work, the computer is 'frozen'. (i use kde and rarely log off. last night i had run several programs that interact with MySQL running on the same server. number of select statements == 100,000 if not more.)

i had to resort to telnetting in because tty7 was frozen and couldn't do ctrl-alt-Fn

i tried kill -9 <zombiePID>. no kill. (i guess i needed a wooden stick!)

i had about 38 processes, 33 of em were zombies. including all forks of httpd, mysqld, smbd etc..

did sync and ran a script that goes:
#!/bin/bash
sleep 60 #so that i get time to get out of telnet before the halt begins.
halt

typed:
nohup <scriptname> &
exit

nothing happened. i log back in thru telnet, turns out that script had been zombified too. what's up?
the only way i could turn that sucker off (to kill those zombies) was to flip the power switch at the back. now i'm waiting for fsck to finish checking some 6-8Gigs of harddisk.

anyone have any ideas of what to do in this kind of situation? why this happened in the first place?


thanks.
0
Comment
Question by:aaryal
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
4 Comments
 
LVL 2

Accepted Solution

by:
bernardh earned 300 total points
ID: 2606093
a process started by the kernel like init, which you have no control must have died. that's why even if you kill the parent process of those zombies nothing will happen.

if the process seems to have no parent, kill -s SIGHUP 1 will probably clean up zombies. the command will send a hungup signal to init.
0
 
LVL 2

Expert Comment

by:bernardh
ID: 2606136
Another culprit might be the kernel daemon. Kerneld has forked request-route and not bothered to wait for it to terminate. Kerneld is still there, and the request-route which is marked as zombie will go away as soon as kerneld do a wait for it. An easy way to force kerneld to do that is to just kill it.
0
 
LVL 2

Author Comment

by:aaryal
ID: 2606183
i'm taking your word for it. i have to way to test this theory until this things happens again and since it only happened once in 5 years....

but then probability theory (the no-memory property of some distribution, i forget.) dictates that it could happen again soon :)

but seems like a logical thing to do.

thanks bro,
anoop
0
 
LVL 2

Author Comment

by:aaryal
ID: 2612788
well, whaddya know!! it happened again. and the kill -s SIGHUP 1 didn't work. nor did killing kerneld.

this time, someone put an sql statement in an infinite loop. and that zombified a mysqld process. then, although there weren't any other zombies, the system essentially 'froze'. not exactly, since, we had control over everything except for process management. ie. kill

running processes was not a problem. couldn't run anything in the background.

that's a very weird problem.

0

Featured Post

Veeam Disaster Recovery in Microsoft Azure

Veeam PN for Microsoft Azure is a FREE solution designed to simplify and automate the setup of a DR site in Microsoft Azure using lightweight software-defined networking. It reduces the complexity of VPN deployments and is designed for businesses of ALL sizes.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Setting up Secure Ubuntu server on VMware 1.      Insert the Ubuntu Server distribution CD or attach the ISO of the CD which is in the “Datastore”. Note that it is important to install the x64 edition on servers, not the X86 editions. 2.      Power on th…
In the first part of this tutorial we will cover the prerequisites for installing SQL Server vNext on Linux.
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
Suggested Courses

636 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question