FFT
asked on
Qmail Supervise : no way to kill it
Hello,
Our QMail installation run under svscan (daemontools v0.76).
We had a crash 1 month ago (overload), I managed to put everything back up and it's working fine, but for obvious reason there is no way to kill qmail processes (to clean the qmail queue for example), i tried this so far :
kill -9 [PID] -> using all of the below PIDS as they are displayed via "ps -fx" command
svs -k qmail-pop3d
svs -k qmail-send
svs -k qmail-smtpd
None ove the above commands give results (or errors), the svscan relaunch itself with a new PID and all subsequent services (qmail daemons)
Here is the output of 'ps -fx' (only on the qmail part), you'll see that readproctitle gives an error, i don't know if is related to my problem but I remember that before the crash the kill commands I use before worked (I then restarted Qmail using the following commands : "csh -cf '/command/svscanboot &'" to launch dameontools without the need of reboot and then "/etc/init.d/qmail restart" wich worked fine)
30526 ? S 0:00 /bin/sh /command/svscanboot
30528 ? S 0:00 \_ svscan /service
2965 ? S 0:00 | \_ supervise qmail-pop3d
29620 ? Z 0:00 | | \_ [tcpserver <defunct>]
2967 ? S 0:00 | \_ supervise log
2969 ? S 0:00 | \_ supervise log
2971 ? S 0:00 | \_ supervise qmail-smtpd
29619 ? Z 0:00 | | \_ [tcpserver <defunct>]
2973 ? S 0:00 | \_ supervise log
5098 ? S 0:00 | \_ supervise qmail-send
30532 ? S 0:00 \_ readproctitle service errors: ..............supervise: fatal: unable to acquire qmail-pop3d/supervise/lock
So how do I kill this tricky stuff ? ;-)
Thanks for your help
Our QMail installation run under svscan (daemontools v0.76).
We had a crash 1 month ago (overload), I managed to put everything back up and it's working fine, but for obvious reason there is no way to kill qmail processes (to clean the qmail queue for example), i tried this so far :
kill -9 [PID] -> using all of the below PIDS as they are displayed via "ps -fx" command
svs -k qmail-pop3d
svs -k qmail-send
svs -k qmail-smtpd
None ove the above commands give results (or errors), the svscan relaunch itself with a new PID and all subsequent services (qmail daemons)
Here is the output of 'ps -fx' (only on the qmail part), you'll see that readproctitle gives an error, i don't know if is related to my problem but I remember that before the crash the kill commands I use before worked (I then restarted Qmail using the following commands : "csh -cf '/command/svscanboot &'" to launch dameontools without the need of reboot and then "/etc/init.d/qmail restart" wich worked fine)
30526 ? S 0:00 /bin/sh /command/svscanboot
30528 ? S 0:00 \_ svscan /service
2965 ? S 0:00 | \_ supervise qmail-pop3d
29620 ? Z 0:00 | | \_ [tcpserver <defunct>]
2967 ? S 0:00 | \_ supervise log
2969 ? S 0:00 | \_ supervise log
2971 ? S 0:00 | \_ supervise qmail-smtpd
29619 ? Z 0:00 | | \_ [tcpserver <defunct>]
2973 ? S 0:00 | \_ supervise log
5098 ? S 0:00 | \_ supervise qmail-send
30532 ? S 0:00 \_ readproctitle service errors: ..............supervise: fatal: unable to acquire qmail-pop3d/supervise/lock
So how do I kill this tricky stuff ? ;-)
Thanks for your help
Duncan Roe
You could try taking the system sown to single-user mode with "telinit s". Then you could perform your housekeeping, and then reboot.
ASKER
Unfortunately it is our production server, and getting it down is not really the first option it think ok (and it is located on a dedicated datafloor 10 kilometers away...) can you think of an alternative ? ;-)
Thanks
Thanks
The general method would be to mv binaries to their name with some suffix (.backup, .real &c) and replace them with shell scripts which sleep for a long time. Then kill the process hierarchy and the scripts will get run in their place.
Does init start svscanboot? Perhaps you could modify inittab.
Does init start svscanboot? Perhaps you could modify inittab.
ASKER
I see your point but what should contain these shell scripts ?
Yes Svscanboot is launched at startup, but since I do not wish to reboot...
/etc/inittab
[...]
SV:123456:respawn:/command
[...]
Could you precise a little bit ?
Thanks for your help
Bye
Yes Svscanboot is launched at startup, but since I do not wish to reboot...
/etc/inittab
[...]
SV:123456:respawn:/command
[...]
Could you precise a little bit ?
Thanks for your help
Bye
Hi,
I faced the same problem in my Server...
i did this way...
1) Kill all the "qmail" related process
2) Comment out the following line in "/etc/inittab"...
SV:123456:respawn:/command
3) # init q
4) Rebuid the binary.
5) Enable the above commented line in "/etc/inittab"
6) #init q
7) start Qmail using "qmailctl" script or ur customised script.
Hope this will solve ur problem.
If this will not solve the issue make sure u have correct error patches installed for "Qmail","Daemon-tool" Etc...
You can get the pathes from here...
http://djbware.csi.hu/patches/
All the best.
Thanks
Deepak
I faced the same problem in my Server...
i did this way...
1) Kill all the "qmail" related process
2) Comment out the following line in "/etc/inittab"...
SV:123456:respawn:/command
3) # init q
4) Rebuid the binary.
5) Enable the above commented line in "/etc/inittab"
6) #init q
7) start Qmail using "qmailctl" script or ur customised script.
Hope this will solve ur problem.
If this will not solve the issue make sure u have correct error patches installed for "Qmail","Daemon-tool" Etc...
You can get the pathes from here...
http://djbware.csi.hu/patches/
All the best.
Thanks
Deepak
Edit /etc/inittab - comment out the line that fires up svscanboot. By the way, you should not start it in runlevels 1 or 6(probably).
Once you have done that, send a signal to init:
kill -HUP 1
This will cause init to reread /etc/inittab. It will kill svscanboot since this is no longer in its table. After that you can kill the qmail processes. Do your housekeeping, uncomment the line in inittab, kill -HUP 1 again, and the whole show should restart.
Once you have done that, send a signal to init:
kill -HUP 1
This will cause init to reread /etc/inittab. It will kill svscanboot since this is no longer in its table. After that you can kill the qmail processes. Do your housekeeping, uncomment the line in inittab, kill -HUP 1 again, and the whole show should restart.
ASKER
Ok thanks, I will test all of that tonight, but I'm a bit afraid of the 'kill -HUP 1' stuff, will it make my server reboot and Kill all other running process (Apache, Mysql, etc...) ?
Also About rebuilding binaries I use qmail-scanner (wich mcafee anti virus) and spam assassin bind to qmail, I used the following page to do it : http://sylvestre.ledru.info/howto/howto_qmail_vpopmail.php. So, will it kill all my current configuration if I rebuild them ? (pop accounts, etc.).
Also About rebuilding binaries I use qmail-scanner (wich mcafee anti virus) and spam assassin bind to qmail, I used the following page to do it : http://sylvestre.ledru.info/howto/howto_qmail_vpopmail.php. So, will it kill all my current configuration if I rebuild them ? (pop accounts, etc.).
kill -HUP 1 is a pretty standard thing to do. See man 8 init.
Actually I see the modern way to do it is "telinit q", having checked the man page myself.
Actually I see the modern way to do it is "telinit q", having checked the man page myself.
ASKER
Hello,
I did not have the time to make the test yesterday, but I would like to have some precision before to do this, since I'm will do this thru an SSH session, I assume I'll lost it as soon as I enter 'telinit q" right ? This sounds like a reboot to me and I'm not sure that all services are configured to start automatically after this (especially SSH server wich would force me to drive 10 km away ;-)
I did not have the time to make the test yesterday, but I would like to have some precision before to do this, since I'm will do this thru an SSH session, I assume I'll lost it as soon as I enter 'telinit q" right ? This sounds like a reboot to me and I'm not sure that all services are configured to start automatically after this (especially SSH server wich would force me to drive 10 km away ;-)
No, it's nothing like a reboot. All it does is get init to re-read /etc/inittab. If services are no longer listed that used to be, then init kills them (sends TERM signal and if they don't exit in a few seconds then sends KILL). If services are now listed that didn't used to be, then it starts them. It leaves the rest *alone*. I use it frequently at work (actually kill -HUP 1, I haven't tried telinit q yet). What I *don't* know is how powerful is svscanboot. Does that kill services under its control when it gets a TERM signal? Your doco should tell you if it does. How many services do you have under its control anyway?
ASKER
My /etc/inittab :
id:3:initdefault:
si::sysinit:/etc/rc.d/rc.s
l0:0:wait:/etc/rc.d/rc 0
l1:1:wait:/etc/rc.d/rc 1
l2:2:wait:/etc/rc.d/rc 2
l3:3:wait:/etc/rc.d/rc 3
l4:4:wait:/etc/rc.d/rc 4
l5:5:wait:/etc/rc.d/rc 5
l6:6:wait:/etc/rc.d/rc 6
ca::ctrlaltdel:/sbin/shutd
pf::powerfail:/sbin/shutdo
pr:12345:powerokwait:/sbin
1:2345:respawn:/sbin/minge
2:2345:respawn:/sbin/minge
3:2345:respawn:/sbin/minge
4:2345:respawn:/sbin/minge
5:2345:respawn:/sbin/minge
6:2345:respawn:/sbin/minge
x:5:respawn:/etc/X11/prefd
SV:123456:respawn:/command
id:3:initdefault:
si::sysinit:/etc/rc.d/rc.s
l0:0:wait:/etc/rc.d/rc 0
l1:1:wait:/etc/rc.d/rc 1
l2:2:wait:/etc/rc.d/rc 2
l3:3:wait:/etc/rc.d/rc 3
l4:4:wait:/etc/rc.d/rc 4
l5:5:wait:/etc/rc.d/rc 5
l6:6:wait:/etc/rc.d/rc 6
ca::ctrlaltdel:/sbin/shutd
pf::powerfail:/sbin/shutdo
pr:12345:powerokwait:/sbin
1:2345:respawn:/sbin/minge
2:2345:respawn:/sbin/minge
3:2345:respawn:/sbin/minge
4:2345:respawn:/sbin/minge
5:2345:respawn:/sbin/minge
6:2345:respawn:/sbin/minge
x:5:respawn:/etc/X11/prefd
SV:123456:respawn:/command
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Ok duncan, everything went fine as you explained, svscanboot was killed and merelly all other supervised qmail processes went down as well, I just killed the tcpserver to be sure everything went down and restarted everything back with no problem, everything is working fine now ! Thanks very much for your help.
I would like to also thanks deepakjena_2003 about his(her ?) suggestions.
Have a good day.
I would like to also thanks deepakjena_2003 about his(her ?) suggestions.
Have a good day.