Link to home
Start Free TrialLog in
Avatar of CityInfoSys
CityInfoSysFlag for United States of America

asked on

Apache Issues

Suppose you had a LAMP stack, and the Apache portion of this LAMP stack would work fine and serve a bunch of pages for a few days, but then it would randomly stop serving pages, and then when you tried to do a graceful Apache restart of this A portion of the LAMP stack, it wouldn't work, so you'd have to completely reboot the entire server to get it to work. Any idea on what is happening? We are using a apache2ctl restart command but it does not work so we have to reboot the server.

This is a SuSE Linux box.
Avatar of Jason Carson
Jason Carson
Flag of Canada image

I have very little experience with suse in particular but is their anything in the log files that will help point you in the right direction and give you an idea of what is going wrong?
Avatar of CityInfoSys

ASKER

The log file makes no indication of what is causing the issue it is totally random.
Avatar of Dr. Klahn
Dr. Klahn

Use apache2ctl stop and apache2ctl start instead.  This will break existing sessions and invalidate cookies, but it should always work.  Restart just tickles the running server.  

Check the memory usage on the system when the problem occurs and see if the system is page thrashing or using excessive pagefile.
you'd need to debug when the problem occurs

post the output of "top" and "ps -auxww | grep httpd", try apachetop as well

try to kill the apache processes in other (more violent) ways such as "pkill httpd" or "pkill -9 httpd"

there are quite a few reasons why an apache process may become unresponsive... do you serve files over NFS ? do you have php processes that may last forever and clutter server threads ? is the machine properly responsive when apache dies ? which threading mechanism do you use ? mpm_prefork ?
On the latest crash Apache2ctl stop and start were successful in stopping and starting the processes, but I still couldn't access the website. Only a full off/on brought it back. Here are the outputs you requested from the most resent crashes.

Please scroll down it posted the photos differently.

ps -auxww | grep httpd
 User generated imageps -auxww | grep httpd
User generated imageTop
User generated image
I do have to say I'm curious about how an Apache system is using up 5 GB of the available 8 GB of memory (according to top).  The processes shown on the top display account for around 2% of the 8 GB.
given the output of ps, it does not seem apache is the one using up 5Gb of ram ( unless i misread something )... a complete "ps -faux" or "pstree" might shed some light regarding what is actually using up the 5GB if necessary

... and there is apparently no resource shortage either ( cpu ok, ram ok ), no long wait for IOs ( waiting for a slow remote filesystem would produce a higher load average )...

my guess would be one of :
- either a script runs with no timeout and clutters all the available threads... looks unlikely and should be visible in the logs and/or apachetop or the likes so
- or apache is serving files from a mounted remote filesystem ( most likely NFS ) and the mount point broke
- if something like heartbeat is in use, network issues could arise as well and be solved by a reboot but you'd most likely have noticed a long time ago

when you reach a page when the problem is occuring
- does apache log the query ?
- does it actually answer with a blank page ? an error ? no answer at all ?

if there is no log and no answer, you might want to check there is no weird network issue : try the page locally with wget, or maybe with "apachectl status", or run "tcpdump -i any port 80" and check that your queries reach the server

if the queries do reach the server and unless one of the above information rings a bell, i'd probably go on running an strace on the apache processes while the problem is occuring and possibly sending a few queries manually while the strace is running so we know what apache is doing. if the httpd processes are locked, their status in top is an indication, and the current system call should be output by strace
ASKER CERTIFIED SOLUTION
Avatar of skullnobrains
skullnobrains

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
did you find what was wrong ?

if yes i'm interested, and others might be

if no you can accept your own solution for 0 points cost