Apache Issues

Suppose you had a LAMP stack, and the Apache portion of this LAMP stack would work fine and serve a bunch of pages for a few days, but then it would randomly stop serving pages, and then when you tried to do a graceful Apache restart of this A portion of the LAMP stack, it wouldn't work, so you'd have to completely reboot the entire server to get it to work. Any idea on what is happening? We are using a apache2ctl restart command but it does not work so we have to reboot the server.

This is a SuSE Linux box.
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Jason CarsonComputer TechnicianCommented:
I have very little experience with suse in particular but is their anything in the log files that will help point you in the right direction and give you an idea of what is going wrong?
CityInfoSysAuthor Commented:
The log file makes no indication of what is causing the issue it is totally random.
Dr. KlahnPrincipal Software EngineerCommented:
Use apache2ctl stop and apache2ctl start instead.  This will break existing sessions and invalidate cookies, but it should always work.  Restart just tickles the running server.  

Check the memory usage on the system when the problem occurs and see if the system is page thrashing or using excessive pagefile.
10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

you'd need to debug when the problem occurs

post the output of "top" and "ps -auxww | grep httpd", try apachetop as well

try to kill the apache processes in other (more violent) ways such as "pkill httpd" or "pkill -9 httpd"

there are quite a few reasons why an apache process may become unresponsive... do you serve files over NFS ? do you have php processes that may last forever and clutter server threads ? is the machine properly responsive when apache dies ? which threading mechanism do you use ? mpm_prefork ?
CityInfoSysAuthor Commented:
On the latest crash Apache2ctl stop and start were successful in stopping and starting the processes, but I still couldn't access the website. Only a full off/on brought it back. Here are the outputs you requested from the most resent crashes.

Please scroll down it posted the photos differently.

ps -auxww | grep httpd
 1.pngps -auxww | grep httpd
Dr. KlahnPrincipal Software EngineerCommented:
I do have to say I'm curious about how an Apache system is using up 5 GB of the available 8 GB of memory (according to top).  The processes shown on the top display account for around 2% of the 8 GB.
given the output of ps, it does not seem apache is the one using up 5Gb of ram ( unless i misread something )... a complete "ps -faux" or "pstree" might shed some light regarding what is actually using up the 5GB if necessary

... and there is apparently no resource shortage either ( cpu ok, ram ok ), no long wait for IOs ( waiting for a slow remote filesystem would produce a higher load average )...

my guess would be one of :
- either a script runs with no timeout and clutters all the available threads... looks unlikely and should be visible in the logs and/or apachetop or the likes so
- or apache is serving files from a mounted remote filesystem ( most likely NFS ) and the mount point broke
- if something like heartbeat is in use, network issues could arise as well and be solved by a reboot but you'd most likely have noticed a long time ago

when you reach a page when the problem is occuring
- does apache log the query ?
- does it actually answer with a blank page ? an error ? no answer at all ?

if there is no log and no answer, you might want to check there is no weird network issue : try the page locally with wget, or maybe with "apachectl status", or run "tcpdump -i any port 80" and check that your queries reach the server

if the queries do reach the server and unless one of the above information rings a bell, i'd probably go on running an strace on the apache processes while the problem is occuring and possibly sending a few queries manually while the strace is running so we know what apache is doing. if the httpd processes are locked, their status in top is an indication, and the current system call should be output by strace
On the latest crash Apache2ctl stop and start were successful

just to make sure, did you actually check that the apache processes were dead after running "apache2ctl stop" ?

if the process were actually restarted, that would indicate a system problem such as a lost mount point, if no, that would direct us towards some lock in apache's internals or a script cluttering the available threads


you may also want to check for established connexion during downtimes

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
did you find what was wrong ?

if yes i'm interested, and others might be

if no you can accept your own solution for 0 points cost
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Apache Web Server

From novice to tech pro — start learning today.