• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 372
  • Last Modified:

Apache Crashes frequently

Hi,

We are using Apache web server(Apache 2.0.52) on Solaris 9 and 10 boxes and Adobe Coldfusion in business tier.

We are facing the problem of Apache being crashed regularly (bi weekly or every 3 weeks).

We have the Core dump error on all the occasions of crash.

Currently we are restarting apache even on production env as a work around  which is not an acceptable solution.


Can any one help on how to fix this issue.

Thanks,
Rishi

0
RishiBang
Asked:
RishiBang
3 Solutions
 
arnoldCommented:
What is the cause of the crashes reported in the core dumps?
There are tools like adb, gdb etc. that you can use to analyze the coredumps to determine the cause i.e. memory leak, not enough resources available, access to an NFS mount that has a hard failure?  Too many or not enough child processies. etc.
If instead of restart you use the graceful or reload option does that alter the likelyhood of the crash?
What are the ulimit settings on the apache user (open files, memory/threads, etc.)?
0
 
madunixChief Information Security Officer Commented:
http://httpd.apache.org/dev/debugging.html    is a must.
http://www.linuxjournal.com/article/4222
can you check the followng:
- look in /var/log any suspicious
- do you have free drive space, but do all partitions have free space
- are all the file systems OK? fsck
- memory diagnostic.... could be a bad piece of RAM
- check apache config
- check mysql config it could memory setting bigger than your actual RAM (if you have mysql)
- Apache service starts with no errors??
- check Apache error log if contains hints
.....

madunix

 
0
 
RishiBangAuthor Commented:
Hi arnold,
Sorry fo the delayed reply.
I have tried to use the gdb for analyzing the core dump, but it is availble only on unix platforms.
We are not allowed to install anyting on the Unix boxes in our platfom.
Pls let me know if any windows based tools are available for analyzing Apache Core dump.
I have attached the core dump file. If possbile try to open it with the tool you are  using.

Thanks,
Rishi




core.httpd.2075.txt
0
Free learning courses: Active Directory Deep Dive

Get a firm grasp on your IT environment when you learn Active Directory best practices with Veeam! Watch all, or choose any amount, of this three-part webinar series to improve your skills. From the basics to virtualization and backup, we got you covered.

 
RishiBangAuthor Commented:
arnold,
Sorry I forgot to mention that the file attached in the previous reply is a .7z file since the UI has not accepted it I have renamed to .txt.
Thanks
Rishi
0
 
arnoldCommented:
adb/mdb is usually available on the system and also can be used to analyze the core dump.

I do not have a way to analyze the coredump.

http://developers.sun.com/solaris/articles/manage_core_dump.html
You could use the tools referenced in the above article and post the text output.

mdb/pstack

If you knew approximately when apache is going to crash, you can use truss -f -p <pid of apache> to capture the event.
You mentioned a production environment which suggest that there is a test environment.  Do you have or can you get GDB on one of those?

0
 
RishiBangAuthor Commented:
Hi anold,

yes you are right we have test env but all are unix boxes and power broker audited.

Anything that needs to be installed on unix(except Websphere related) we have to go thru the unix team.

WAS admins are not allowed to do so.

That's why looking for any windows based Core dump analyzer since apache is available on windows as well.

Thanks
Rishi
0
 
arnoldCommented:
The hastle if it is possible is less than asking the unix admin team to install gdb on one of the test environments or see if they can analyze the coredump on one of their Solaris workstations.

You could try using GDB for windows:
http://hamsterrepublic.com/ohrrpgce/index.php/GDB_on_Windows.html
The difficulty you have is that your environment is Solaris, the GDB is a "MinGW or cygwin" linux wihtin windows.

If you have a virtualization option on your windows workstation vmware, virtualbox, etc. you can use opensolaris to install a VM where you can analyze the coredump.

Ask the Unix Admins to analyze the coredump.

Can you setup cron jobs on the production system?
You can get a snapshot using vmstat, iostat, pstack, memstat, netstat -an | grep '(TIME_WAIT|ESTABLISHED|etc.)' while testing the apache service. This way when the apache test fails, you have somewhat of an explanation.
Also check whether the condition deals with too many open clients and apache is waiting until one is freed up?
i.e. the issue could be that a connection remains stale i.e. is not closed but is seen by apache as being in use (netstat). The problem depending on usage can occur in two weeks or three depending on when the resources are exhausted reach the limit.

GDB is a useful tool in isolating/resolving problems and I would think would normally be available on any system.

See if you have GDB already installed in /usr/sfw/bin/gdb.
pkginfo | grep -i 'gdb'
It depends on which package options were used during the install.
0
 
RishiBangAuthor Commented:
Thanks Arnold,
I greatly appreciate your detailed explanation.
Sure I will try to work on the possiblities you have mentioned and come back with some useful info to disucss further.
Regards,
Rishi

0
 
Brian UtterbackPrinciple Software EngineerCommented:
Instead of installing gdb, try running pstack on the core.  You will want to run pstack on the core in place on the production system, but it should already be there without requiring any new installation.
0
 
NoBugCommented:
can you check the followng:
look in /var/log any suspicious
do you have free drive space, but do all partitions have free space
are all the file systems OK? fsck
check Apache error log if contains hints
mem diagnostic.... could be a bad piece of RAM
check apache config
check mysql config it could memory setting bigger than your actual RAM (if you have mysql)
Apache service starts with no errors??

.....
0
 
RishiBangAuthor Commented:
The Info provided by the madunix and arnold is useful and easy to follow
0

Featured Post

NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now