linux server keeps failing - need walkthrough solution

I have a remote server running apache - just a standard webserver. I am developing some sites but they are not public yet, so there is no load.

At least once each day, the server hangs. If I go into WHM and look at the server status, it says httpd has failed, and that the cpu load is 1.2. If I bring up a shell and look at ps aux, it shows /usr/local/apache using 99.8% of the cpu. Then, if I use WHM to initiate a server reboot, it comes up with httpd still failed.

I have contracted with a server management company. If I can get through to them, somehow they get it going again, but don't tell me how. They are not particularly responsive on this issue, so I need to figure this out myself. I am supposed to go live with some client sites in a few days and I can't have this going on.

Can somebody walk me though this? I am not sure where the logs are or what to look at first.

Any help greatly appreciated.
alcibiadesAsked:
Who is Participating?
 
NopiusConnect With a Mentor Commented:
Ok, try this:
/usr/local/apache/bin/httpd -F -c "LogLevel debug"
0
 
NopiusCommented:
Do you have root access to the  server? Are you the only owner of that server?
If you have root access you may look what is your server doing while it's heavy loaded.

1) Become a root
su - root

2) ps aux | grep httpd
find it's PID, then see what it's doing on system level: strace -p PID

3) Also a good reference for an error is error_log.
You may see it's location either in httpd.conf file directly (it's usially located in /etc/httpd/conf or /usr/local/etc/)
or  with lsof command indirectly (also from root):
lsof | grep httpd | grep _log





0
 
alcibiadesAuthor Commented:
I have root access, I am the only one who has access.

this is the ps aux line for httpd, you can see 99.9% cpu usage:

root      3121 99.9  0.3  7684 3140 ?        R    21:46   6:32 /usr/local/apache/bin/httpd -DSSL

strace -p 3121    does not give me anything

here is the end of the error_log file, the last entry is several hours old.

[Thu May 11 08:28:16 2006] [error] [client 216.129.98.137] File does not exist: /usr/local/apache/htdocs/404.shtml
[Thu May 11 10:33:50 2006] [error] [client 64.246.165.180] File does not exist: /home/tuulsite/public_html/robots.txt
[Thu May 11 10:33:50 2006] [error] [client 64.246.165.180] File does not exist: /home/tuulsite/public_html/404.shtml
[Thu May 11 17:46:32 2006] [notice] caught SIGTERM, shutting down
[Thu May 11 17:46:36 2006] [notice] Apache/1.3.34 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.3.10 FrontPage/5.0.2.2635 mod_ssl/2.8.25 OpenSSL/0.9.7a configured -- resuming normal operations
[Thu May 11 17:46:36 2006] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Thu May 11 17:46:36 2006] [notice] Accept mutex: sysvsem (Default: sysvsem)
[Thu May 11 17:51:34 2006] [notice] caught SIGTERM, shutting down

0
Cloud Class® Course: Certified Penetration Testing

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

 
NopiusCommented:
OK.
1) If strace gives your nothing, then your server does only a computation and no system calls.
2) Look inside access_log, also
3) Possible, you have some misconfigured apache modules. It may be misconfigured SSL (no priv key files or there are priv key files, that needs  password upon startup).
4) Try to start apache with LogLevel set to the maximum: debug either in config file or in command line with '-e debug' flag. Also try to turn off SSL and see is the problem persists.




0
 
NopiusCommented:
Also please provide the output for:
/usr/local/apache/bin/httpd -V
I like to see what mutex types are compiled in.
0
 
alcibiadesAuthor Commented:

root@ts1 [~]# /usr/local/apache/bin/httpd -V
Server version: Apache/1.3.34 (Unix)
Server built:   May 11 2006 17:46:31
Server's Module Magic Number: 19990320:18
Server compiled with....
 -D EAPI
 -D HAVE_MMAP
 -D HAVE_SHMGET
 -D USE_SHMGET_SCOREBOARD
 -D USE_MMAP_FILES
 -D HAVE_FCNTL_SERIALIZED_ACCEPT
 -D HAVE_SYSVSEM_SERIALIZED_ACCEPT
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D DYNAMIC_MODULE_LIMIT=64
 -D HARD_SERVER_LIMIT=2048
 -D HTTPD_ROOT="/usr/local/apache"
 -D SUEXEC_BIN="/usr/local/apache/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/httpd.scoreboard"
 -D DEFAULT_LOCKFILE="logs/httpd.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"
 -D ACCESS_CONFIG_FILE="conf/access.conf"
 -D RESOURCE_CONFIG_FILE="conf/srm.conf"
root@ts1 [~]#

0
 
alcibiadesAuthor Commented:
nothing to note in access log

from ssl_engine_log:

[12/May/2006 00:08:18 20959] [info]  Server: Apache/1.3.34, Interface: mod_ssl/2.8.25, Library: OpenSSL/0.9.7a
[12/May/2006 00:08:18 20959] [info]  Init: 1st startup round (still not detached)
[12/May/2006 00:08:18 20959] [info]  Init: Initializing OpenSSL library
[12/May/2006 00:08:18 20959] [info]  Init: Seeding PRNG with 136 bytes of entropy
[12/May/2006 00:08:18 20959] [info]  Init: Generating temporary RSA private keys (512/1024 bits)
[12/May/2006 00:08:18 20959] [info]  Init: Configuring temporary DH parameters (512/1024 bits)
[12/May/2006 00:16:41 09600] [info]  Server: Apache/1.3.34, Interface: mod_ssl/2.8.25, Library: OpenSSL/0.9.7a
[12/May/2006 00:16:41 09600] [info]  Init: 1st startup round (still not detached)
[12/May/2006 00:16:41 09600] [info]  Init: Initializing OpenSSL library
[12/May/2006 00:16:41 09600] [info]  Init: Seeding PRNG with 136 bytes of entropy
[12/May/2006 00:16:41 09600] [info]  Init: Generating temporary RSA private keys (512/1024 bits)
[12/May/2006 00:16:42 09600] [info]  Init: Configuring temporary DH parameters (512/1024 bits)
[12/May/2006 00:23:59 04535] [info]  Server: Apache/1.3.34, Interface: mod_ssl/2.8.25, Library: OpenSSL/0.9.7a
[12/May/2006 00:23:59 04535] [info]  Init: 1st startup round (still not detached)
[12/May/2006 00:23:59 04535] [info]  Init: Initializing OpenSSL library
[12/May/2006 00:23:59 04535] [info]  Init: Seeding PRNG with 136 bytes of entropy
[12/May/2006 00:23:59 04535] [info]  Init: Generating temporary RSA private keys (512/1024 bits)
[12/May/2006 00:23:59 04535] [info]  Init: Configuring temporary DH parameters (512/1024 bits)
[12/May/2006 00:25:04 08721] [info]  Server: Apache/1.3.34, Interface: mod_ssl/2.8.25, Library: OpenSSL/0.9.7a
[12/May/2006 00:25:04 08721] [info]  Init: 1st startup round (still not detached)
[12/May/2006 00:25:04 08721] [info]  Init: Initializing OpenSSL library
[12/May/2006 00:25:04 08721] [info]  Init: Seeding PRNG with 136 bytes of entropy
[12/May/2006 00:25:04 08721] [info]  Init: Generating temporary RSA private keys (512/1024 bits)
[12/May/2006 00:25:05 08721] [info]  Init: Configuring temporary DH parameters (512/1024 bits)
0
 
alcibiadesAuthor Commented:
I am running out of steam. Tomorrow I will try to start apache without ssl

I really appreciate your help with this
0
 
NopiusCommented:
Your server is just built. There may some kind of  build or installation problem (old config or old modules with a new server).
Also please enable LogLevel debug and see what happens (in error_log and in access_log).
Then stop your server:
/usr/local/apache/bin/httpd -k stop
and run it from command line in debug mode:
/usr/local/apache/bin/httpd -X -e debug
It's better to turn on logging in your terminal program before doing that.
0
 
alcibiadesAuthor Commented:

A couple of things I dont understand:

How and where do I set 'LogLevel' ?

How do I turn on logging in my terminal? I am using WinSCP3 and Putty to access the command line.

0
 
NopiusCommented:
Putty has Logfile property
The easiest way to setup loglevel is to run apache directly with the command: /usr/local/apache/bin/httpd -X -e debug
0
 
alcibiadesAuthor Commented:
What are we trying to achieve here? . My shell does not have -k -e or -X options for httpd


/usr/local/apache/bin/httpd: invalid option -- e
Usage: /usr/local/apache/bin/httpd [-D name] [-d directory] [-f file]
                                   [-C "directive"] [-c "directive"]
                                   [-v] [-V] [-h] [-l] [-L] [-S] [-t] [-T] [-F]
Options:
  -D name          : define a name for use in <IfDefine name> directives
  -d directory     : specify an alternate initial ServerRoot
  -f file          : specify an alternate ServerConfigFile
  -C "directive"   : process directive before reading config files
  -c "directive"   : process directive after  reading config files
  -v               : show version number
  -V               : show compile settings
  -h               : list available command line options (this page)
  -l               : list compiled-in modules
  -L               : list available configuration directives
  -S               : show parsed settings (currently only vhost settings)
  -t               : run syntax check for config files (with docroot check)
  -T               : run syntax check for config files (without docroot check)
  -F               : run main process in foreground, for process supervisors
root@ts1 [~]#


0
 
The--CaptainCommented:
man gdb

Cheers,
-Jon
0
 
alcibiadesAuthor Commented:
Thanks nopius!

I was able to figure out from the debug  that the problem was with the dom module. We took it out of apache and that seems to do the trick. Itr wasn't needed anyway. All I needed was xml support for PHP. Anyway, back in the saddle. Thanks a million.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.