linux server keeps failing - need walkthrough solution

Do you have root access to the  server? Are you the only owner of that server?
If you have root access you may look what is your server doing while it's heavy loaded.

1) Become a root
su - root

2) ps aux | grep httpd
find it's PID, then see what it's doing on system level: strace -p PID

3) Also a good reference for an error is error_log.
You may see it's location either in httpd.conf file directly (it's usially located in /etc/httpd/conf or /usr/local/etc/)
or  with lsof command indirectly (also from root):
lsof | grep httpd | grep _log

I have root access, I am the only one who has access.

this is the ps aux line for httpd, you can see 99.9% cpu usage:

root      3121 99.9  0.3  7684 3140 ?        R    21:46   6:32 /usr/local/apache/bin/httpd -DSSL

strace -p 3121    does not give me anything

here is the end of the error_log file, the last entry is several hours old.

[Thu May 11 08:28:16 2006] [error] [client 216.129.98.137] File does not exist: /usr/local/apache/htdocs/404.shtml
[Thu May 11 10:33:50 2006] [error] [client 64.246.165.180] File does not exist: /home/tuulsite/public_html/robots.txt
[Thu May 11 10:33:50 2006] [error] [client 64.246.165.180] File does not exist: /home/tuulsite/public_html/404.shtml
[Thu May 11 17:46:32 2006] [notice] caught SIGTERM, shutting down
[Thu May 11 17:46:36 2006] [notice] Apache/1.3.34 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.3.10 FrontPage/5.0.2.2635 mod_ssl/2.8.25 OpenSSL/0.9.7a configured -- resuming normal operations
[Thu May 11 17:46:36 2006] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Thu May 11 17:46:36 2006] [notice] Accept mutex: sysvsem (Default: sysvsem)
[Thu May 11 17:51:34 2006] [notice] caught SIGTERM, shutting down

OK.
1) If strace gives your nothing, then your server does only a computation and no system calls.
2) Look inside access_log, also
3) Possible, you have some misconfigured apache modules. It may be misconfigured SSL (no priv key files or there are priv key files, that needs  password upon startup).
4) Try to start apache with LogLevel set to the maximum: debug either in config file or in command line with '-e debug' flag. Also try to turn off SSL and see is the problem persists.

Also please provide the output for:
/usr/local/apache/bin/httpd -V
I like to see what mutex types are compiled in.

root@ts1 [~]# /usr/local/apache/bin/httpd -V
Server version: Apache/1.3.34 (Unix)
Server built:   May 11 2006 17:46:31
Server's Module Magic Number: 19990320:18
Server compiled with....
 -D EAPI
 -D HAVE_MMAP
 -D HAVE_SHMGET
 -D USE_SHMGET_SCOREBOARD
 -D USE_MMAP_FILES
 -D HAVE_FCNTL_SERIALIZED_ACCEPT
 -D HAVE_SYSVSEM_SERIALIZED_ACCEPT
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D DYNAMIC_MODULE_LIMIT=64
 -D HARD_SERVER_LIMIT=2048
 -D HTTPD_ROOT="/usr/local/apache"
 -D SUEXEC_BIN="/usr/local/apache/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/httpd.scoreboard"
 -D DEFAULT_LOCKFILE="logs/httpd.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"
 -D ACCESS_CONFIG_FILE="conf/access.conf"
 -D RESOURCE_CONFIG_FILE="conf/srm.conf"
root@ts1 [~]#

nothing to note in access log

from ssl_engine_log:

[12/May/2006 00:08:18 20959] [info]  Server: Apache/1.3.34, Interface: mod_ssl/2.8.25, Library: OpenSSL/0.9.7a
[12/May/2006 00:08:18 20959] [info]  Init: 1st startup round (still not detached)
[12/May/2006 00:08:18 20959] [info]  Init: Initializing OpenSSL library
[12/May/2006 00:08:18 20959] [info]  Init: Seeding PRNG with 136 bytes of entropy
[12/May/2006 00:08:18 20959] [info]  Init: Generating temporary RSA private keys (512/1024 bits)
[12/May/2006 00:08:18 20959] [info]  Init: Configuring temporary DH parameters (512/1024 bits)
[12/May/2006 00:16:41 09600] [info]  Server: Apache/1.3.34, Interface: mod_ssl/2.8.25, Library: OpenSSL/0.9.7a
[12/May/2006 00:16:41 09600] [info]  Init: 1st startup round (still not detached)
[12/May/2006 00:16:41 09600] [info]  Init: Initializing OpenSSL library
[12/May/2006 00:16:41 09600] [info]  Init: Seeding PRNG with 136 bytes of entropy
[12/May/2006 00:16:41 09600] [info]  Init: Generating temporary RSA private keys (512/1024 bits)
[12/May/2006 00:16:42 09600] [info]  Init: Configuring temporary DH parameters (512/1024 bits)
[12/May/2006 00:23:59 04535] [info]  Server: Apache/1.3.34, Interface: mod_ssl/2.8.25, Library: OpenSSL/0.9.7a
[12/May/2006 00:23:59 04535] [info]  Init: 1st startup round (still not detached)
[12/May/2006 00:23:59 04535] [info]  Init: Initializing OpenSSL library
[12/May/2006 00:23:59 04535] [info]  Init: Seeding PRNG with 136 bytes of entropy
[12/May/2006 00:23:59 04535] [info]  Init: Generating temporary RSA private keys (512/1024 bits)
[12/May/2006 00:23:59 04535] [info]  Init: Configuring temporary DH parameters (512/1024 bits)
[12/May/2006 00:25:04 08721] [info]  Server: Apache/1.3.34, Interface: mod_ssl/2.8.25, Library: OpenSSL/0.9.7a
[12/May/2006 00:25:04 08721] [info]  Init: 1st startup round (still not detached)
[12/May/2006 00:25:04 08721] [info]  Init: Initializing OpenSSL library
[12/May/2006 00:25:04 08721] [info]  Init: Seeding PRNG with 136 bytes of entropy
[12/May/2006 00:25:04 08721] [info]  Init: Generating temporary RSA private keys (512/1024 bits)
[12/May/2006 00:25:05 08721] [info]  Init: Configuring temporary DH parameters (512/1024 bits)

I am running out of steam. Tomorrow I will try to start apache without ssl

I really appreciate your help with this

Your server is just built. There may some kind of  build or installation problem (old config or old modules with a new server).
Also please enable LogLevel debug and see what happens (in error_log and in access_log).
Then stop your server:
/usr/local/apache/bin/httpd -k stop
and run it from command line in debug mode:
/usr/local/apache/bin/httpd -X -e debug
It's better to turn on logging in your terminal program before doing that.

A couple of things I dont understand:

How and where do I set 'LogLevel' ?

How do I turn on logging in my terminal? I am using WinSCP3 and Putty to access the command line.

Putty has Logfile property
The easiest way to setup loglevel is to run apache directly with the command: /usr/local/apache/bin/httpd -X -e debug

What are we trying to achieve here? . My shell does not have -k -e or -X options for httpd


/usr/local/apache/bin/httpd: invalid option -- e
Usage: /usr/local/apache/bin/httpd [-D name] [-d directory] [-f file]
                                   [-C "directive"] [-c "directive"]
                                   [-v] [-V] [-h] [-l] [-L] [-S] [-t] [-T] [-F]
Options:
  -D name          : define a name for use in <IfDefine name> directives
  -d directory     : specify an alternate initial ServerRoot
  -f file          : specify an alternate ServerConfigFile
  -C "directive"   : process directive before reading config files
  -c "directive"   : process directive after  reading config files
  -v               : show version number
  -V               : show compile settings
  -h               : list available command line options (this page)
  -l               : list compiled-in modules
  -L               : list available configuration directives
  -S               : show parsed settings (currently only vhost settings)
  -t               : run syntax check for config files (with docroot check)
  -T               : run syntax check for config files (without docroot check)
  -F               : run main process in foreground, for process supervisors
root@ts1 [~]#

man gdb

Cheers,
-Jon

Thanks nopius!

I was able to figure out from the debug  that the problem was with the dom module. We took it out of apache and that seems to do the trick. Itr wasn't needed anyway. All I needed was xml support for PHP. Anyway, back in the saddle. Thanks a million.