?
Solved

linux server keeps failing - need walkthrough solution

Posted on 2006-05-11
14
Medium Priority
?
196 Views
Last Modified: 2010-03-18
I have a remote server running apache - just a standard webserver. I am developing some sites but they are not public yet, so there is no load.

At least once each day, the server hangs. If I go into WHM and look at the server status, it says httpd has failed, and that the cpu load is 1.2. If I bring up a shell and look at ps aux, it shows /usr/local/apache using 99.8% of the cpu. Then, if I use WHM to initiate a server reboot, it comes up with httpd still failed.

I have contracted with a server management company. If I can get through to them, somehow they get it going again, but don't tell me how. They are not particularly responsive on this issue, so I need to figure this out myself. I am supposed to go live with some client sites in a few days and I can't have this going on.

Can somebody walk me though this? I am not sure where the logs are or what to look at first.

Any help greatly appreciated.
0
Comment
Question by:alcibiades
  • 7
  • 6
14 Comments
 
LVL 27

Expert Comment

by:Nopius
ID: 16664187
Do you have root access to the  server? Are you the only owner of that server?
If you have root access you may look what is your server doing while it's heavy loaded.

1) Become a root
su - root

2) ps aux | grep httpd
find it's PID, then see what it's doing on system level: strace -p PID

3) Also a good reference for an error is error_log.
You may see it's location either in httpd.conf file directly (it's usially located in /etc/httpd/conf or /usr/local/etc/)
or  with lsof command indirectly (also from root):
lsof | grep httpd | grep _log





0
 

Author Comment

by:alcibiades
ID: 16664383
I have root access, I am the only one who has access.

this is the ps aux line for httpd, you can see 99.9% cpu usage:

root      3121 99.9  0.3  7684 3140 ?        R    21:46   6:32 /usr/local/apache/bin/httpd -DSSL

strace -p 3121    does not give me anything

here is the end of the error_log file, the last entry is several hours old.

[Thu May 11 08:28:16 2006] [error] [client 216.129.98.137] File does not exist: /usr/local/apache/htdocs/404.shtml
[Thu May 11 10:33:50 2006] [error] [client 64.246.165.180] File does not exist: /home/tuulsite/public_html/robots.txt
[Thu May 11 10:33:50 2006] [error] [client 64.246.165.180] File does not exist: /home/tuulsite/public_html/404.shtml
[Thu May 11 17:46:32 2006] [notice] caught SIGTERM, shutting down
[Thu May 11 17:46:36 2006] [notice] Apache/1.3.34 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.3.10 FrontPage/5.0.2.2635 mod_ssl/2.8.25 OpenSSL/0.9.7a configured -- resuming normal operations
[Thu May 11 17:46:36 2006] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Thu May 11 17:46:36 2006] [notice] Accept mutex: sysvsem (Default: sysvsem)
[Thu May 11 17:51:34 2006] [notice] caught SIGTERM, shutting down

0
 
LVL 27

Expert Comment

by:Nopius
ID: 16664556
OK.
1) If strace gives your nothing, then your server does only a computation and no system calls.
2) Look inside access_log, also
3) Possible, you have some misconfigured apache modules. It may be misconfigured SSL (no priv key files or there are priv key files, that needs  password upon startup).
4) Try to start apache with LogLevel set to the maximum: debug either in config file or in command line with '-e debug' flag. Also try to turn off SSL and see is the problem persists.




0
Get your Conversational Ransomware Defense e‑book

This e-book gives you an insight into the ransomware threat and reviews the fundamentals of top-notch ransomware preparedness and recovery. To help you protect yourself and your organization. The initial infection may be inevitable, so the best protection is to be fully prepared.

 
LVL 27

Expert Comment

by:Nopius
ID: 16664577
Also please provide the output for:
/usr/local/apache/bin/httpd -V
I like to see what mutex types are compiled in.
0
 

Author Comment

by:alcibiades
ID: 16664947

root@ts1 [~]# /usr/local/apache/bin/httpd -V
Server version: Apache/1.3.34 (Unix)
Server built:   May 11 2006 17:46:31
Server's Module Magic Number: 19990320:18
Server compiled with....
 -D EAPI
 -D HAVE_MMAP
 -D HAVE_SHMGET
 -D USE_SHMGET_SCOREBOARD
 -D USE_MMAP_FILES
 -D HAVE_FCNTL_SERIALIZED_ACCEPT
 -D HAVE_SYSVSEM_SERIALIZED_ACCEPT
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D DYNAMIC_MODULE_LIMIT=64
 -D HARD_SERVER_LIMIT=2048
 -D HTTPD_ROOT="/usr/local/apache"
 -D SUEXEC_BIN="/usr/local/apache/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/httpd.scoreboard"
 -D DEFAULT_LOCKFILE="logs/httpd.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"
 -D ACCESS_CONFIG_FILE="conf/access.conf"
 -D RESOURCE_CONFIG_FILE="conf/srm.conf"
root@ts1 [~]#

0
 

Author Comment

by:alcibiades
ID: 16664969
nothing to note in access log

from ssl_engine_log:

[12/May/2006 00:08:18 20959] [info]  Server: Apache/1.3.34, Interface: mod_ssl/2.8.25, Library: OpenSSL/0.9.7a
[12/May/2006 00:08:18 20959] [info]  Init: 1st startup round (still not detached)
[12/May/2006 00:08:18 20959] [info]  Init: Initializing OpenSSL library
[12/May/2006 00:08:18 20959] [info]  Init: Seeding PRNG with 136 bytes of entropy
[12/May/2006 00:08:18 20959] [info]  Init: Generating temporary RSA private keys (512/1024 bits)
[12/May/2006 00:08:18 20959] [info]  Init: Configuring temporary DH parameters (512/1024 bits)
[12/May/2006 00:16:41 09600] [info]  Server: Apache/1.3.34, Interface: mod_ssl/2.8.25, Library: OpenSSL/0.9.7a
[12/May/2006 00:16:41 09600] [info]  Init: 1st startup round (still not detached)
[12/May/2006 00:16:41 09600] [info]  Init: Initializing OpenSSL library
[12/May/2006 00:16:41 09600] [info]  Init: Seeding PRNG with 136 bytes of entropy
[12/May/2006 00:16:41 09600] [info]  Init: Generating temporary RSA private keys (512/1024 bits)
[12/May/2006 00:16:42 09600] [info]  Init: Configuring temporary DH parameters (512/1024 bits)
[12/May/2006 00:23:59 04535] [info]  Server: Apache/1.3.34, Interface: mod_ssl/2.8.25, Library: OpenSSL/0.9.7a
[12/May/2006 00:23:59 04535] [info]  Init: 1st startup round (still not detached)
[12/May/2006 00:23:59 04535] [info]  Init: Initializing OpenSSL library
[12/May/2006 00:23:59 04535] [info]  Init: Seeding PRNG with 136 bytes of entropy
[12/May/2006 00:23:59 04535] [info]  Init: Generating temporary RSA private keys (512/1024 bits)
[12/May/2006 00:23:59 04535] [info]  Init: Configuring temporary DH parameters (512/1024 bits)
[12/May/2006 00:25:04 08721] [info]  Server: Apache/1.3.34, Interface: mod_ssl/2.8.25, Library: OpenSSL/0.9.7a
[12/May/2006 00:25:04 08721] [info]  Init: 1st startup round (still not detached)
[12/May/2006 00:25:04 08721] [info]  Init: Initializing OpenSSL library
[12/May/2006 00:25:04 08721] [info]  Init: Seeding PRNG with 136 bytes of entropy
[12/May/2006 00:25:04 08721] [info]  Init: Generating temporary RSA private keys (512/1024 bits)
[12/May/2006 00:25:05 08721] [info]  Init: Configuring temporary DH parameters (512/1024 bits)
0
 

Author Comment

by:alcibiades
ID: 16664972
I am running out of steam. Tomorrow I will try to start apache without ssl

I really appreciate your help with this
0
 
LVL 27

Expert Comment

by:Nopius
ID: 16664982
Your server is just built. There may some kind of  build or installation problem (old config or old modules with a new server).
Also please enable LogLevel debug and see what happens (in error_log and in access_log).
Then stop your server:
/usr/local/apache/bin/httpd -k stop
and run it from command line in debug mode:
/usr/local/apache/bin/httpd -X -e debug
It's better to turn on logging in your terminal program before doing that.
0
 

Author Comment

by:alcibiades
ID: 16665623

A couple of things I dont understand:

How and where do I set 'LogLevel' ?

How do I turn on logging in my terminal? I am using WinSCP3 and Putty to access the command line.

0
 
LVL 27

Expert Comment

by:Nopius
ID: 16665860
Putty has Logfile property
The easiest way to setup loglevel is to run apache directly with the command: /usr/local/apache/bin/httpd -X -e debug
0
 

Author Comment

by:alcibiades
ID: 16665983
What are we trying to achieve here? . My shell does not have -k -e or -X options for httpd


/usr/local/apache/bin/httpd: invalid option -- e
Usage: /usr/local/apache/bin/httpd [-D name] [-d directory] [-f file]
                                   [-C "directive"] [-c "directive"]
                                   [-v] [-V] [-h] [-l] [-L] [-S] [-t] [-T] [-F]
Options:
  -D name          : define a name for use in <IfDefine name> directives
  -d directory     : specify an alternate initial ServerRoot
  -f file          : specify an alternate ServerConfigFile
  -C "directive"   : process directive before reading config files
  -c "directive"   : process directive after  reading config files
  -v               : show version number
  -V               : show compile settings
  -h               : list available command line options (this page)
  -l               : list compiled-in modules
  -L               : list available configuration directives
  -S               : show parsed settings (currently only vhost settings)
  -t               : run syntax check for config files (with docroot check)
  -T               : run syntax check for config files (without docroot check)
  -F               : run main process in foreground, for process supervisors
root@ts1 [~]#


0
 
LVL 27

Accepted Solution

by:
Nopius earned 2000 total points
ID: 16665999
Ok, try this:
/usr/local/apache/bin/httpd -F -c "LogLevel debug"
0
 
LVL 16

Expert Comment

by:The--Captain
ID: 16671014
man gdb

Cheers,
-Jon
0
 

Author Comment

by:alcibiades
ID: 16672073
Thanks nopius!

I was able to figure out from the debug  that the problem was with the dom module. We took it out of apache and that seems to do the trick. Itr wasn't needed anyway. All I needed was xml support for PHP. Anyway, back in the saddle. Thanks a million.
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I have seen several blogs and forum entries elsewhere state that because NTFS volumes do not support linux ownership or permissions, they cannot be used for anonymous ftp upload through the vsftpd program.   IT can be done and here's how to get i…
Note: for this to work properly you need to use a Cross-Over network cable. 1. Connect both servers S1 and S2 on the second network slots respectively. Note that you can use the 1st slots but usually these would be occupied by the Service Provide…
If you're a developer or IT admin, you’re probably tasked with managing multiple websites, servers, applications, and levels of security on a daily basis. While this can be extremely time consuming, it can also be frustrating when systems aren't wor…
When cloud platforms entered the scene, users and companies jumped on board to take advantage of the many benefits, like the ability to work and connect with company information from various locations. What many didn't foresee was the increased risk…
Suggested Courses
Course of the Month15 days, 21 hours left to enroll

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question