VPS Hosting, Apache/2.2.3 slow connectivity/response with CentOS Linux

I'm using this VPS to serve static content (images) for my other sites on a different hosting.  This is all this server is doing, no script, no dynamic content.

In my Webalizer stats, it's showing 25,000 hits per hour.  Things were fine with the following setting for about 7~8 hours during the day, and now, it's taking me over 60 seconds to get a response.  Browser is showing, connection....waiting for.IP..address... for an extended period of time before the content shows up.

I did get fast response if I do SSH or connect to VZ panel on specific port.  Just the web content that's extremely slow.

Is there anything I need to tweak my Apache setting?

I'm using direct IP for accessing the content, and it should be relatively fast without having to wait for DNS to resolve.

What could be wrong?  What can I do to diagnose the problem?

My VPS work load is pretty low as shown in the top command:

top - 06:08:29 up 2 days,  4:27,  2 users,  load average: 0.00, 0.00, 0.00
Tasks:  27 total,   1 running,  26 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.0%sy,  0.3%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    786432k total,    30576k used,   755856k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

ps command shows the following:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  10348   748 ?        Ss   Mar22   0:01 init [3]      
apache    1629  0.0  0.3 181304  3040 ?        S    05:07   0:00 /usr/sbin/httpd
root      1728  0.0  0.0  12604   676 ?        S<s  Mar22   0:00 /sbin/udevd -d
apache    3123  0.0  0.4 181368  3220 ?        S    05:07   0:00 /usr/sbin/httpd
apache    3130  0.0  0.3 181304  3000 ?        S    05:07   0:00 /usr/sbin/httpd
apache    3572  0.0  0.4 181368  3208 ?        S    05:06   0:00 /usr/sbin/httpd
apache    5920  0.0  0.4 181368  3216 ?        S    05:04   0:00 /usr/sbin/httpd
apache    5926  0.0  0.4 181368  3212 ?        S    05:04   0:00 /usr/sbin/httpd
apache    5943  0.0  0.3 181304  3000 ?        S    05:05   0:00 /usr/sbin/httpd
apache    7183  0.0  0.3 181304  2992 ?        S    05:07   0:00 /usr/sbin/httpd
root      7313  0.0  0.2  12064  1724 pts/0    Ss   Mar23   0:00 -bash
root      7625  0.0  0.1  62616  1208 ?        Ss   Mar23   0:00 /usr/sbin/sshd
root      8116  0.0  0.0   5908   624 ?        Ss   Mar22   0:00 syslogd -m 0
root      9585  0.0  0.2  12168  1744 pts/1    Ss   Mar23   0:00 -bash
root     13811  0.0  0.1  21644   932 ?        Ss   Mar22   0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root     13889  0.0  0.0  44268   660 ?        Ss   Mar22   0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
root     18365  0.0  0.2  63296  2080 ?        Ss   Mar22   0:00 sendmail: accepting connections
smmsp    18419  0.0  0.2  57688  1792 ?        Ss   Mar22   0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
root     19921  0.0  0.1  20880  1180 ?        Ss   Mar22   0:00 crond
root     20014  0.0  0.1  46736   820 ?        Ss   Mar22   0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
root     25676  0.0  0.4  85908  3336 ?        Ss   Mar23   0:00 sshd: root@pts/0
root     26195  0.0  0.4  86212  3512 ?        Ss   Mar23   0:05 sshd: root@pts/1
root     26518  0.0  0.1  12608  1208 pts/1    SN+  Mar23   0:20 top
root     27711  0.0  0.1  10724  1004 pts/0    R+   06:09   0:00 ps -aux
apache   29960  0.0  0.3 181304  2996 ?        S    05:07   0:00 /usr/sbin/httpd
apache   30094  0.0  0.3 181304  2996 ?        S    05:06   0:00 /usr/sbin/httpd
root     30300  0.0  0.0  46736   552 ?        S    Mar22   0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
root     31929  0.0  0.4 181304  3248 ?        Ss   Mar23   0:00 /usr/sbin/httpd


Here are my apache httpd.conf

Timeout 60
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15

# prefork MPM
<IfModule prefork.c>
StartServers         8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

Following are my apache info:

httpd -V

Server version: Apache/2.2.3
Server built:   Nov 12 2009 18:43:41
Server's Module Magic Number: 20051115:3
Server loaded:  APR 1.2.7, APR-Util 1.2.7
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"


httpd -l

Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

httpd -M

Loaded Modules:
 core_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
 auth_basic_module (shared)
 auth_digest_module (shared)
 authn_file_module (shared)
 authn_alias_module (shared)
 authn_anon_module (shared)
 authn_dbm_module (shared)
 authn_default_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 authz_owner_module (shared)
 authz_groupfile_module (shared)
 authz_dbm_module (shared)
 authz_default_module (shared)
 ldap_module (shared)
 authnz_ldap_module (shared)
 include_module (shared)
 log_config_module (shared)
 logio_module (shared)
 env_module (shared)
 ext_filter_module (shared)
 mime_magic_module (shared)
 expires_module (shared)
 deflate_module (shared)
 headers_module (shared)
 usertrack_module (shared)
 setenvif_module (shared)
 mime_module (shared)
 dav_module (shared)
 status_module (shared)
 autoindex_module (shared)
 info_module (shared)
 dav_fs_module (shared)
 vhost_alias_module (shared)
 negotiation_module (shared)
 dir_module (shared)
 actions_module (shared)
 speling_module (shared)
 userdir_module (shared)
 alias_module (shared)
 rewrite_module (shared)
 proxy_module (shared)
 proxy_balancer_module (shared)
 proxy_ftp_module (shared)
 proxy_http_module (shared)
 proxy_connect_module (shared)
 cache_module (shared)
 suexec_module (shared)
 disk_cache_module (shared)
 file_cache_module (shared)
 mem_cache_module (shared)
 cgi_module (shared)
 version_module (shared)
 proxy_ajp_module (shared)
Syntax OK
jay28leeAsked:
Who is Participating?
 
jay28leeAuthor Commented:
I've had a look at the error_log and ended up with the following:

WARNING: MaxClients of 512 exceeds ServerLimit value of 10 servers, lowering MaxClients to 10.  To increase, please see the ServerLimit directive.

For some reason the ServerLimit doesn't match what I've configured in the httpd.conf, and I have my hosting taking care of that.

Things are working fine now.
0
 
Monis MontherSystem ArchitectCommented:
1- You said

I'm using this VPS to serve static content (images) for my other sites on a different hosting.  This is all this server is doing, no script, no dynamic content.

Does this mean that only your other servers should access this server content ?? if so check from apache access log if other IPs are connecting to your server (Hot Linking on your content)

2- Use apache bench ab to check server response

a/ on the server

ab -n 1 -c 1 http://127.0.0.1/path/to/slow/content

Now check the response time and start increasing the values

ab -n 100 -c 5 http://127.0.0.1/path/to/slow/content

-n means number of requests
-c mean concurrent

b/ Do the same test from the server that connects to it, if there is a big difference it might be a Network problem between them that is causing this delay


0
 
Monis MontherSystem ArchitectCommented:
Your prefork settings are fine so don't worry about them

While testing open another shell and run the top command and see what is causing the delay, it might be your CPU

also after the test post the output of the free command

free

0
Cloud Class® Course: C++ 11 Fundamentals

This course will introduce you to C++ 11 and teach you about syntax fundamentals.

 
jay28leeAuthor Commented:
i've set mod_rewrite in httpd.conf so that only specific hosting can link to the images.

my vps (the one with issue) got a timeout result, i tried several times:

(be patient)....apr_poll: The timeout specified has expired (70007)



the server connecting it got almost instantaneous response.

Server Software:        Apache                                            
Server Hostname:        my.ip.address
Server Port:            80

Document Path:          /
Document Length:        198 bytes

Concurrency Level:      5
Time taken for tests:   0.074 seconds
Complete requests:      100
Failed requests:        0
Broken pipe errors:     0
Non-2xx responses:      102
Total transferred:      34782 bytes
HTML transferred:       20196 bytes
Requests per second:    1351.35 [#/sec] (mean)
Time per request:       3.70 [ms] (mean)
Time per request:       0.74 [ms] (mean, across all concurrent requests)
Transfer rate:          470.03 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0     0    0.3      0     3
Processing:     1     2    0.9      2     8
Waiting:        0     1    0.7      1     4
Total:          1     3    0.9      2     8
WARING: The median and mean for the initial connection time are not within a normal deviation
        These results are propably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      3
  75%      3
  80%      3
  90%      4
  95%      5
  98%      7
  99%      8
 100%      8 (last request)


Here's a screen capture of the top command, there's plenty of free resource:

top - 07:33:50 up 2 days,  5:53,  2 users,  load average: 0.00, 0.01, 0.00
Tasks:  28 total,   1 running,  27 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    786432k total,    30756k used,   755676k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                            
26518 root      25  10 12608 1208  932 S  0.3  0.2   0:22.84 top                                                                                                                                                                                                  
    1 root      18   0 10348  748  624 S  0.0  0.1   0:01.59 init                                                                                                                                                                                                
 1407 apache    15   0  176m 2972 1156 S  0.0  0.4   0:00.54 httpd                                                                                                                                                                                                
 1410 apache    15   0  176m 2972 1156 S  0.0  0.4   0:00.55 httpd                                                                                                                                                                                                
 1416 apache    15   0  176m 2964 1156 S  0.0  0.4   0:00.73 httpd                                                                                                                                                                                                
 1422 apache    18   0  176m 3012 1192 S  0.0  0.4   0:00.51 httpd                                                                                                                                                                                                
 1637 root      25  10 12608 1208  932 R  0.0  0.2   0:00.00 top                                                                                                                                                                                                  
 1728 root      15  -4 12604  676  360 S  0.0  0.1   0:00.00 udevd                                                                                                                                                                                                
 6040 apache    15   0  176m 2972 1156 S  0.0  0.4   0:00.52 httpd                                                                                                                                                                                                
 6044 apache    15   0  176m 2972 1156 S  0.0  0.4   0:00.52 httpd                                                                                                                                                                                                
 7313 root      15   0 12064 1744 1304 S  0.0  0.2   0:00.24 bash                                                                                                                                                                                                
 7625 root      15   0 62616 1208  652 S  0.0  0.2   0:00.53 sshd                                                                                                                                                                                                
 8116 root      15   0  5908  624  500 S  0.0  0.1   0:00.73 syslogd                                                                                                                                                                                              
 9585 root      15   0 12168 1744 1292 S  0.0  0.2   0:00.08 bash                                                                                                                                                                                                
13811 root      19   0 21644  932  716 S  0.0  0.1   0:00.00 xinetd                                                                                                                                                                                              
13889 root      15   0 44268  660  268 S  0.0  0.1   0:00.01 vsftpd                                                                                                                                                                                              
17759 root      15   0  176m 3128 1396 S  0.0  0.4   0:00.02 httpd                                                                                                                                                                                                
18365 root      18   0 63296 2080  692 S  0.0  0.3   0:00.58 sendmail                                                                                                                                                                                            
18419 smmsp     18   0 57688 1792  644 S  0.0  0.2   0:00.00 sendmail                                                                                                                                                                                            
19921 root      15   0 20880 1180  592 S  0.0  0.2   0:00.14 crond                                                                                                                                                                                                
20014 root      18   0 46736  820  428 S  0.0  0.1   0:00.00 saslauthd                                                                                                                                                                                            
23875 apache    15   0  176m 2972 1156 S  0.0  0.4   0:00.50 httpd                                                                                                                                                                                                
25676 root      15   0 85908 3336 2616 S  0.0  0.4   0:00.36 sshd                                                                                                                                                                                                
25827 apache    18   0  176m 2972 1156 S  0.0  0.4   0:00.42 httpd                                                                                                                                                                                                
26195 root      16   0 86212 3512 2628 S  0.0  0.4   0:05.36 sshd                                                                                                                                                                                                
28520 apache    18   0  176m 2968 1156 S  0.0  0.4   0:00.55 httpd                                                                                                                                                                                                
28527 apache    18   0  176m 2972 1156 S  0.0  0.4   0:00.45 httpd                                                                                                                                                                                                
30300 root      18   0 46736  552  160 S  0.0  0.1   0:00.00 saslauthd                                                                                                                                                                                            


Could it be I have too many requests at any given time that apache wasn't configured to handle?

The command free gives me:

             total       used       free     shared    buffers     cached
Mem:        786432      29916     756516          0          0          0
-/+ buffers/cache:      29916     756516
Swap:            0          0          0
0
 
jay28leeAuthor Commented:
I also use netstat -ant | grep :80 | wc -l to see my tcp connection and it shows 545.

is it too high?
0
 
Monis MontherSystem ArchitectCommented:
Yes that is high and should show plenty of resource usage, but these might not be Established connections

do

netstat -ant | grep :80 |grep EST |wc -l


I also noticed something why is your SWAP 0 you dont have any swap partition or swap file, although its not good for apache to use swap it degrades performance and means your memory is at full use but I am not sure if this is advisable.

Also your apache bench results are not normal, These is something strange or that I dont understand

0
 
jay28leeAuthor Commented:
the netstat -ant | grep :80 |grep EST |wc -l command give me 103

it shows i only used 30mb of memory and there's still 750mb free (which is my vps hosting package provided)

how much swap space should i allocate in such a configuration with 780mb of memory?

my hosting was suggestion using MPM Event, how should i do that?  Will it help?  since it's not a managed hosting, i need to do everything by myself.  any further suggestion / instruction?
0
 
Monis MontherSystem ArchitectCommented:
1-
No MPM Event is an experimental MPM module for the apache 2.2.x server check this Link from the official Apache Docs

http://httpd.apache.org/docs/2.2/mod/event.html

So [IMHO] dont think this is a good option for you


2-
1GB Swap is enough , but I dont think either would fix the problem, but its a step that we ensure to exculde something with (To make sure this is not the problem)

3-
103 Established connections is also a high number , can you check from what IPs they are coming according to your configuration they should all come from your other server, confirm this to exclude an external attack

4-
Still Resource should be higher than this.

run the following command

rpm -V top
rpm -V free
rpm -V ps

After each one you run you should get back to the prompt with no results , if you got any results post them
0
 
jay28leeAuthor Commented:
I got:

package top is not installed
package free is not installed
package ps is not installed

after each execution.

But I can use them without problem.

How do I check (3) where the IPs coming from?  Because this is an image server, the other one is hosting the content.  The IP should be visitors requesting for content and then pull the images from this server with issue.
0
 
Monis MontherSystem ArchitectCommented:
Ok its either one of two

1- The packages are installed from source not via rpm and would not be seen from rpm output

2- They have a problem, something is wrong

Now to check where the IPs are coming from

tail -f /var/log/httpd/access_log

This will show you the requests as they are coming to your server each line will show a request including the IP that is requesting

Also the output of netstat will show you the connecting IPs
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.