asked on

VPS Hosting, Apache/2.2.3 slow connectivity/response with CentOS Linux

I'm using this VPS to serve static content (images) for my other sites on a different hosting. This is all this server is doing, no script, no dynamic content.

In my Webalizer stats, it's showing 25,000 hits per hour. Things were fine with the following setting for about 7~8 hours during the day, and now, it's taking me over 60 seconds to get a response. Browser is showing, connection....waiting for.IP..address... for an extended period of time before the content shows up.

I did get fast response if I do SSH or connect to VZ panel on specific port. Just the web content that's extremely slow.

Is there anything I need to tweak my Apache setting?

I'm using direct IP for accessing the content, and it should be relatively fast without having to wait for DNS to resolve.

What could be wrong? What can I do to diagnose the problem?

My VPS work load is pretty low as shown in the top command:

top - 06:08:29 up 2 days, 4:27, 2 users, load average: 0.00, 0.00, 0.00
Tasks: 27 total, 1 running, 26 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.0%sy, 0.3%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 786432k total, 30576k used, 755856k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

ps command shows the following:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 10348 748 ? Ss Mar22 0:01 init [3]
apache 1629 0.0 0.3 181304 3040 ? S 05:07 0:00 /usr/sbin/httpd
root 1728 0.0 0.0 12604 676 ? S<s Mar22 0:00 /sbin/udevd -d
apache 3123 0.0 0.4 181368 3220 ? S 05:07 0:00 /usr/sbin/httpd
apache 3130 0.0 0.3 181304 3000 ? S 05:07 0:00 /usr/sbin/httpd
apache 3572 0.0 0.4 181368 3208 ? S 05:06 0:00 /usr/sbin/httpd
apache 5920 0.0 0.4 181368 3216 ? S 05:04 0:00 /usr/sbin/httpd
apache 5926 0.0 0.4 181368 3212 ? S 05:04 0:00 /usr/sbin/httpd
apache 5943 0.0 0.3 181304 3000 ? S 05:05 0:00 /usr/sbin/httpd
apache 7183 0.0 0.3 181304 2992 ? S 05:07 0:00 /usr/sbin/httpd
root 7313 0.0 0.2 12064 1724 pts/0 Ss Mar23 0:00 -bash
root 7625 0.0 0.1 62616 1208 ? Ss Mar23 0:00 /usr/sbin/sshd
root 8116 0.0 0.0 5908 624 ? Ss Mar22 0:00 syslogd -m 0
root 9585 0.0 0.2 12168 1744 pts/1 Ss Mar23 0:00 -bash
root 13811 0.0 0.1 21644 932 ? Ss Mar22 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root 13889 0.0 0.0 44268 660 ? Ss Mar22 0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
root 18365 0.0 0.2 63296 2080 ? Ss Mar22 0:00 sendmail: accepting connections
smmsp 18419 0.0 0.2 57688 1792 ? Ss Mar22 0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
root 19921 0.0 0.1 20880 1180 ? Ss Mar22 0:00 crond
root 20014 0.0 0.1 46736 820 ? Ss Mar22 0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
root 25676 0.0 0.4 85908 3336 ? Ss Mar23 0:00 sshd: root@pts/0
root 26195 0.0 0.4 86212 3512 ? Ss Mar23 0:05 sshd: root@pts/1
root 26518 0.0 0.1 12608 1208 pts/1 SN+ Mar23 0:20 top
root 27711 0.0 0.1 10724 1004 pts/0 R+ 06:09 0:00 ps -aux
apache 29960 0.0 0.3 181304 2996 ? S 05:07 0:00 /usr/sbin/httpd
apache 30094 0.0 0.3 181304 2996 ? S 05:06 0:00 /usr/sbin/httpd
root 30300 0.0 0.0 46736 552 ? S Mar22 0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
root 31929 0.0 0.4 181304 3248 ? Ss Mar23 0:00 /usr/sbin/httpd

Here are my apache httpd.conf

Timeout 60
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15

# prefork MPM
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
</IfModule>

Following are my apache info:

httpd -V

Server version: Apache/2.2.3
Server built: Nov 12 2009 18:43:41
Server's Module Magic Number: 20051115:3
Server loaded: APR 1.2.7, APR-Util 1.2.7
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

httpd -l

Compiled in modules:
core.c
prefork.c
http_core.c
mod_so.c

httpd -M

Loaded Modules:
core_module (static)
mpm_prefork_module (static)
http_module (static)
so_module (static)
auth_basic_module (shared)
auth_digest_module (shared)
authn_file_module (shared)
authn_alias_module (shared)
authn_anon_module (shared)
authn_dbm_module (shared)
authn_default_module (shared)
authz_host_module (shared)
authz_user_module (shared)
authz_owner_module (shared)
authz_groupfile_module (shared)
authz_dbm_module (shared)
authz_default_module (shared)
ldap_module (shared)
authnz_ldap_module (shared)
include_module (shared)
log_config_module (shared)
logio_module (shared)
env_module (shared)
ext_filter_module (shared)
mime_magic_module (shared)
expires_module (shared)
deflate_module (shared)
headers_module (shared)
usertrack_module (shared)
setenvif_module (shared)
mime_module (shared)
dav_module (shared)
status_module (shared)
autoindex_module (shared)
info_module (shared)
dav_fs_module (shared)
vhost_alias_module (shared)
negotiation_module (shared)
dir_module (shared)
actions_module (shared)
speling_module (shared)
userdir_module (shared)
alias_module (shared)
rewrite_module (shared)
proxy_module (shared)
proxy_balancer_module (shared)
proxy_ftp_module (shared)
proxy_http_module (shared)
proxy_connect_module (shared)
cache_module (shared)
suexec_module (shared)
disk_cache_module (shared)
file_cache_module (shared)
mem_cache_module (shared)
cgi_module (shared)
version_module (shared)
proxy_ajp_module (shared)
Syntax OK

Monis Monther

1- You said

I'm using this VPS to serve static content (images) for my other sites on a different hosting. This is all this server is doing, no script, no dynamic content.

Does this mean that only your other servers should access this server content ?? if so check from apache access log if other IPs are connecting to your server (Hot Linking on your content)

2- Use apache bench ab to check server response

a/ on the server

ab -n 1 -c 1 http://127.0.0.1/path/to/slow/content

Now check the response time and start increasing the values

ab -n 100 -c 5 http://127.0.0.1/path/to/slow/content

-n means number of requests
-c mean concurrent

b/ Do the same test from the server that connects to it, if there is a big difference it might be a Network problem between them that is causing this delay

Monis Monther

Your prefork settings are fine so don't worry about them

While testing open another shell and run the top command and see what is causing the delay, it might be your CPU

also after the test post the output of the free command

free

jay28lee

ASKER

i've set mod_rewrite in httpd.conf so that only specific hosting can link to the images.

my vps (the one with issue) got a timeout result, i tried several times:

(be patient)....apr_poll: The timeout specified has expired (70007)

the server connecting it got almost instantaneous response.

Server Software: Apache
Server Hostname: my.ip.address
Server Port: 80

Document Path: /
Document Length: 198 bytes

Concurrency Level: 5
Time taken for tests: 0.074 seconds
Complete requests: 100
Failed requests: 0
Broken pipe errors: 0
Non-2xx responses: 102
Total transferred: 34782 bytes
HTML transferred: 20196 bytes
Requests per second: 1351.35 [#/sec] (mean)
Time per request: 3.70 [ms] (mean)
Time per request: 0.74 [ms] (mean, across all concurrent requests)
Transfer rate: 470.03 [Kbytes/sec] received

Connnection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.3 0 3
Processing: 1 2 0.9 2 8
Waiting: 0 1 0.7 1 4
Total: 1 3 0.9 2 8
WARING: The median and mean for the initial connection time are not within a normal deviation
These results are propably not that reliable.

Percentage of the requests served within a certain time (ms)
50% 2
66% 3
75% 3
80% 3
90% 4
95% 5
98% 7
99% 8
100% 8 (last request)

Here's a screen capture of the top command, there's plenty of free resource:

top - 07:33:50 up 2 days, 5:53, 2 users, load average: 0.00, 0.01, 0.00
Tasks: 28 total, 1 running, 27 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 786432k total, 30756k used, 755676k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26518 root 25 10 12608 1208 932 S 0.3 0.2 0:22.84 top
1 root 18 0 10348 748 624 S 0.0 0.1 0:01.59 init
1407 apache 15 0 176m 2972 1156 S 0.0 0.4 0:00.54 httpd
1410 apache 15 0 176m 2972 1156 S 0.0 0.4 0:00.55 httpd
1416 apache 15 0 176m 2964 1156 S 0.0 0.4 0:00.73 httpd
1422 apache 18 0 176m 3012 1192 S 0.0 0.4 0:00.51 httpd
1637 root 25 10 12608 1208 932 R 0.0 0.2 0:00.00 top
1728 root 15 -4 12604 676 360 S 0.0 0.1 0:00.00 udevd
6040 apache 15 0 176m 2972 1156 S 0.0 0.4 0:00.52 httpd
6044 apache 15 0 176m 2972 1156 S 0.0 0.4 0:00.52 httpd
7313 root 15 0 12064 1744 1304 S 0.0 0.2 0:00.24 bash
7625 root 15 0 62616 1208 652 S 0.0 0.2 0:00.53 sshd
8116 root 15 0 5908 624 500 S 0.0 0.1 0:00.73 syslogd
9585 root 15 0 12168 1744 1292 S 0.0 0.2 0:00.08 bash
13811 root 19 0 21644 932 716 S 0.0 0.1 0:00.00 xinetd
13889 root 15 0 44268 660 268 S 0.0 0.1 0:00.01 vsftpd
17759 root 15 0 176m 3128 1396 S 0.0 0.4 0:00.02 httpd
18365 root 18 0 63296 2080 692 S 0.0 0.3 0:00.58 sendmail
18419 smmsp 18 0 57688 1792 644 S 0.0 0.2 0:00.00 sendmail
19921 root 15 0 20880 1180 592 S 0.0 0.2 0:00.14 crond
20014 root 18 0 46736 820 428 S 0.0 0.1 0:00.00 saslauthd
23875 apache 15 0 176m 2972 1156 S 0.0 0.4 0:00.50 httpd
25676 root 15 0 85908 3336 2616 S 0.0 0.4 0:00.36 sshd
25827 apache 18 0 176m 2972 1156 S 0.0 0.4 0:00.42 httpd
26195 root 16 0 86212 3512 2628 S 0.0 0.4 0:05.36 sshd
28520 apache 18 0 176m 2968 1156 S 0.0 0.4 0:00.55 httpd
28527 apache 18 0 176m 2972 1156 S 0.0 0.4 0:00.45 httpd
30300 root 18 0 46736 552 160 S 0.0 0.1 0:00.00 saslauthd

Could it be I have too many requests at any given time that apache wasn't configured to handle?

The command free gives me:

total used free shared buffers cached
Mem: 786432 29916 756516 0 0 0
-/+ buffers/cache: 29916 756516
Swap: 0 0 0

jay28lee

ASKER

I also use netstat -ant | grep :80 | wc -l to see my tcp connection and it shows 545.

is it too high?

Monis Monther

Yes that is high and should show plenty of resource usage, but these might not be Established connections

do

netstat -ant | grep :80 |grep EST |wc -l

I also noticed something why is your SWAP 0 you dont have any swap partition or swap file, although its not good for apache to use swap it degrades performance and means your memory is at full use but I am not sure if this is advisable.

Also your apache bench results are not normal, These is something strange or that I dont understand

jay28lee

ASKER

the netstat -ant | grep :80 |grep EST |wc -l command give me 103

it shows i only used 30mb of memory and there's still 750mb free (which is my vps hosting package provided)

how much swap space should i allocate in such a configuration with 780mb of memory?

my hosting was suggestion using MPM Event, how should i do that? Will it help? since it's not a managed hosting, i need to do everything by myself. any further suggestion / instruction?

Monis Monther

1-
No MPM Event is an experimental MPM module for the apache 2.2.x server check this Link from the official Apache Docs

http://httpd.apache.org/docs/2.2/mod/event.html

So [IMHO] dont think this is a good option for you

2-
1GB Swap is enough , but I dont think either would fix the problem, but its a step that we ensure to exculde something with (To make sure this is not the problem)

3-
103 Established connections is also a high number , can you check from what IPs they are coming according to your configuration they should all come from your other server, confirm this to exclude an external attack

4-
Still Resource should be higher than this.

run the following command

rpm -V top
rpm -V free
rpm -V ps

After each one you run you should get back to the prompt with no results , if you got any results post them

jay28lee

ASKER

I got:

package top is not installed
package free is not installed
package ps is not installed

after each execution.

But I can use them without problem.

How do I check (3) where the IPs coming from? Because this is an image server, the other one is hosting the content. The IP should be visitors requesting for content and then pull the images from this server with issue.

Monis Monther

Ok its either one of two

1- The packages are installed from source not via rpm and would not be seen from rpm output

2- They have a problem, something is wrong

Now to check where the IPs are coming from

tail -f /var/log/httpd/access_log

This will show you the requests as they are coming to your server each line will show a request including the IP that is requesting

Also the output of netstat will show you the connecting IPs

ASKER CERTIFIED SOLUTION

jay28lee

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial