Apache Mod Proxy with Cache throwing 503 responses and (OS 10060) errors

As said here: http://httpd.apache.org/docs/2.0/mod/mod_proxy.html mod_proxy doesn't proxy https requests.

"The ability to contact remote servers using the SSL/TLS protocol is provided by the SSLProxy* directives of mod_ssl. These additional modules will need to be loaded and configured to take advantage of these features.|

Thanks for the comments.

A few things:

-We are running SSL proxy and mod_SSL.

-This server is soley used to setup an internal website https://foo/ that will proxy and external site for security reasons.  Would Squid be able to function as a proxy in this manner?

Does anyone have any thoughts on why we are getting the 503 HTTP response errors in the access log and the (OS 10060) errors in the error log?

Tthanks!

PS: In other words, the apache server is only functioning as a reverse proxy for this one external (slow) site.

> Does anyone have any thoughts on why we are getting the 503 HTTP response errors in the access log
Because it can't connect remote site.
OS Error 10060 - TCP/IP time-out error.
Why it can't connect to it - I don't know. It may be firewall/bad link/dead remote server... whatever

Give me some credit, buddy.  We can access the remote server just fine and the 503 is "periodic".  It can happen on any file.  We do not get any 503s when we go direct to the remote server.

Periodically, how often?
How much is CPU load on server when you get errors?
How many sessions does it forward simultaneously?
If not a secret, what is your configuration for mod_ssl, mod_proxy and mod_cache?

hi,

just got back!

my apology on adding more complexity by intriducing squid to the topic -- but it does not hurt.  For some tutorial on setting up squid to run in acclerator mode here - http://www.visolve.com/squid/squid24s1/httpd_accelerator.php

some standard stuff on apache as proxy : http://httpd.apache.org/docs/2.0/mod/mod_proxy.html

yes -- some config as requested by Nopius would be good (just change the stuff on IP and hostname) to protect the innocent :).

back on apache -- did you try to use a regular http (in stead of https) for your local user.  this is to isolate the problem with apache (and ssl).

such as:
ProxyPass               /           http://www.externalsite.com/
ProxyPassReverse /            http://www.externalsite.com/

>Periodically, how often?
About 2% of the time.

>How much is CPU load on server when you get errors?
5-10% of CPU max.  Same for disk i/o and network i/o.  Doesn't look like a HW issue.

>How many sessions does it forward simultaneously?
Probably at busy times about 100.

>If not a secret, what is your configuration for mod_ssl, mod_proxy and mod_cache?

mod_ssl:
------------------------------------------------------------------
# Secure (SSL/TLS) connections
#Include conf/extra/httpd-ssl.conf
#
# Note: The following must must be present to support
#       starting without SSL on platforms with no /dev/random equivalent
#       but a statically compiled-in mod_ssl.
#

<IfModule ssl_module>
      SSLRandomSeed startup builtin
      SSLRandomSeed connect builtin

      # see http://www.modssl.org/docs/2.8/ssl_reference.html for more info
      SSLMutex default
      SSLRandomSeed startup builtin
      SSLSessionCache none

      #SSLLogFile      logs/ssl.log
</IfModule>

      # MSIE SSL bug workarounds
      BrowserMatch "MSIE [1-4]" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
      BrowserMatch "MSIE [5-9]" ssl-unclean-shutdown nokeepalive

      # trying to fix SSL 10600 error issues (did not help)
      SetEnv force-proxy-request-1.0 1
        SetEnv proxy-nokeepalive 1

      # turn SSL on for this virtualhost
      SSLEngine On

general:
------------------------------------------------------------------
# turn on extended status handling (this needs to be at the base server config level)
ExtendedStatus On

<VirtualHost *:80>
      # turn on status reporting so we can see what's going on
      <Location /server-status>
            SetHandler server-status

            Order Deny,Allow
            Deny from all
            Allow from 127.0.0.1 xxx.xxx.xxx.xxx
      </Location>

      ###########################
      ## Begin mod_rewrite rules

      # enable mod_rewrite
      RewriteEngine On

      # enable the SSL proxy engine (required to talk to the back-end via SSL)
      SSLProxyEngine On

      # don't rewrite requests for server status page
      RewriteCond %{REQUEST_URI}      !^/server-status

      # redirect users at the root to the correct page
      RewriteRule ^/(.*) https://foo/s55s/TS/index.php [R]

      # rewrite all other queries to the back-end server
      #RewriteRule ^/(.*) https://remoteproxiedsite.com/$1 [P,L]

      ## End mod_rewrite rules
      ###########################
</VirtualHost>

<VirtualHost *:443>
      ###########################
      ## Begin mod_rewrite rules

      # enable mod_rewrite
      RewriteEngine On

      # enable the SSL proxy engine (required to talk to the back-end via SSL)
      SSLProxyEngine On

      # redirect users at the root to the correct page
      RewriteRule ^/$ https://foo/s55s/TS/index.php [R]

      # rewrite all other queries to the back-end server
      RewriteRule ^/(.*) https://remoteproxiedsite.com/$1 [P,L]

      #ProxyPassReverse / https://remoteproxiedsite.com/

      ## End mod_rewrite rules
      ###########################









mod_cache
--------------------------
<IfModule mod_cache.c>
      LoadModule disk_cache_module modules/mod_disk_cache.so

      # If you want to use mod_disk_cache instead of mod_mem_cache,
      # uncomment the line above and comment out the LoadModule line below.
      <IfModule mod_disk_cache.c>
            CacheRoot c:/cacheroot
            CacheEnable disk /
            CacheDirLevels 5
            CacheDirLength 3
      </IfModule>

      #LoadModule mem_cache_module modules/mod_mem_cache.so
      <IfModule mod_mem_cache.c>
            CacheEnable mem /
            MCacheSize 4096
            MCacheMaxObjectCount 100
            MCacheMinObjectSize 1
            MCacheMaxObjectSize 2048
      </IfModule>

      # When acting as a proxy, don't cache the list of security updates
      #CacheDisable http://security.update.server/update-list/
</IfModule>

Thanks for the Squid reference, we are trying that and fastream.com's reverse proxy, today.

Regular HTTP is not an option, as the proxied site requires SSL.


Thanks, guys.  I know this is a tricky one!

This weekend we tracked down the source of the 503 errors, but we still need help with getting a proxy solution working.

It took some digging in the mod_proxy source code and observations with Ethereal but It turns out mod_proxy does not reuse SSL tunnels between requests and creates a new one for every file request sent.  The site we are connecting to is very client heavy and requires about 60 files just to login.  This is overwhelming one of the servers and we end up with lots of packet drops and resends due to the constant SSL tunnel resetting.

The problems are now
1. Can we do anything with Apache to increase the capacity of SSL connections and avoid the disruptions?
2. Is there another proxy server known to use a single SSL tunnel for all requests from a client?

Any thoughts?

We ended up going with a Squid reverse proxy, rather than trying to fix the Apache SSL proxy.  Squid is working nicely as an SSL reverse proxy.

PS: Thanks Nopius and Samri.

no prob.

cheers.