Apache Mod Proxy with Cache throwing 503 responses and (OS 10060) errors

We are running Apache 2.2.3 with proxy and caching on Windows Server 2003.  It is proxying to an external site.  When the external site is access directly we do not get 503 HTTP Response Codes, however when we access the site through the proxy it does throw 503 responses periodically.  When Apache Proxy throws a 503 response it also generates the following message in the error.log:

[Wed Apr 04 17:55:41 2007] [error] (OS 10060)A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.  : proxy: HTTPS: attempt to connect to xxx.xxx.xxx.xxx:443 (*) failed

Any help would be greatly appreciated!

Thank you.
4isteamAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Artysystem administratorCommented:
As said here: http://httpd.apache.org/docs/2.0/mod/mod_proxy.html mod_proxy doesn't proxy https requests.

"The ability to contact remote servers using the SSL/TLS protocol is provided by the SSLProxy* directives of mod_ssl. These additional modules will need to be loaded and configured to take advantage of these features.|
0
samriCommented:
hi 4isteam,

I would agree with Nopius on the proxying SSL request for Apache.

You could start building/customizing the SSL support for your existing Apache installation, OR you could grab a binary of Apache with SSL support from ApacheLounge website:

http://www.apachelounge.com/download/

* never tried any of this, but this should be doable.

Just an off-topic comment -  I would personally like to recommend splitting the functionality of web hosting, and web proxying.  I would run apache, and for proxying, I would use squid (http://squid.acmeconsulting.it/SquidNT25.html) I hope I am not offending anyone with this :)

cheers.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
4isteamAuthor Commented:
Thanks for the comments.

A few things:

-We are running SSL proxy and mod_SSL.

-This server is soley used to setup an internal website https://foo/ that will proxy and external site for security reasons.  Would Squid be able to function as a proxy in this manner?

Does anyone have any thoughts on why we are getting the 503 HTTP response errors in the access log and the (OS 10060) errors in the error log?

Tthanks!
0
10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

4isteamAuthor Commented:
PS: In other words, the apache server is only functioning as a reverse proxy for this one external (slow) site.
0
Artysystem administratorCommented:
> Does anyone have any thoughts on why we are getting the 503 HTTP response errors in the access log
Because it can't connect remote site.
OS Error 10060 - TCP/IP time-out error.
Why it can't connect to it - I don't know. It may be firewall/bad link/dead remote server... whatever
0
4isteamAuthor Commented:
Give me some credit, buddy.  We can access the remote server just fine and the 503 is "periodic".  It can happen on any file.  We do not get any 503s when we go direct to the remote server.
0
Artysystem administratorCommented:
Periodically, how often?
How much is CPU load on server when you get errors?
How many sessions does it forward simultaneously?
If not a secret, what is your configuration for mod_ssl, mod_proxy and mod_cache?
0
samriCommented:
hi,

just got back!

my apology on adding more complexity by intriducing squid to the topic -- but it does not hurt.  For some tutorial on setting up squid to run in acclerator mode here - http://www.visolve.com/squid/squid24s1/httpd_accelerator.php

some standard stuff on apache as proxy : http://httpd.apache.org/docs/2.0/mod/mod_proxy.html

yes -- some config as requested by Nopius would be good (just change the stuff on IP and hostname) to protect the innocent :).

back on apache -- did you try to use a regular http (in stead of https) for your local user.  this is to isolate the problem with apache (and ssl).

such as:
ProxyPass               /           http://www.externalsite.com/
ProxyPassReverse /            http://www.externalsite.com/
0
4isteamAuthor Commented:
>Periodically, how often?
About 2% of the time.

>How much is CPU load on server when you get errors?
5-10% of CPU max.  Same for disk i/o and network i/o.  Doesn't look like a HW issue.

>How many sessions does it forward simultaneously?
Probably at busy times about 100.

>If not a secret, what is your configuration for mod_ssl, mod_proxy and mod_cache?

mod_ssl:
------------------------------------------------------------------
# Secure (SSL/TLS) connections
#Include conf/extra/httpd-ssl.conf
#
# Note: The following must must be present to support
#       starting without SSL on platforms with no /dev/random equivalent
#       but a statically compiled-in mod_ssl.
#

<IfModule ssl_module>
      SSLRandomSeed startup builtin
      SSLRandomSeed connect builtin

      # see http://www.modssl.org/docs/2.8/ssl_reference.html for more info
      SSLMutex default
      SSLRandomSeed startup builtin
      SSLSessionCache none

      #SSLLogFile      logs/ssl.log
</IfModule>

      # MSIE SSL bug workarounds
      BrowserMatch "MSIE [1-4]" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
      BrowserMatch "MSIE [5-9]" ssl-unclean-shutdown nokeepalive

      # trying to fix SSL 10600 error issues (did not help)
      SetEnv force-proxy-request-1.0 1
        SetEnv proxy-nokeepalive 1

      # turn SSL on for this virtualhost
      SSLEngine On

general:
------------------------------------------------------------------
# turn on extended status handling (this needs to be at the base server config level)
ExtendedStatus On

<VirtualHost *:80>
      # turn on status reporting so we can see what's going on
      <Location /server-status>
            SetHandler server-status

            Order Deny,Allow
            Deny from all
            Allow from 127.0.0.1 xxx.xxx.xxx.xxx
      </Location>

      ###########################
      ## Begin mod_rewrite rules

      # enable mod_rewrite
      RewriteEngine On

      # enable the SSL proxy engine (required to talk to the back-end via SSL)
      SSLProxyEngine On

      # don't rewrite requests for server status page
      RewriteCond %{REQUEST_URI}      !^/server-status

      # redirect users at the root to the correct page
      RewriteRule ^/(.*) https://foo/s55s/TS/index.php [R]

      # rewrite all other queries to the back-end server
      #RewriteRule ^/(.*) https://remoteproxiedsite.com/$1 [P,L]

      ## End mod_rewrite rules
      ###########################
</VirtualHost>

<VirtualHost *:443>
      ###########################
      ## Begin mod_rewrite rules

      # enable mod_rewrite
      RewriteEngine On

      # enable the SSL proxy engine (required to talk to the back-end via SSL)
      SSLProxyEngine On

      # redirect users at the root to the correct page
      RewriteRule ^/$ https://foo/s55s/TS/index.php [R]

      # rewrite all other queries to the back-end server
      RewriteRule ^/(.*) https://remoteproxiedsite.com/$1 [P,L]

      #ProxyPassReverse / https://remoteproxiedsite.com/

      ## End mod_rewrite rules
      ###########################









mod_cache
--------------------------
<IfModule mod_cache.c>
      LoadModule disk_cache_module modules/mod_disk_cache.so

      # If you want to use mod_disk_cache instead of mod_mem_cache,
      # uncomment the line above and comment out the LoadModule line below.
      <IfModule mod_disk_cache.c>
            CacheRoot c:/cacheroot
            CacheEnable disk /
            CacheDirLevels 5
            CacheDirLength 3
      </IfModule>

      #LoadModule mem_cache_module modules/mod_mem_cache.so
      <IfModule mod_mem_cache.c>
            CacheEnable mem /
            MCacheSize 4096
            MCacheMaxObjectCount 100
            MCacheMinObjectSize 1
            MCacheMaxObjectSize 2048
      </IfModule>

      # When acting as a proxy, don't cache the list of security updates
      #CacheDisable http://security.update.server/update-list/
</IfModule>

Thanks for the Squid reference, we are trying that and fastream.com's reverse proxy, today.

Regular HTTP is not an option, as the proxied site requires SSL.


Thanks, guys.  I know this is a tricky one!
0
4isteamAuthor Commented:
This weekend we tracked down the source of the 503 errors, but we still need help with getting a proxy solution working.

It took some digging in the mod_proxy source code and observations with Ethereal but It turns out mod_proxy does not reuse SSL tunnels between requests and creates a new one for every file request sent.  The site we are connecting to is very client heavy and requires about 60 files just to login.  This is overwhelming one of the servers and we end up with lots of packet drops and resends due to the constant SSL tunnel resetting.

The problems are now
1. Can we do anything with Apache to increase the capacity of SSL connections and avoid the disruptions?
2. Is there another proxy server known to use a single SSL tunnel for all requests from a client?

Any thoughts?
0
Artysystem administratorCommented:
Try to use option 'SSLSessionCache shm://...' http://httpd.apache.org/docs/2.0/mod/mod_ssl.html#sslsessioncache
and 'SSLOptions +OptRenegotiate'/
0
4isteamAuthor Commented:
We ended up going with a Squid reverse proxy, rather than trying to fix the Apache SSL proxy.  Squid is working nicely as an SSL reverse proxy.
0
4isteamAuthor Commented:
PS: Thanks Nopius and Samri.
0
samriCommented:
no prob.

cheers.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Web Servers

From novice to tech pro — start learning today.