Solved

Apache Mod Proxy with Cache throwing 503 responses and (OS 10060) errors

Posted on 2007-04-04
14
4,744 Views
Last Modified: 2007-11-27
We are running Apache 2.2.3 with proxy and caching on Windows Server 2003.  It is proxying to an external site.  When the external site is access directly we do not get 503 HTTP Response Codes, however when we access the site through the proxy it does throw 503 responses periodically.  When Apache Proxy throws a 503 response it also generates the following message in the error.log:

[Wed Apr 04 17:55:41 2007] [error] (OS 10060)A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.  : proxy: HTTPS: attempt to connect to xxx.xxx.xxx.xxx:443 (*) failed

Any help would be greatly appreciated!

Thank you.
0
Comment
Question by:4isteam
  • 7
  • 4
  • 3
14 Comments
 
LVL 27

Expert Comment

by:Nopius
Comment Utility
As said here: http://httpd.apache.org/docs/2.0/mod/mod_proxy.html mod_proxy doesn't proxy https requests.

"The ability to contact remote servers using the SSL/TLS protocol is provided by the SSLProxy* directives of mod_ssl. These additional modules will need to be loaded and configured to take advantage of these features.|
0
 
LVL 15

Accepted Solution

by:
samri earned 250 total points
Comment Utility
hi 4isteam,

I would agree with Nopius on the proxying SSL request for Apache.

You could start building/customizing the SSL support for your existing Apache installation, OR you could grab a binary of Apache with SSL support from ApacheLounge website:

http://www.apachelounge.com/download/

* never tried any of this, but this should be doable.

Just an off-topic comment -  I would personally like to recommend splitting the functionality of web hosting, and web proxying.  I would run apache, and for proxying, I would use squid (http://squid.acmeconsulting.it/SquidNT25.html) I hope I am not offending anyone with this :)

cheers.
0
 

Author Comment

by:4isteam
Comment Utility
Thanks for the comments.

A few things:

-We are running SSL proxy and mod_SSL.

-This server is soley used to setup an internal website https://foo/ that will proxy and external site for security reasons.  Would Squid be able to function as a proxy in this manner?

Does anyone have any thoughts on why we are getting the 503 HTTP response errors in the access log and the (OS 10060) errors in the error log?

Tthanks!
0
 

Author Comment

by:4isteam
Comment Utility
PS: In other words, the apache server is only functioning as a reverse proxy for this one external (slow) site.
0
 
LVL 27

Expert Comment

by:Nopius
Comment Utility
> Does anyone have any thoughts on why we are getting the 503 HTTP response errors in the access log
Because it can't connect remote site.
OS Error 10060 - TCP/IP time-out error.
Why it can't connect to it - I don't know. It may be firewall/bad link/dead remote server... whatever
0
 

Author Comment

by:4isteam
Comment Utility
Give me some credit, buddy.  We can access the remote server just fine and the 503 is "periodic".  It can happen on any file.  We do not get any 503s when we go direct to the remote server.
0
 
LVL 27

Expert Comment

by:Nopius
Comment Utility
Periodically, how often?
How much is CPU load on server when you get errors?
How many sessions does it forward simultaneously?
If not a secret, what is your configuration for mod_ssl, mod_proxy and mod_cache?
0
Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

 
LVL 15

Expert Comment

by:samri
Comment Utility
hi,

just got back!

my apology on adding more complexity by intriducing squid to the topic -- but it does not hurt.  For some tutorial on setting up squid to run in acclerator mode here - http://www.visolve.com/squid/squid24s1/httpd_accelerator.php

some standard stuff on apache as proxy : http://httpd.apache.org/docs/2.0/mod/mod_proxy.html

yes -- some config as requested by Nopius would be good (just change the stuff on IP and hostname) to protect the innocent :).

back on apache -- did you try to use a regular http (in stead of https) for your local user.  this is to isolate the problem with apache (and ssl).

such as:
ProxyPass               /           http://www.externalsite.com/
ProxyPassReverse /            http://www.externalsite.com/
0
 

Author Comment

by:4isteam
Comment Utility
>Periodically, how often?
About 2% of the time.

>How much is CPU load on server when you get errors?
5-10% of CPU max.  Same for disk i/o and network i/o.  Doesn't look like a HW issue.

>How many sessions does it forward simultaneously?
Probably at busy times about 100.

>If not a secret, what is your configuration for mod_ssl, mod_proxy and mod_cache?

mod_ssl:
------------------------------------------------------------------
# Secure (SSL/TLS) connections
#Include conf/extra/httpd-ssl.conf
#
# Note: The following must must be present to support
#       starting without SSL on platforms with no /dev/random equivalent
#       but a statically compiled-in mod_ssl.
#

<IfModule ssl_module>
      SSLRandomSeed startup builtin
      SSLRandomSeed connect builtin

      # see http://www.modssl.org/docs/2.8/ssl_reference.html for more info
      SSLMutex default
      SSLRandomSeed startup builtin
      SSLSessionCache none

      #SSLLogFile      logs/ssl.log
</IfModule>

      # MSIE SSL bug workarounds
      BrowserMatch "MSIE [1-4]" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
      BrowserMatch "MSIE [5-9]" ssl-unclean-shutdown nokeepalive

      # trying to fix SSL 10600 error issues (did not help)
      SetEnv force-proxy-request-1.0 1
        SetEnv proxy-nokeepalive 1

      # turn SSL on for this virtualhost
      SSLEngine On

general:
------------------------------------------------------------------
# turn on extended status handling (this needs to be at the base server config level)
ExtendedStatus On

<VirtualHost *:80>
      # turn on status reporting so we can see what's going on
      <Location /server-status>
            SetHandler server-status

            Order Deny,Allow
            Deny from all
            Allow from 127.0.0.1 xxx.xxx.xxx.xxx
      </Location>

      ###########################
      ## Begin mod_rewrite rules

      # enable mod_rewrite
      RewriteEngine On

      # enable the SSL proxy engine (required to talk to the back-end via SSL)
      SSLProxyEngine On

      # don't rewrite requests for server status page
      RewriteCond %{REQUEST_URI}      !^/server-status

      # redirect users at the root to the correct page
      RewriteRule ^/(.*) https://foo/s55s/TS/index.php [R]

      # rewrite all other queries to the back-end server
      #RewriteRule ^/(.*) https://remoteproxiedsite.com/$1 [P,L]

      ## End mod_rewrite rules
      ###########################
</VirtualHost>

<VirtualHost *:443>
      ###########################
      ## Begin mod_rewrite rules

      # enable mod_rewrite
      RewriteEngine On

      # enable the SSL proxy engine (required to talk to the back-end via SSL)
      SSLProxyEngine On

      # redirect users at the root to the correct page
      RewriteRule ^/$ https://foo/s55s/TS/index.php [R]

      # rewrite all other queries to the back-end server
      RewriteRule ^/(.*) https://remoteproxiedsite.com/$1 [P,L]

      #ProxyPassReverse / https://remoteproxiedsite.com/

      ## End mod_rewrite rules
      ###########################









mod_cache
--------------------------
<IfModule mod_cache.c>
      LoadModule disk_cache_module modules/mod_disk_cache.so

      # If you want to use mod_disk_cache instead of mod_mem_cache,
      # uncomment the line above and comment out the LoadModule line below.
      <IfModule mod_disk_cache.c>
            CacheRoot c:/cacheroot
            CacheEnable disk /
            CacheDirLevels 5
            CacheDirLength 3
      </IfModule>

      #LoadModule mem_cache_module modules/mod_mem_cache.so
      <IfModule mod_mem_cache.c>
            CacheEnable mem /
            MCacheSize 4096
            MCacheMaxObjectCount 100
            MCacheMinObjectSize 1
            MCacheMaxObjectSize 2048
      </IfModule>

      # When acting as a proxy, don't cache the list of security updates
      #CacheDisable http://security.update.server/update-list/
</IfModule>

Thanks for the Squid reference, we are trying that and fastream.com's reverse proxy, today.

Regular HTTP is not an option, as the proxied site requires SSL.


Thanks, guys.  I know this is a tricky one!
0
 

Author Comment

by:4isteam
Comment Utility
This weekend we tracked down the source of the 503 errors, but we still need help with getting a proxy solution working.

It took some digging in the mod_proxy source code and observations with Ethereal but It turns out mod_proxy does not reuse SSL tunnels between requests and creates a new one for every file request sent.  The site we are connecting to is very client heavy and requires about 60 files just to login.  This is overwhelming one of the servers and we end up with lots of packet drops and resends due to the constant SSL tunnel resetting.

The problems are now
1. Can we do anything with Apache to increase the capacity of SSL connections and avoid the disruptions?
2. Is there another proxy server known to use a single SSL tunnel for all requests from a client?

Any thoughts?
0
 
LVL 27

Assisted Solution

by:Nopius
Nopius earned 250 total points
Comment Utility
Try to use option 'SSLSessionCache shm://...' http://httpd.apache.org/docs/2.0/mod/mod_ssl.html#sslsessioncache
and 'SSLOptions +OptRenegotiate'/
0
 

Author Comment

by:4isteam
Comment Utility
We ended up going with a Squid reverse proxy, rather than trying to fix the Apache SSL proxy.  Squid is working nicely as an SSL reverse proxy.
0
 

Author Comment

by:4isteam
Comment Utility
PS: Thanks Nopius and Samri.
0
 
LVL 15

Expert Comment

by:samri
Comment Utility
no prob.

cheers.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

If your site has a few sections that need to be secure when data is transmitted between the server and local computer, such as a /order/ section for ordering or /customer/ which contains customer data, etc it would of course be recommended to secure…
Even if you have implemented a Mobile Device Management solution company wide, it is a good idea to make sure you are taking into account all of the major risks to your electronic protected health information (ePHI).
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now