Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Apache Mod Proxy with Cache throwing 503 responses and (OS 10060) errors

Posted on 2007-04-04
14
Medium Priority
?
5,111 Views
Last Modified: 2007-11-27
We are running Apache 2.2.3 with proxy and caching on Windows Server 2003.  It is proxying to an external site.  When the external site is access directly we do not get 503 HTTP Response Codes, however when we access the site through the proxy it does throw 503 responses periodically.  When Apache Proxy throws a 503 response it also generates the following message in the error.log:

[Wed Apr 04 17:55:41 2007] [error] (OS 10060)A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.  : proxy: HTTPS: attempt to connect to xxx.xxx.xxx.xxx:443 (*) failed

Any help would be greatly appreciated!

Thank you.
0
Comment
Question by:4isteam
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 7
  • 4
  • 3
14 Comments
 
LVL 27

Expert Comment

by:Nopius
ID: 18855751
As said here: http://httpd.apache.org/docs/2.0/mod/mod_proxy.html mod_proxy doesn't proxy https requests.

"The ability to contact remote servers using the SSL/TLS protocol is provided by the SSLProxy* directives of mod_ssl. These additional modules will need to be loaded and configured to take advantage of these features.|
0
 
LVL 15

Accepted Solution

by:
samri earned 750 total points
ID: 18856685
hi 4isteam,

I would agree with Nopius on the proxying SSL request for Apache.

You could start building/customizing the SSL support for your existing Apache installation, OR you could grab a binary of Apache with SSL support from ApacheLounge website:

http://www.apachelounge.com/download/

* never tried any of this, but this should be doable.

Just an off-topic comment -  I would personally like to recommend splitting the functionality of web hosting, and web proxying.  I would run apache, and for proxying, I would use squid (http://squid.acmeconsulting.it/SquidNT25.html) I hope I am not offending anyone with this :)

cheers.
0
 

Author Comment

by:4isteam
ID: 18861637
Thanks for the comments.

A few things:

-We are running SSL proxy and mod_SSL.

-This server is soley used to setup an internal website https://foo/ that will proxy and external site for security reasons.  Would Squid be able to function as a proxy in this manner?

Does anyone have any thoughts on why we are getting the 503 HTTP response errors in the access log and the (OS 10060) errors in the error log?

Tthanks!
0
Use Case: Protecting a Hybrid Cloud Infrastructure

Microsoft Azure is rapidly becoming the norm in dynamic IT environments. This document describes the challenges that organizations face when protecting data in a hybrid cloud IT environment and presents a use case to demonstrate how Acronis Backup protects all data.

 

Author Comment

by:4isteam
ID: 18861718
PS: In other words, the apache server is only functioning as a reverse proxy for this one external (slow) site.
0
 
LVL 27

Expert Comment

by:Nopius
ID: 18861881
> Does anyone have any thoughts on why we are getting the 503 HTTP response errors in the access log
Because it can't connect remote site.
OS Error 10060 - TCP/IP time-out error.
Why it can't connect to it - I don't know. It may be firewall/bad link/dead remote server... whatever
0
 

Author Comment

by:4isteam
ID: 18862215
Give me some credit, buddy.  We can access the remote server just fine and the 503 is "periodic".  It can happen on any file.  We do not get any 503s when we go direct to the remote server.
0
 
LVL 27

Expert Comment

by:Nopius
ID: 18862330
Periodically, how often?
How much is CPU load on server when you get errors?
How many sessions does it forward simultaneously?
If not a secret, what is your configuration for mod_ssl, mod_proxy and mod_cache?
0
 
LVL 15

Expert Comment

by:samri
ID: 18863195
hi,

just got back!

my apology on adding more complexity by intriducing squid to the topic -- but it does not hurt.  For some tutorial on setting up squid to run in acclerator mode here - http://www.visolve.com/squid/squid24s1/httpd_accelerator.php

some standard stuff on apache as proxy : http://httpd.apache.org/docs/2.0/mod/mod_proxy.html

yes -- some config as requested by Nopius would be good (just change the stuff on IP and hostname) to protect the innocent :).

back on apache -- did you try to use a regular http (in stead of https) for your local user.  this is to isolate the problem with apache (and ssl).

such as:
ProxyPass               /           http://www.externalsite.com/
ProxyPassReverse /            http://www.externalsite.com/
0
 

Author Comment

by:4isteam
ID: 18863711
>Periodically, how often?
About 2% of the time.

>How much is CPU load on server when you get errors?
5-10% of CPU max.  Same for disk i/o and network i/o.  Doesn't look like a HW issue.

>How many sessions does it forward simultaneously?
Probably at busy times about 100.

>If not a secret, what is your configuration for mod_ssl, mod_proxy and mod_cache?

mod_ssl:
------------------------------------------------------------------
# Secure (SSL/TLS) connections
#Include conf/extra/httpd-ssl.conf
#
# Note: The following must must be present to support
#       starting without SSL on platforms with no /dev/random equivalent
#       but a statically compiled-in mod_ssl.
#

<IfModule ssl_module>
      SSLRandomSeed startup builtin
      SSLRandomSeed connect builtin

      # see http://www.modssl.org/docs/2.8/ssl_reference.html for more info
      SSLMutex default
      SSLRandomSeed startup builtin
      SSLSessionCache none

      #SSLLogFile      logs/ssl.log
</IfModule>

      # MSIE SSL bug workarounds
      BrowserMatch "MSIE [1-4]" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
      BrowserMatch "MSIE [5-9]" ssl-unclean-shutdown nokeepalive

      # trying to fix SSL 10600 error issues (did not help)
      SetEnv force-proxy-request-1.0 1
        SetEnv proxy-nokeepalive 1

      # turn SSL on for this virtualhost
      SSLEngine On

general:
------------------------------------------------------------------
# turn on extended status handling (this needs to be at the base server config level)
ExtendedStatus On

<VirtualHost *:80>
      # turn on status reporting so we can see what's going on
      <Location /server-status>
            SetHandler server-status

            Order Deny,Allow
            Deny from all
            Allow from 127.0.0.1 xxx.xxx.xxx.xxx
      </Location>

      ###########################
      ## Begin mod_rewrite rules

      # enable mod_rewrite
      RewriteEngine On

      # enable the SSL proxy engine (required to talk to the back-end via SSL)
      SSLProxyEngine On

      # don't rewrite requests for server status page
      RewriteCond %{REQUEST_URI}      !^/server-status

      # redirect users at the root to the correct page
      RewriteRule ^/(.*) https://foo/s55s/TS/index.php [R]

      # rewrite all other queries to the back-end server
      #RewriteRule ^/(.*) https://remoteproxiedsite.com/$1 [P,L]

      ## End mod_rewrite rules
      ###########################
</VirtualHost>

<VirtualHost *:443>
      ###########################
      ## Begin mod_rewrite rules

      # enable mod_rewrite
      RewriteEngine On

      # enable the SSL proxy engine (required to talk to the back-end via SSL)
      SSLProxyEngine On

      # redirect users at the root to the correct page
      RewriteRule ^/$ https://foo/s55s/TS/index.php [R]

      # rewrite all other queries to the back-end server
      RewriteRule ^/(.*) https://remoteproxiedsite.com/$1 [P,L]

      #ProxyPassReverse / https://remoteproxiedsite.com/

      ## End mod_rewrite rules
      ###########################









mod_cache
--------------------------
<IfModule mod_cache.c>
      LoadModule disk_cache_module modules/mod_disk_cache.so

      # If you want to use mod_disk_cache instead of mod_mem_cache,
      # uncomment the line above and comment out the LoadModule line below.
      <IfModule mod_disk_cache.c>
            CacheRoot c:/cacheroot
            CacheEnable disk /
            CacheDirLevels 5
            CacheDirLength 3
      </IfModule>

      #LoadModule mem_cache_module modules/mod_mem_cache.so
      <IfModule mod_mem_cache.c>
            CacheEnable mem /
            MCacheSize 4096
            MCacheMaxObjectCount 100
            MCacheMinObjectSize 1
            MCacheMaxObjectSize 2048
      </IfModule>

      # When acting as a proxy, don't cache the list of security updates
      #CacheDisable http://security.update.server/update-list/
</IfModule>

Thanks for the Squid reference, we are trying that and fastream.com's reverse proxy, today.

Regular HTTP is not an option, as the proxied site requires SSL.


Thanks, guys.  I know this is a tricky one!
0
 

Author Comment

by:4isteam
ID: 18877985
This weekend we tracked down the source of the 503 errors, but we still need help with getting a proxy solution working.

It took some digging in the mod_proxy source code and observations with Ethereal but It turns out mod_proxy does not reuse SSL tunnels between requests and creates a new one for every file request sent.  The site we are connecting to is very client heavy and requires about 60 files just to login.  This is overwhelming one of the servers and we end up with lots of packet drops and resends due to the constant SSL tunnel resetting.

The problems are now
1. Can we do anything with Apache to increase the capacity of SSL connections and avoid the disruptions?
2. Is there another proxy server known to use a single SSL tunnel for all requests from a client?

Any thoughts?
0
 
LVL 27

Assisted Solution

by:Nopius
Nopius earned 750 total points
ID: 18879211
Try to use option 'SSLSessionCache shm://...' http://httpd.apache.org/docs/2.0/mod/mod_ssl.html#sslsessioncache
and 'SSLOptions +OptRenegotiate'/
0
 

Author Comment

by:4isteam
ID: 18918570
We ended up going with a Squid reverse proxy, rather than trying to fix the Apache SSL proxy.  Squid is working nicely as an SSL reverse proxy.
0
 

Author Comment

by:4isteam
ID: 18918575
PS: Thanks Nopius and Samri.
0
 
LVL 15

Expert Comment

by:samri
ID: 18919530
no prob.

cheers.
0

Featured Post

Flexible connectivity for any environment

The KE6900 series can extend and deploy computers with high definition displays across multiple stations in a variety of applications that suit any environment. Expand computer use to stations across multiple rooms with dynamic access.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Most of the applications these days are on Cloud. Cloud is ubiquitous with many service providers in the market. Since it has many benefits such as cost reduction, software updates, remote access, disaster recovery and much more.
This article explains the fundamentals of industrial networking which ultimately is the backbone network which is providing communications for process devices like robots and other not so interesting stuff.
NetCrunch network monitor is a highly extensive platform for network monitoring and alert generation. In this video you'll see a live demo of NetCrunch with most notable features explained in a walk-through manner. You'll also get to know the philos…
In this brief tutorial Pawel from AdRem Software explains how you can quickly find out which services are running on your network, or what are the IP addresses of servers responsible for each service. Software used is freeware NetCrunch Tools (https…

670 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question