Apache SSL ReverseProxy fails when moved from 2.2.15 to 2.2.16

I have an odd problem with a reverse proxy setup breaking when moved from Apache 2.2.15 on Centos6 to 2.2.16 on Debian Squeeze.

Basic setup is to use Apache to proxy to JBoss 4.2.3 on backend. The reverse proxy on port 80 has always beena snap.  It gets interesting when I try to reverse proxy SSL.
Here is the JBoss connection definition:

<Connector port="9080" address="${jboss.bind.address}"    
         maxThreads="250" maxHttpHeaderSize="8192"
         emptySessionPath="false" protocol="HTTP/1.1"
         enableLookups="false" redirectPort="443" acceptCount="100"
         connectionTimeout="20000" disableUploadTimeout="true" 
	proxyName="public.example.com" proxyPort="80"/>
	
<Connector port="9443" protocol="HTTP/1.1" SSLEnabled="true"
        maxThreads="150" scheme="https" secure="true"
        clientAuth="false"  proxyName="public.example.com" proxyPort="443"
	keystoreFile="${jboss.server.home.dir}/conf/tcsserver_keystore.jks"
        keystorePass="password" sslProtocol = "TLS" />	

Open in new window

Notice the redirectPort setting in the connection for port 9080 to 443.  If I leave this at 9443 I do not have any problems, although as soon as the user goes to a secure page, all the traffic is on 9443 and that presents a problem with some firewalls that only allow SSL over 443.

Here are the Virtual Hosts in Apache. This all works fine on the Centos6 machine.  Note that several apps are running in JBoss and so we rewrite to a specific app context:
For Port 80....
<VirtualHost 274.x.y.40:80>
    ServerName public.example.com
  ProxyRequests off
  ProxyPreservehost on
  RewriteEngine on

<Proxy http://public.example.com:9080/>
Order Deny,Allow
Allow from all
</Proxy>
RewriteRule ^/$  http://public.example.com/public/ [R=302]

ProxyPass /  http://127.0.0.1:9080/
ProxyPassReverse /  http://127.0.0.1:9080/

<Location / >
  Order allow,deny
  Allow from all
</Location>
</VirtualHost>

Open in new window

For SSL (443)...
<VirtualHost 274.x.y.40:443>
ServerName public.example.com
SSLEngine on
SSLProxyEngine on
RewriteEngine on
SSLProtocol all -SSLv2
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW

ProxyReceiveBufferSize 4096
ProxyPreserveHost On
ProxyRequests Off

<Proxy public.example.com:9443/>
Order Deny,Allow
Allow from all
</Proxy>
 
ProxyPass /public/     https://127.0.0.1:9443/public/
ProxyPassReverse /public/        https://127.0.0.1:9443/public/
RewriteRule ^/public$ /public/ [R]

[Certificate defintions....]
</VirtualHost>

Open in new window

Same config on the Debian machine will just hang as soon as user tries to navigate to a page with a CONFIDENTIAL transport guarantee.

Couple of things in the Apache logs on the Debian machine look odd but I cannot trace them down:

1.  Never saw this before: [warn] Init: (example.com:443) You configured HTTP(80) on the standard HTTPS(443) port!

2.  Lots of this in the log:   [client 127.0.0.1] File does not exist: /etc/apache2/htdocs

As I said, if I have the 9080 port redirect to 9443, all is well except traffic is on port 9443; trying to redirect to 443 is when we have a problem and it works on the Centos6 with 2.2.15 but not on Debian with 2.2.16.   The symptom is that it just hangs and them times out.
Bartleby429Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

arnoldCommented:
Why are using SSL on the inside?

Your use of the reverse proxy seems too fold
1) deals with shielding the jboss service from direct attack.
2) deals with having the apache reverse proxy terminate the secure connection.

In your 443 config, your proxy the connection via a secure connection. What certificate sits on the 9443 port of the jboss config? Is that certificate still valid? Is the error relates to the certificate being "invalid?" Expired, can not be verified?

Any thought of configuring the 443 proxy to a non SSL configuration jboss  setup with a specific home dir that is not accessible via the 80/9080 port combination?

I think there is an option you can set in the SSL apache config to not validate the certificate on the 9443 port if the jboss setup.
0
arnoldCommented:
It is not clear from which log your posted errors are coming from.
0
arnoldCommented:
Your port 80 rewrite rule for request of /public redirects back to itself or should it be https:// on line 11 of the port 80 config?
0
The Firewall Audit Checklist

Preparing for a firewall audit today is almost impossible.
AlgoSec, together with some of the largest global organizations and auditors, has created a checklist to follow when preparing for your firewall audit. Simplify risk mitigation while staying compliant all of the time!

Bartleby429Author Commented:
Same certificate is used in JBoss as in Apache.

There isn't any reason to use SSL on the inside and I will deal with that as soon as this other issue is resolved.

Not all the site content is encrypted, so I think the  port 80 rewrite rule is correct.
0
arnoldCommented:
Look at whether using worker process within apache that will interact with the jboss system rather than redirecting/proxying request.

The example deals with setup within
http://docs.jboss.org/jbossas/docs/Clustering_Guide/beta422/html/clustering-http-nodes.html
https://community.jboss.org/thread/38749?start=0&tstart=0
0
Bartleby429Author Commented:
So far as I can tell, use of mod_jk for what we are doing is not favored, and the best practice is using mod_proxy.  It should not be anything as complicated as the mod_jk solution, and remember that this did work in 2.2.15 as expected.
0
arnoldCommented:
The difficulty I have is that when going from version to version, the this worked might be impacted by a change.
A slight misconfiguration. i.e. in the one that worked included a module, that via an oversight this instance has not enabled.

It's similar to a person going from point A to point B. This type the person rented a truck, and says that there is an issue going from point A to point B eventhough this route has been traveled for years prior. In this case the issue is trucks can not enter a portion of the route such that a different approach is need to actually get to point B while driving a truck.


Unfortunately, you have to determine where the break is. Is the issue with the jboss portion not responding on the port referenced. Or is it within apache not proxying or something is corrupting the request that misdirects.

looking at the URl, apache logs, jboss/tomcat logs to see what is going on.
..
.
.

I have no way of knowing within which portion your issue starts.
0
Bartleby429Author Commented:
Checking also to see if this might be a firewall issue....
0
arnoldCommented:
You are proxying localhost traffic, so firewall does/should not come into play.
0
Bartleby429Author Commented:
What do you make of the fact that the first time anyone hits the page that requires the redirect to the encrypted page after a JBoss restart, it works; but all subsequent hits on the page just hang until it times out?
0
arnoldCommented:
What do the logs on both jboss, and apache say?  You may have resource issues.

Look at setting up a mod_jk for testing.
Resource consumption by apache, jboss/tomcat, any database?
0
Bartleby429Author Commented:
Upgraded the server to Debian 7 to get Apache 2.2.22.  No more problems.  Seems to have been isolated to Apache 2.2.16
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Bartleby429Author Commented:
Issue was resolved by our own actions.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Apache Web Server

From novice to tech pro — start learning today.