Best remote caching proxy to reduce webserver bandwidth

ecocozza
ecocozza used Ask the Experts™
on
I have my 'main' server located in a great facility.  However bandwidth is a bit on the pricey side there.

I have several servers at budget hosts, which I use for static content to reduce my bandwidth bill at the better facility.

Currently I'm using lighttpd and rsync on these hosts.  This is not optimal for a few reasons:
1. I'm 'caching' everything which I don't need to do, as only a small fraction of content is active at a given time.
2. rsync takes a very long time, as I have hundreds of thousands of small files that it needs to make sure are up to date.

I've come across the flowing caching proxies... some are specifically caching proxies, while others offer configuration options which offer the same end result.

Here are the contenders:
Nginx
Varnish
Squid
Apache
Lighttpd
HAProxy
Unicorn

I would love feedback on even just one of these ideas, or any other caching proxy to help me narrow down the list.

Thanks,

 -Eric
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Because of issues with performance and concurrency, avoid Apache if possible.  Of all the items you listed, you will get the best performance from Nginx.  The only options that could out perform Nginx are based on Erlang, and these Erlang servers (such as YAWS and MochiWeb) would not really do what you need out of the box, although they could easily be modified to do what you need.

You may be interested in reading this:
http://www.sics.se/~joe/apachevsyaws.html
http://www.joeandmotorboat.com/2009/01/03/nginx-vs-yaws-vs-mochiweb-web-server-performance-deathmatch-part-2/

Squid is easy to setup and use, and it was developed to meet your exact needs.  But from the way you wrote your question, I think you already know that.

First choice: Yaws (You could be running in 2 hours if you know Erlang.  Good free support available.)
Second choice: Write a custom mochiweb application (You could be running in 8 hours if you know Erlang.  Good free support available.)
Third choice: Nginx
Fourth choice: Squid






Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial