Apache Denial of Service

So, we had a bot that was "scraping" content from us, images, videos etc. It was able to use up all available slots on apache therefor causing a DoS. Normally this is pretty simple to prevent using IPtables. The problem is our setup is more complex.

Load Balancer > Proxie > Front-End > Applications > - > Shared Assets

Our front ends do not see the "source ip" because the traffic gets sent from the load balance which then inputs a "x-forward-for" into the http header. Wondering what the best way of limiting connections via the x-forward-for are without adding much overhead, ie: mod_security
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Why not limit it on the proxy?
I do not think you would be able to use redirect, conditional redirect referencing a http header that is being passed by the proxy.
Is the proxy setup as reverse?
syscrashAuthor Commented:
Yes, setup as reverse. The proxy in this case is actually not caching the "shared assets" but some day will be. So, the shared asset(front end) is the issue. It sees the source ip4 address only via the x-forward-for in the http header.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
The proxy presumably also has iptables and that is where you can and should block the source ip. Otherwise you have to go up the routing path to the firewall in front of the loadbalancer if any, or up to the router etc.
SolarWinds® VoIP and Network Quality Manager(VNQM)

WAN and VoIP monitoring tools that can help with troubleshooting via an intuitive web interface. Review quality of service data, including jitter, latency, packet loss, and MOS. Troubleshoot call performance and correlate call issues with WAN performance for Cisco and Avaya calls

syscrashAuthor Commented:
Well it would have to be done at the perimeter, which is one option. Any major side affects that anyone can think of? My thought is to limit tcp traffic to port 80 and 8080 something like 50 connections per ip or similar something high enough to not get false positives.
Your apache config allows per Clint sequential requests? I.e. single connection retrieves multiple page/objects?

You could configure apache to proxy the tomcat such that access to port 8080 will not be necessary.
Depending on where you are you might limit proxy servers that are common in some countries/regions.
syscrashAuthor Commented:
Used reverse proxy setup (squid) not as vulnerable to slowloris types of attacks.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.