• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1006
  • Last Modified:

Ban or limit site crawlers by ip

Hi

I have a problem of people crawling my site using proxies - seems to be steeling content or at times a minor DOS problem. I have blocked lots of user agents and have mod_evasive in place and working. Helpfully, some of the crawlers are using a malformed URL and getting a 404 - so I can see them in my logs.  

What I am looking for is some kind of logic to ban the ip address which requests a particular URL, temporarily would be fine. Bandwidth throttling would also be fine, but ideally to target ip address of culprits.  The ideal solution would sit in the virtual host - <Location /honeytrap>

I have mod_security installed and think it is possible to use this to as described by B1vr half way down this page: http://www.linuxquestions.org/questions/linux-security-4/apache_mod_security-setup-help-607846/

Though I can't get it to work. The logic could be - if url x is requested, ban all requests from that IP for ten minutes.  One slight complexity is that the server is behind a proxy, so I use X-Forwarded-For in the logs - don't want to ban the downstream proxy!

I know how to block access to the single URL, but the crawlers are then grabbing lots of other pages as well - I can see who they are only because of the malformed POST that I assume is designed to hit the server resources.

Thanks for any help
0
richp10
Asked:
richp10
  • 2
  • 2
1 Solution
 
simoespCommented:
0
 
richp10Author Commented:
No, it's not images it's the main pages of the site (images are all on a cdn anyway).

I wonder whether I could use the solution at the end of this: http://www.experts-exchange.com/Software/Server_Software/Web_Servers/Apache/Q_23722587.html?sfQueryTermInfo=1+10+30+block+ip+mod+secur

Any thoughts on how I could call this blocking programme using X-Forwarded-For and without PHP??!
0
 
richp10Author Commented:
Very good advice - not quite sure yet whether it will work correctly for X-Forwarded-For though this does seem to answer to the main part of my question.
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 2
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now