• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1008
  • Last Modified:

Ban or limit site crawlers by ip

Hi

I have a problem of people crawling my site using proxies - seems to be steeling content or at times a minor DOS problem. I have blocked lots of user agents and have mod_evasive in place and working. Helpfully, some of the crawlers are using a malformed URL and getting a 404 - so I can see them in my logs.  

What I am looking for is some kind of logic to ban the ip address which requests a particular URL, temporarily would be fine. Bandwidth throttling would also be fine, but ideally to target ip address of culprits.  The ideal solution would sit in the virtual host - <Location /honeytrap>

I have mod_security installed and think it is possible to use this to as described by B1vr half way down this page: http://www.linuxquestions.org/questions/linux-security-4/apache_mod_security-setup-help-607846/

Though I can't get it to work. The logic could be - if url x is requested, ban all requests from that IP for ten minutes.  One slight complexity is that the server is behind a proxy, so I use X-Forwarded-For in the logs - don't want to ban the downstream proxy!

I know how to block access to the single URL, but the crawlers are then grabbing lots of other pages as well - I can see who they are only because of the malformed POST that I assume is designed to hit the server resources.

Thanks for any help
0
richp10
Asked:
richp10
  • 2
  • 2
1 Solution
 
simoespCommented:
0
 
richp10Author Commented:
No, it's not images it's the main pages of the site (images are all on a cdn anyway).

I wonder whether I could use the solution at the end of this: http://www.experts-exchange.com/Software/Server_Software/Web_Servers/Apache/Q_23722587.html?sfQueryTermInfo=1+10+30+block+ip+mod+secur

Any thoughts on how I could call this blocking programme using X-Forwarded-For and without PHP??!
0
 
richp10Author Commented:
Very good advice - not quite sure yet whether it will work correctly for X-Forwarded-For though this does seem to answer to the main part of my question.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 2
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now