Solved

Ban or limit site crawlers by ip

Posted on 2010-09-02
4
991 Views
Last Modified: 2012-05-10
Hi

I have a problem of people crawling my site using proxies - seems to be steeling content or at times a minor DOS problem. I have blocked lots of user agents and have mod_evasive in place and working. Helpfully, some of the crawlers are using a malformed URL and getting a 404 - so I can see them in my logs.  

What I am looking for is some kind of logic to ban the ip address which requests a particular URL, temporarily would be fine. Bandwidth throttling would also be fine, but ideally to target ip address of culprits.  The ideal solution would sit in the virtual host - <Location /honeytrap>

I have mod_security installed and think it is possible to use this to as described by B1vr half way down this page: http://www.linuxquestions.org/questions/linux-security-4/apache_mod_security-setup-help-607846/

Though I can't get it to work. The logic could be - if url x is requested, ban all requests from that IP for ten minutes.  One slight complexity is that the server is behind a proxy, so I use X-Forwarded-For in the logs - don't want to ban the downstream proxy!

I know how to block access to the single URL, but the crawlers are then grabbing lots of other pages as well - I can see who they are only because of the malformed POST that I assume is designed to hit the server resources.

Thanks for any help
0
Comment
Question by:richp10
  • 2
  • 2
4 Comments
 
LVL 3

Expert Comment

by:simoesp
ID: 33588397
0
 

Author Comment

by:richp10
ID: 33588507
No, it's not images it's the main pages of the site (images are all on a cdn anyway).

I wonder whether I could use the solution at the end of this: http://www.experts-exchange.com/Software/Server_Software/Web_Servers/Apache/Q_23722587.html?sfQueryTermInfo=1+10+30+block+ip+mod+secur

Any thoughts on how I could call this blocking programme using X-Forwarded-For and without PHP??!
0
 
LVL 3

Accepted Solution

by:
simoesp earned 500 total points
ID: 33588658
0
 

Author Closing Comment

by:richp10
ID: 33607404
Very good advice - not quite sure yet whether it will work correctly for X-Forwarded-For though this does seem to answer to the main part of my question.
0

Featured Post

Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

Join & Write a Comment

In my time as an SEO for the last 2 years and in the questions I have assisted with on here I have always seen the need to redirect from non-www urls to their www versions. For instance redirecting http://domain.com (http://domain.com) to http…
If you've heard about htaccess and it sounds like it does what you want, but you're not sure how it works... well, you're in the right place. Read on. Some Basics #1. It's a file and its filename is .htaccess (yes, with a dot in the front). #…
Access reports are powerful and flexible. Learn how to create a query and then a grouped report using the wizard. Modify the report design after the wizard is done to make it look better. There will be another video to explain how to put the final p…
When you create an app prototype with Adobe XD, you can insert system screens -- sharing or Control Center, for example -- with just a few clicks. This video shows you how. You can take the full course on Experts Exchange at http://bit.ly/XDcourse.

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now