• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 234
  • Last Modified:

Hardware solution for detecting unwanted spiders or bots

We have a problem with unwanted bots crawling our site and stealing our content. Is there any sort of hardware (or less ideal a software) solution that can detect too much traffic coming from a single IP address and block the IP address?

I would think there would be some sort of solution (hardware or software) that detects an abnormal amount of traffic coming from a single IP address and then throttles their connection by increasing the latency for each subsequent request in a given time period. I would of course want to not restrict major search engines from crawling the site, just poachers.

Thanks!!
0
CoconutTelegraph
Asked:
CoconutTelegraph
  • 3
  • 3
1 Solution
 
grbladesCommented:
Thats a feature normally done on the web server itself.
Have you analysed the logs to see what the 'user-agent' is reported as for these bots?

What webserver are you using?
If you are using apache then you can block certain user agents. See http://www.blether.com/archives/2006/05/blocking_unwant.php
0
 
CoconutTelegraphAuthor Commented:
Thanks for the response. I'm using IIS 6. I haven't checked the user-agents but I would assume they are spoofing those.

0
 
grbladesCommented:
I dont deal with IIS. If nobody else comes along you might want to ask for this question to be closed and post a new question in the IIS section.
0
 The Evil-ution of Network Security Threats

What are the hacks that forever changed the security industry? To answer that question, we created an exciting new eBook that takes you on a trip through hacking history. It explores the top hacks from the 80s to 2010s, why they mattered, and how the security industry responded.

 
CoconutTelegraphAuthor Commented:
Thanks... I'll leave it open for a bit. Someone outside of EE recommended Tipping Point as a hardware solution. I need to research that and see. I'm surprised there isn't a hardware turnkey solution for this. I would think a lot of people would have to deal with this sort of thing.

It really wouldn't be that complicated either, just a firewall box that detects a certain number of requests or a certain amount of throughput from any single IP address. Once it exceeds the threshold it either gradually increases the latency of each request if there's too many requests or it it rate limits the throughput if someone is consuming too much data. Doesn't seem that complicated.

Anyhow thanks for responding.
0
 
grbladesCommented:
The only hardware appliance that would do that sort of thing would be something designed to protect against denial of service attacks.

I cant remember if it was RobWill or PeteLong but I think one of them worked for a company in the denial of service appliance business.
0
 
CoconutTelegraphAuthor Commented:
Thanks, we may look into DOS protection in the future but we'll handle it in software for now.
0

Featured Post

Cyber Threats to Small Businesses (Part 2)

The evolving cybersecurity landscape presents SMBs with a host of new threats to their clients, their data, and their bottom line. In part 2 of this blog series, learn three quick processes Webroot’s CISO, Gary Hayslip, recommends to help small businesses beat modern threats.

  • 3
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now