Hardware solution for detecting unwanted spiders or bots

We have a problem with unwanted bots crawling our site and stealing our content. Is there any sort of hardware (or less ideal a software) solution that can detect too much traffic coming from a single IP address and block the IP address?

I would think there would be some sort of solution (hardware or software) that detects an abnormal amount of traffic coming from a single IP address and then throttles their connection by increasing the latency for each subsequent request in a given time period. I would of course want to not restrict major search engines from crawling the site, just poachers.

Who is Participating?
grbladesConnect With a Mentor Commented:
The only hardware appliance that would do that sort of thing would be something designed to protect against denial of service attacks.

I cant remember if it was RobWill or PeteLong but I think one of them worked for a company in the denial of service appliance business.
Thats a feature normally done on the web server itself.
Have you analysed the logs to see what the 'user-agent' is reported as for these bots?

What webserver are you using?
If you are using apache then you can block certain user agents. See http://www.blether.com/archives/2006/05/blocking_unwant.php
CoconutTelegraphAuthor Commented:
Thanks for the response. I'm using IIS 6. I haven't checked the user-agents but I would assume they are spoofing those.

Worried about phishing attacks?

90% of attacks start with a phish. It’s critical that IT admins and MSSPs have the right security in place to protect their end users from these phishing attacks. Check out our latest feature brief for tips and tricks to keep your employees off a hackers line!

I dont deal with IIS. If nobody else comes along you might want to ask for this question to be closed and post a new question in the IIS section.
CoconutTelegraphAuthor Commented:
Thanks... I'll leave it open for a bit. Someone outside of EE recommended Tipping Point as a hardware solution. I need to research that and see. I'm surprised there isn't a hardware turnkey solution for this. I would think a lot of people would have to deal with this sort of thing.

It really wouldn't be that complicated either, just a firewall box that detects a certain number of requests or a certain amount of throughput from any single IP address. Once it exceeds the threshold it either gradually increases the latency of each request if there's too many requests or it it rate limits the throughput if someone is consuming too much data. Doesn't seem that complicated.

Anyhow thanks for responding.
CoconutTelegraphAuthor Commented:
Thanks, we may look into DOS protection in the future but we'll handle it in software for now.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.