• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 322
  • Last Modified:

Need to block BOTS for website

One site get a lot of bandwith because to many bots reach it everyday.
I'm looking for other way to stop this for now.
I edit the HTACCESS file to deny some ip address and some country.
The site will be completely removed and recreated because it's use old unsafe code.
I'm wondering if the bots will still come after even if the site is recreate?

Any advice is welcome

1 Solution
If these are malevolent bots, you will be left with editing the .htaccess for each one of them, but if they are regular legit bots and crawlers you could easily help yourself with a robots.txt file: http://www.robotstxt.org/robotstxt.html
As torimar says, you can use a robots.txt file to influence (not control, but influence) "legitimate" bots and crawlers.

There is absolutely nothing that forces any bot/crawler to obey a robots.txt file. Think of it as asking someone to be polite. They can still be a jerk.

Bots can come back even after a site is re-written. Chances are they will.

If you want to block specific IP sub-nets/countries, I'd do it at the firewall. If you don't want those IPs scanning your website, why allow them to waste your bandwidth (making all those HTTP requests) and CPU (parsing and utilizing .htaccess)? Drop the packets at the firewall and sooner or later they'll get the hint - in the mean time, you minimize the negative impact on your systems.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

WEBINAR: 10 Easy Ways to Lose a Password

Join us on June 27th at 8 am PDT to learn about the methods that hackers use to lift real, working credentials from even the most security-savvy employees. We'll cover the importance of multi-factor authentication and how these solutions can better protect your business!

Tackle projects and never again get stuck behind a technical roadblock.
Join Now