[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now


Need to block BOTS for website

Posted on 2011-05-03
Medium Priority
Last Modified: 2012-05-11
One site get a lot of bandwith because to many bots reach it everyday.
I'm looking for other way to stop this for now.
I edit the HTACCESS file to deny some ip address and some country.
The site will be completely removed and recreated because it's use old unsafe code.
I'm wondering if the bots will still come after even if the site is recreate?

Any advice is welcome

Question by:lenamtl
LVL 35

Accepted Solution

torimar earned 2000 total points
ID: 35560163
If these are malevolent bots, you will be left with editing the .htaccess for each one of them, but if they are regular legit bots and crawlers you could easily help yourself with a robots.txt file: http://www.robotstxt.org/robotstxt.html
LVL 34

Expert Comment

ID: 35618719
As torimar says, you can use a robots.txt file to influence (not control, but influence) "legitimate" bots and crawlers.

There is absolutely nothing that forces any bot/crawler to obey a robots.txt file. Think of it as asking someone to be polite. They can still be a jerk.

Bots can come back even after a site is re-written. Chances are they will.

If you want to block specific IP sub-nets/countries, I'd do it at the firewall. If you don't want those IPs scanning your website, why allow them to waste your bandwidth (making all those HTTP requests) and CPU (parsing and utilizing .htaccess)? Drop the packets at the firewall and sooner or later they'll get the hint - in the mean time, you minimize the negative impact on your systems.

Featured Post

New Tabletop Appliances Blow Competitors Away!

WatchGuard’s new T15, T35 and T55 tabletop UTMs provide the highest-performing security inspection in their class, allowing users at small offices, home offices and distributed enterprises to experience blazing-fast Internet speeds without sacrificing enterprise-grade security.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Considering today’s continual security threats, which affect Information technology networks and systems worldwide, it is very important to practice basic security awareness. A normal system user can secure himself or herself by following these simp…
The Internet has made sending and receiving information online a breeze. But there is also the threat of unauthorized viewing, data tampering, and phoney messages. Surprisingly, a lot of business owners do not fully understand how to use security t…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…
Video by: Mark
This lesson goes over how to construct ordered and unordered lists and how to create hyperlinks.

834 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question