Need to block BOTS for website

Posted on 2011-05-03
Last Modified: 2012-05-11
One site get a lot of bandwith because to many bots reach it everyday.
I'm looking for other way to stop this for now.
I edit the HTACCESS file to deny some ip address and some country.
The site will be completely removed and recreated because it's use old unsafe code.
I'm wondering if the bots will still come after even if the site is recreate?

Any advice is welcome

Question by:lenamtl
    LVL 35

    Accepted Solution

    If these are malevolent bots, you will be left with editing the .htaccess for each one of them, but if they are regular legit bots and crawlers you could easily help yourself with a robots.txt file:
    LVL 34

    Expert Comment

    As torimar says, you can use a robots.txt file to influence (not control, but influence) "legitimate" bots and crawlers.

    There is absolutely nothing that forces any bot/crawler to obey a robots.txt file. Think of it as asking someone to be polite. They can still be a jerk.

    Bots can come back even after a site is re-written. Chances are they will.

    If you want to block specific IP sub-nets/countries, I'd do it at the firewall. If you don't want those IPs scanning your website, why allow them to waste your bandwidth (making all those HTTP requests) and CPU (parsing and utilizing .htaccess)? Drop the packets at the firewall and sooner or later they'll get the hint - in the mean time, you minimize the negative impact on your systems.

    Featured Post

    How to run any project with ease

    Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
    - Combine task lists, docs, spreadsheets, and chat in one
    - View and edit from mobile/offline
    - Cut down on emails

    Join & Write a Comment

    I thought I'd write this up for anyone who has a request to create an anonymous whistle-blower-type submission form created using SharePoint 2010 (this would probably work the same for 2013). It's not 100% fool-proof but it's as close as you can get…
    If you're not part of the solution, you're part of the problem.   Tips on how to secure IoT devices, even the dumbest ones, so they can't be used as part of a DDoS botnet.  Use PRTG Network Monitor as one of the building blocks, to detect unusual…
    The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.
    The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

    729 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    18 Experts available now in Live!

    Get 1:1 Help Now