Stop search bots from cataloging site

Posted on 2004-11-14
Last Modified: 2013-12-24
I'm building a few sites in which I do not want search bots to catalog the site Or put another way I would like them to ignore the site. How do I do this?
Question by:markloessi
    LVL 13

    Accepted Solution

    create a robots.txt file in your root directory with the following line:

    User-agent: *
    Disallow: /
    LVL 13

    Expert Comment

    Here is a link to a tutorial explaining all of the features of robots.txt in detail:
    LVL 13

    Expert Comment

    > create a robots.txt file in your root directory with the following line:

    This should read 'lines'.. so two separate lines - one for 'User-agent: *' and one for 'Disallow: /'

    Featured Post

    Find Ransomware Secrets With All-Source Analysis

    Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

    Join & Write a Comment

    Suggested Solutions

    Title # Comments Views Activity
    Problem to master page 11 52
    Web Expression 4 + PHP v5.5 2 52
    Dynamically highly text on a webpage 9 87
    Problem to echo 6 29
    Turn A Profile Picture Into A Cartoon Using Photoshop And Illustrator This tutorial will teach you how to make a cartoon style image out of a regular picture. I have tried to keep the tutorial as simple as possible. I used Adobe CS4 for this tuto…
    Objective of This Article In 1990’s, when I was a budding software professional, I had a lot of confusion about which stream or technology, I had to choose to build my career. In those days, I had lot of confusion like whether to choose System so…
    The purpose of this video is to demonstrate how to manually back up a WordPress Database. This will be demonstrated using a Windows 8 PC. The Host used will be Log into your Hosting account. IPage will be used for demonstration : Locat…
    The purpose of this video is to demonstrate how to Test the speed of a WordPress Website. Site Speed is an important metric of a site’s health. Slow site speed can result in viewers leaving your site quickly and not seeing your content. This…

    728 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    20 Experts available now in Live!

    Get 1:1 Help Now