is there a robots.txt equivalent for the "no archive" meta tag

Posted on 2006-05-05
Last Modified: 2012-06-21
I need to stop Google from caching a lot of pages on a website.  Is there a way I can tell google not to archive a whole directory or even the entire website without having to add the following meta tag to each page: <META content="noarchive" NAME="robots">.  I would like to use the robots.txt file for this if possible.
Question by:mrwhipple
    LVL 9

    Expert Comment

    Sure, just include the following in the robots.txt file in the root of your website

    User-agent: *
    Disallow: /

    Now, this means Googlebot won't index any of the pages in that site, so none of your internal pages will have a pagerank unless links exist to them from outside your website.
    LVL 1

    Author Comment

    That would solve my issue but create another.  I still want to allow google to crawl and index the entire site, but I don't want them to be able to cache it.  i.e. there would be no "cached" link next to the search results for the site.
    LVL 9

    Accepted Solution

    Robots.txt doesn't handle that, it just deals with whether bots are allowed to access particular pages.  Once they access them, what they do with them is totally up to them, as far as the robots.txt file is concerned.

    What you want to do, then, is configure your webserver to add the

    Cache-control: no-store

    header to every response going out.  Unfortunately, I can't find anything specific that says whether webbots actually respect this header.

    Featured Post

    Free Trending Threat Insights Every Day

    Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

    Join & Write a Comment

    When pages do not download correctly, and you don't know why, the first thing you do is to look at the HTML source code of that page, but not all the downloaded files appear always clearly. If your source includes a javascript that computes the name…
    Why do we like using grid based layouts in website design? Let's look at the live examples of websites and compare them to grid based WordPress themes.
    This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details. Login to your Google Account, then search for "Google Mapmaker…
    The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.

    734 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    19 Experts available now in Live!

    Get 1:1 Help Now