[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now


is there a robots.txt equivalent for the "no archive" meta tag

Posted on 2006-05-05
Medium Priority
Last Modified: 2012-06-21
I need to stop Google from caching a lot of pages on a website.  Is there a way I can tell google not to archive a whole directory or even the entire website without having to add the following meta tag to each page: <META content="noarchive" NAME="robots">.  I would like to use the robots.txt file for this if possible.
Question by:mrwhipple
  • 2

Expert Comment

ID: 16618355
Sure, just include the following in the robots.txt file in the root of your website

User-agent: *
Disallow: /

Now, this means Googlebot won't index any of the pages in that site, so none of your internal pages will have a pagerank unless links exist to them from outside your website.

Author Comment

ID: 16618380
That would solve my issue but create another.  I still want to allow google to crawl and index the entire site, but I don't want them to be able to cache it.  i.e. there would be no "cached" link next to the search results for the site.

Accepted Solution

smidgie82 earned 150 total points
ID: 16618639
Robots.txt doesn't handle that, it just deals with whether bots are allowed to access particular pages.  Once they access them, what they do with them is totally up to them, as far as the robots.txt file is concerned.

What you want to do, then, is configure your webserver to add the

Cache-control: no-store

header to every response going out.  Unfortunately, I can't find anything specific that says whether webbots actually respect this header.

Featured Post

Vote for the Most Valuable Expert

It’s time to recognize experts that go above and beyond with helpful solutions and engagement on site. Choose from the top experts in the Hall of Fame or on the right rail of your favorite topic page. Look for the blue “Nominate” button on their profile to vote.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When crafting your “Why Us” page, there are a plethora of pitfalls to avoid. Follow these five tips, and you’ll be well on your way to creating an effective page.
Australian government abolished Visa 457 earlier this April and this article describes how this decision might affect Australian IT scene and IT experts.
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…
Video by: Mark
This lesson goes over how to construct ordered and unordered lists and how to create hyperlinks.
Suggested Courses
Course of the Month19 days, 22 hours left to enroll

872 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question