I need to stop Google from caching a lot of pages on a website. Is there a way I can tell google not to archive a whole directory or even the entire website without having to add the following meta tag to each page: <META content="noarchive" NAME="robots">. I would like to use the robots.txt file for this if possible.
This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.
Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.
If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.
Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.
Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.
Access the answers to your technology questions today.
30-day free trial. Register in 60 seconds.
Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.
Try it out and discover for yourself.
30-day free trial. Register in 60 seconds.
Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.
Robots.txt doesn't handle that, it just deals with whether bots are allowed to access particular pages. Once they access them, what they do with them is totally up to them, as far as the robots.txt file is concerned.
What you want to do, then, is configure your webserver to add the
Cache-control: no-store
header to every response going out. Unfortunately, I can't find anything specific that says whether webbots actually respect this header.
Business Accounts
Answer for Membership
by: smidgie82Posted on 2006-05-05 at 14:14:52ID: 16618355
Sure, just include the following in the robots.txt file in the root of your website
User-agent: *
Disallow: /
Now, this means Googlebot won't index any of the pages in that site, so none of your internal pages will have a pagerank unless links exist to them from outside your website.