Solved

Robots.txt to stop just one link being spidered?

Posted on 2006-11-08
5
173 Views
Last Modified: 2010-04-06
Hi,

Is there a way to wrap some kind of robots.txt code around just one link one page to stop it being spidered by Google et al, or some otherway of doing the same?
I had a client link on a site and it's a really good demo for my software, but he wanted it removed because it showed up in google as being on my site.

<a href="http://www.this-site-i-don't-want-spidered.co.uk">this-site</a><br>

thanks
0
Comment
Question by:macuser777
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
5 Comments
 
LVL 75

Expert Comment

by:Michel Plungjan
ID: 17898056
In "http://www.this-site-i-don't-want-spidered.co.uk/robots.txt you can have
User-agent: *
Disallow: /

and in the page that links to it you can have

<meta name="robots" content="noindex,nofollow">

or have this around the link

  <!-- robots content="nofollow" --><a href="http://www.this-site-i-don't-want-spidered.co.uk/"> Do not follow this</a><!-- /robots -->


Michel
0
 
LVL 75

Accepted Solution

by:
Michel Plungjan earned 100 total points
ID: 17898114
Ahh, google also likes this:

<a href="..." rel="nofollow">Do not follow link</a>
0
 
LVL 4

Author Comment

by:macuser777
ID: 17899824
<!-- robots content="nofollow" --><a href="http://www.this-site-i-don't-want-spidered.co.uk/"> Do not follow this</a><!-- /robots -->

hi,

pardon my ignorance, but i thought <!--  something --> turned the code into a comment and effectively disabled it. does the term 'robots' have an exclusion on this?

<a href="..." rel="nofollow">Do not follow link</a> - looks nice and simple anyhow - thx i'll use that when i get home. but i'd like to understand the <!-- thing -> thing.

macuser
0
 
LVL 75

Expert Comment

by:Michel Plungjan
ID: 17900029
<!--#include file .... --> will mean something to a server side include enabled server

There are many html comments that are read by something
0
 
LVL 4

Author Comment

by:macuser777
ID: 17900702
Thank you for that. Good to know.

macuser
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Preface This article introduces an authentication and authorization system for a website.  It is understood by the author and the project contributors that there is no such thing as a "one size fits all" system.  That being said, there is a certa…
Introduction Knockoutjs (Knockout) is a JavaScript framework (Model View ViewModel or MVVM framework).   The main ideology behind Knockout is to control from JavaScript how a page looks whilst creating an engaging user experience in the least …
The viewer will the learn the benefit of plain text editors and code an HTML5 based template for use in further tutorials.
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

688 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question