Robots.txt to stop just one link being spidered?

Posted on 2006-11-08
Last Modified: 2010-04-06

Is there a way to wrap some kind of robots.txt code around just one link one page to stop it being spidered by Google et al, or some otherway of doing the same?
I had a client link on a site and it's a really good demo for my software, but he wanted it removed because it showed up in google as being on my site.

<a href="http://www.this-site-i-don'">this-site</a><br>

Question by:macuser777
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
LVL 75

Expert Comment

by:Michel Plungjan
ID: 17898056
In "http://www.this-site-i-don' you can have
User-agent: *
Disallow: /

and in the page that links to it you can have

<meta name="robots" content="noindex,nofollow">

or have this around the link

  <!-- robots content="nofollow" --><a href="http://www.this-site-i-don'"> Do not follow this</a><!-- /robots -->

LVL 75

Accepted Solution

Michel Plungjan earned 100 total points
ID: 17898114
Ahh, google also likes this:

<a href="..." rel="nofollow">Do not follow link</a>

Author Comment

ID: 17899824
<!-- robots content="nofollow" --><a href="http://www.this-site-i-don'"> Do not follow this</a><!-- /robots -->


pardon my ignorance, but i thought <!--  something --> turned the code into a comment and effectively disabled it. does the term 'robots' have an exclusion on this?

<a href="..." rel="nofollow">Do not follow link</a> - looks nice and simple anyhow - thx i'll use that when i get home. but i'd like to understand the <!-- thing -> thing.

LVL 75

Expert Comment

by:Michel Plungjan
ID: 17900029
<!--#include file .... --> will mean something to a server side include enabled server

There are many html comments that are read by something

Author Comment

ID: 17900702
Thank you for that. Good to know.


Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Contact Forms for Website 6 85
Getting attribute Values using xslt 4 61
Problem to ToolkitScriptManager 2 76
Fixed div within Bootstrap carousel item 11 62
This article covers the basics of the Sass, which is a CSS extension language. You will learn about variables, mixins, and nesting.
Have you tried to learn about Unicode, UTF-8, and multibyte text encoding and all the articles are just too "academic" or too technical? This article aims to make the whole topic easy for just about anyone to understand.
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…
HTML5 has deprecated a few of the older ways of showing media as well as offering up a new way to create games and animations. Audio, video, and canvas are just a few of the adjustments made between XHTML and HTML5. As we learned in our last micr…

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question