Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Robots.txt to stop just one link being spidered?

Posted on 2006-11-08
5
Medium Priority
?
177 Views
Last Modified: 2010-04-06
Hi,

Is there a way to wrap some kind of robots.txt code around just one link one page to stop it being spidered by Google et al, or some otherway of doing the same?
I had a client link on a site and it's a really good demo for my software, but he wanted it removed because it showed up in google as being on my site.

<a href="http://www.this-site-i-don't-want-spidered.co.uk">this-site</a><br>

thanks
0
Comment
Question by:macuser777
  • 3
  • 2
5 Comments
 
LVL 75

Expert Comment

by:Michel Plungjan
ID: 17898056
In "http://www.this-site-i-don't-want-spidered.co.uk/robots.txt you can have
User-agent: *
Disallow: /

and in the page that links to it you can have

<meta name="robots" content="noindex,nofollow">

or have this around the link

  <!-- robots content="nofollow" --><a href="http://www.this-site-i-don't-want-spidered.co.uk/"> Do not follow this</a><!-- /robots -->


Michel
0
 
LVL 75

Accepted Solution

by:
Michel Plungjan earned 400 total points
ID: 17898114
Ahh, google also likes this:

<a href="..." rel="nofollow">Do not follow link</a>
0
 
LVL 4

Author Comment

by:macuser777
ID: 17899824
<!-- robots content="nofollow" --><a href="http://www.this-site-i-don't-want-spidered.co.uk/"> Do not follow this</a><!-- /robots -->

hi,

pardon my ignorance, but i thought <!--  something --> turned the code into a comment and effectively disabled it. does the term 'robots' have an exclusion on this?

<a href="..." rel="nofollow">Do not follow link</a> - looks nice and simple anyhow - thx i'll use that when i get home. but i'd like to understand the <!-- thing -> thing.

macuser
0
 
LVL 75

Expert Comment

by:Michel Plungjan
ID: 17900029
<!--#include file .... --> will mean something to a server side include enabled server

There are many html comments that are read by something
0
 
LVL 4

Author Comment

by:macuser777
ID: 17900702
Thank you for that. Good to know.

macuser
0

Featured Post

Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

It's sometimes a bit tricky to use date functions in Oracle BPEL. I'll explain quickly how you can add N days to the current date. In a BPEL process this can be useful, and you can adapt it to fit your needs. First of all, let's see how to add 1 …
Shoutout to Emily Plummer (http://www.experts-exchange.com/members/eplummer26.html) for giving me this article! She did most of it, I just finished it up and posted it for her :)    Introduction In a previous article (http://www.experts-exchang…
The viewer will learn the benefit of using external CSS files and the relationship between class and ID selectors. Create your external css file by saving it as style.css then set up your style tags: (CODE) Reference the nav tag and set your prop…
The viewer will learn the basics of jQuery including how to code hide show and toggles. Reference your jQuery libraries: (CODE) Include your new external js/jQuery file: (CODE) Write your first lines of code to setup your site for jQuery…
Suggested Courses

810 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question