?
Solved

Robots.txt to stop just one link being spidered?

Posted on 2006-11-08
5
Medium Priority
?
174 Views
Last Modified: 2010-04-06
Hi,

Is there a way to wrap some kind of robots.txt code around just one link one page to stop it being spidered by Google et al, or some otherway of doing the same?
I had a client link on a site and it's a really good demo for my software, but he wanted it removed because it showed up in google as being on my site.

<a href="http://www.this-site-i-don't-want-spidered.co.uk">this-site</a><br>

thanks
0
Comment
Question by:macuser777
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
5 Comments
 
LVL 75

Expert Comment

by:Michel Plungjan
ID: 17898056
In "http://www.this-site-i-don't-want-spidered.co.uk/robots.txt you can have
User-agent: *
Disallow: /

and in the page that links to it you can have

<meta name="robots" content="noindex,nofollow">

or have this around the link

  <!-- robots content="nofollow" --><a href="http://www.this-site-i-don't-want-spidered.co.uk/"> Do not follow this</a><!-- /robots -->


Michel
0
 
LVL 75

Accepted Solution

by:
Michel Plungjan earned 400 total points
ID: 17898114
Ahh, google also likes this:

<a href="..." rel="nofollow">Do not follow link</a>
0
 
LVL 4

Author Comment

by:macuser777
ID: 17899824
<!-- robots content="nofollow" --><a href="http://www.this-site-i-don't-want-spidered.co.uk/"> Do not follow this</a><!-- /robots -->

hi,

pardon my ignorance, but i thought <!--  something --> turned the code into a comment and effectively disabled it. does the term 'robots' have an exclusion on this?

<a href="..." rel="nofollow">Do not follow link</a> - looks nice and simple anyhow - thx i'll use that when i get home. but i'd like to understand the <!-- thing -> thing.

macuser
0
 
LVL 75

Expert Comment

by:Michel Plungjan
ID: 17900029
<!--#include file .... --> will mean something to a server side include enabled server

There are many html comments that are read by something
0
 
LVL 4

Author Comment

by:macuser777
ID: 17900702
Thank you for that. Good to know.

macuser
0

Featured Post

Get real performance insights from real users

Key features:
- Total Pages Views and Load times
- Top Pages Viewed and Load Times
- Real Time Site Page Build Performance
- Users’ Browser and Platform Performance
- Geographic User Breakdown
- And more

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I will show you how to create a ASP.NET Captcha control without using any HTTP HANDELRS or what so ever. you can easily plug it into your web pages. For Example a = 2 + 3 (where 2 and 3 are 2 random numbers) Session("Answer") = 5 then we…
SASS allows you to treat your CSS code in a more OOP way. Let's have a look on how you can structure your code in order for it to be easily maintained and reused.
Viewers will learn about basic arrays, how to declare them, and how to use them. Introduction and definition: Declare an array and cover the syntax of declaring them: Initialize every index in the created array: Example/Features of a basic arr…
Video by: Mark
This lesson goes over how to construct ordered and unordered lists and how to create hyperlinks.
Suggested Courses

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question