Solved

robots.txt

Posted on 2004-04-07
2
317 Views
Last Modified: 2010-04-27
Could someone kindly post a sample robots.txt file that would encourage all the spiders?   and then just as a sample, a second file that would exclude one particular page named, say, xyz.html

thanks
0
Comment
Question by:linque
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
2 Comments
 
LVL 24

Accepted Solution

by:
duz earned 500 total points
ID: 10778947
Hi linque -

The following allows all robots to visit all files because the wildcard "*" specifies all robots.

User-agent: *
Disallow:

This one bars all robots from the the images directory.

User-agent: *
Disallow: /images/

This one bars googlebot from getting at the xyz.html file.

User-agent: googlebot
Disallow: xyz.html

- duz
0
 

Author Comment

by:linque
ID: 10779524
THANK YOU DUZ!
0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this blog, I will share you some basic tips for content marketing and to rank your website on Google.
CTAs encourage people to do something specific to show interest in your company, product or service. Keep reading to learn why CTAs should always be thought of as extremely important, albeit small, sections of websites.
Viewers will get an overview of the benefits and risks of using Bitcoin to accept payments. What Bitcoin is: Legality: Risks: Benefits: Which businesses are best suited?: Other things you should know: How to get started:
An overview of how to create reports in Adobe Analytics (formerly Omniture Site Catalyst) using pageNames, events, eVars and props. This video will show you how to install the Omniture Debugger tool so can see (and test) what is being passed int…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question