Solved

robots.txt

Posted on 2004-04-07
2
295 Views
Last Modified: 2010-04-27
Could someone kindly post a sample robots.txt file that would encourage all the spiders?   and then just as a sample, a second file that would exclude one particular page named, say, xyz.html

thanks
0
Comment
Question by:linque
2 Comments
 
LVL 24

Accepted Solution

by:
duz earned 500 total points
ID: 10778947
Hi linque -

The following allows all robots to visit all files because the wildcard "*" specifies all robots.

User-agent: *
Disallow:

This one bars all robots from the the images directory.

User-agent: *
Disallow: /images/

This one bars googlebot from getting at the xyz.html file.

User-agent: googlebot
Disallow: xyz.html

- duz
0
 

Author Comment

by:linque
ID: 10779524
THANK YOU DUZ!
0

Featured Post

3 Use Cases for Connected Systems

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, testing some more, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Owning a franchise can be the dream of a lifetime. It provides a chance for economic growth. You can be as successful as you want.  To make your franchise successful, you need to market it successfully. Here are six of the best marketing strategies …
For many of us, the  holiday season kindles the natural urge to give back to our friends, family members and communities. While it's easy for friends to notice the impact of such deeds, understanding the contributions of businesses and enterprises i…
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to choose which pages of your form are visible to your users based on their inputs. The page rules feature provides you with an opportunity to create if:then statements for y…
Learn how to set-up custom confirmation messages to users who complete your Wufoo form. Include inputs from fields in your form, webpage redirects, and more with Wufoo’s confirmation options.

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question