Robot.txt files

Can I make certain folders off limits to search engines with a robot.txt file? And If I can block certain folders and files from being indexed how do I write the code to go in the notepad file?

Thanks
Phil
LVL 1
smphilAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ShaneJonesCommented:
yes you can use the following and replace parts with the folders you need.

use the following link

http://www.searchengineworld.com/robots/robots_tutorial.htm

Shane

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Ashish ThakkarWebsite Designer & SEO ConsultantCommented:
User-agent: *
Disallow: /cgi-bin/
Disallow: /reports/
Disallow: /fulldownloads/

These directories will be disallowed.

You can save this in robots.txt and upload it.

The best thing about robots.txt is you can learn from example and there is no restriction.

Example if you wish to see microsoft's robots file visit http://www.microsoft.com/robots.txt , if you wish to see EE's robots.txt file visit http://www.experts-exchange.com/robots.txt


Regards,
ASHISH T.
Protheus99Commented:
It's great that you ask this question because google sitemaps now has a great robots.txt analysis tool. This tool first appeared a couple weeks ago and is pretty much what you expect from google, something nice that works right. If you sign up to google sitemaps (you don't even need to create a sitemap by the way to see your site's statistics, you only need to place a verification file and your good to go) you will be able to verify any robots.txt files associated with your domain name.

For more information on this visit:
http://sitemaps.blogspot.com/2006/02/more-stats-and-analysis-of-robotstxt.html
http://www.mattcutts.com/blog/new-robotstxt-tool/

With this tool you will be able to know exactly how most search engine spiders will react to your instructions, which is way better then trial and error from the old days.

Hope this helps
ShaneJonesCommented:
I also too spotted that on mattcuts blog, very usefull too I thought.

Worth a look in.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Internet Marketing

From novice to tech pro — start learning today.