Link to home
Start Free TrialLog in
Avatar of nsitedesigns
nsitedesignsFlag for United States of America

asked on

Robots.txt file is it needed

I submitted a site to a search engine and got a warning that this site does NOT have a robots.txt file.  It is my understanding that this file tells robots NOT to visit certain pages of your site.  Since there aren't any pages we are "hiding" why would I need this document?
Avatar of nsitedesigns
nsitedesigns
Flag of United States of America image

ASKER

Oh, and if I do in fact need one, what do I put in it?
ASKER CERTIFIED SOLUTION
Avatar of Gary
Gary
Flag of Ireland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Thanks all!
Even if you do not want to disallow anything, you probably should add it anyway to give the path to your sitemap.  It can also be used to indicate a speed limit for the spiders so they do not request pages so quickly that it causes a performance problem on the site.

So even if you only define Crawl-delay: and Sitemap: it is worth having, and you get brownie points from Google for being co-operative.

Cd&
thank you.