We help IT Professionals succeed at work.

Robots.txt file is it needed

438 Views
Last Modified: 2014-05-05
I submitted a site to a search engine and got a warning that this site does NOT have a robots.txt file.  It is my understanding that this file tells robots NOT to visit certain pages of your site.  Since there aren't any pages we are "hiding" why would I need this document?
Comment
Watch Question

Author

Commented:
Oh, and if I do in fact need one, what do I put in it?
CERTIFIED EXPERT
Expert of the Year 2014
Top Expert 2014
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION
Dave BaldwinFixer of Problems
CERTIFIED EXPERT
Most Valuable Expert 2014
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION
CERTIFIED EXPERT
Expert of the Year 2014
Top Expert 2014
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
Thanks all!
Top Expert 2013

Commented:
Even if you do not want to disallow anything, you probably should add it anyway to give the path to your sitemap.  It can also be used to indicate a speed limit for the spiders so they do not request pages so quickly that it causes a performance problem on the site.

So even if you only define Crawl-delay: and Sitemap: it is worth having, and you get brownie points from Google for being co-operative.

Cd&

Author

Commented:
thank you.

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.