Avatar of nsitedesigns
nsitedesigns
Flag for United States of America asked on

Robots.txt file is it needed

I submitted a site to a search engine and got a warning that this site does NOT have a robots.txt file.  It is my understanding that this file tells robots NOT to visit certain pages of your site.  Since there aren't any pages we are "hiding" why would I need this document?
Web DevelopmentHTMLWeb Browsers

Avatar of undefined
Last Comment
nsitedesigns

8/22/2022 - Mon
nsitedesigns

ASKER
Oh, and if I do in fact need one, what do I put in it?
ASKER CERTIFIED SOLUTION
Gary

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
SOLUTION
Log in to continue reading
Log In
Sign up - Free for 7 days
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
SOLUTION
Log in to continue reading
Log In
Sign up - Free for 7 days
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
nsitedesigns

ASKER
Thanks all!
COBOLdinosaur

Even if you do not want to disallow anything, you probably should add it anyway to give the path to your sitemap.  It can also be used to indicate a speed limit for the spiders so they do not request pages so quickly that it causes a performance problem on the site.

So even if you only define Crawl-delay: and Sitemap: it is worth having, and you get brownie points from Google for being co-operative.

Cd&
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
nsitedesigns

ASKER
thank you.