robots.txt is allow used by google


I see keywords

is 'allow' used by google
Who is Participating?
netcmhConnect With a Mentor Commented:
According to "If you want all of Google to be able to crawl your pages, you don't need a robots.txt file at all."


below is an excerpt from the above link:

Where several user-agents are recognized in the robots.txt file, Google will follow the most specific. If you want all of Google to be able to crawl your pages, you don't need a robots.txt file at all. If you want to block or allow all of Google's crawlers from accessing some of your content, you can do this by specifying Googlebot as the user-agent. For example, if you want all your pages to appear in Google search, and if you want AdSense ads to appear on your pages, you don't need a robots.txt file. Similarly, if you want to block some pages from Google altogether, blocking the user-agent Googlebot will also block all Google's other user-agents.
But if you want more fine-grained control, you can get more specific. For example, you might want all your pages to appear in Google Search, but you don't want images in your personal directory to be crawled. In this case, use robots.txt to disallow the user-agent Googlebot-image from crawling the files in your /personal directory (while allowing Googlebot to crawl all files), like this:
 User-agent: Googlebot

User-agent: Googlebot-Image
Disallow: /personal
 To take another example, say that you want ads on all your pages, but you don't want those pages to appear in Google Search. Here, you'd block Googlebot, but allow Mediapartners-Google, like this: User-agent: Googlebot
Disallow: /

User-agent: Mediapartners-Google
Ray PaseurConnect With a Mentor Commented:
Google and every other "legit" search engine will obey robots.txt.  The Chinese government, Bulgarian hackers, Nigerian fraudsters and the US NSA will not.  Hope that helps guide your planning about what you choose to publish on the internet.
rgb192Author Commented:
thanks for robots.txt description and who follows
Thanks for the grade. Good luck.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.