robots.txt is allow used by google

robots.txt

I see keywords
'user-agent'
'allow'
'disallow'

is 'allow' used by google
LVL 1
rgb192Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

netcmhCommented:
According to google.com "If you want all of Google to be able to crawl your pages, you don't need a robots.txt file at all."

See: https://support.google.com/webmasters/answer/1061943?hl=en

below is an excerpt from the above link:

Where several user-agents are recognized in the robots.txt file, Google will follow the most specific. If you want all of Google to be able to crawl your pages, you don't need a robots.txt file at all. If you want to block or allow all of Google's crawlers from accessing some of your content, you can do this by specifying Googlebot as the user-agent. For example, if you want all your pages to appear in Google search, and if you want AdSense ads to appear on your pages, you don't need a robots.txt file. Similarly, if you want to block some pages from Google altogether, blocking the user-agent Googlebot will also block all Google's other user-agents.
 
But if you want more fine-grained control, you can get more specific. For example, you might want all your pages to appear in Google Search, but you don't want images in your personal directory to be crawled. In this case, use robots.txt to disallow the user-agent Googlebot-image from crawling the files in your /personal directory (while allowing Googlebot to crawl all files), like this:
 User-agent: Googlebot
Disallow:

User-agent: Googlebot-Image
Disallow: /personal
 To take another example, say that you want ads on all your pages, but you don't want those pages to appear in Google Search. Here, you'd block Googlebot, but allow Mediapartners-Google, like this: User-agent: Googlebot
Disallow: /

User-agent: Mediapartners-Google
Disallow:
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Ray PaseurCommented:
Google and every other "legit" search engine will obey robots.txt.  The Chinese government, Bulgarian hackers, Nigerian fraudsters and the US NSA will not.  Hope that helps guide your planning about what you choose to publish on the internet.
0
rgb192Author Commented:
thanks for robots.txt description and who follows
0
netcmhCommented:
Thanks for the grade. Good luck.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
PHP

From novice to tech pro — start learning today.