Meta robots or robots.txt

burnedfaceless
burnedfaceless used Ask the Experts™
on
How should you communicate with robots?

Also with access control don't worry about it right guys?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Fixer of Problems
Most Valuable Expert 2014
Commented:
If it requires a login to see the page, then it doesn't matter... but only for that page.  Meta robots applies also to only the page it is on.  http://www.robotstxt.org/meta.html   robots.txt applies to the whole site.   http://www.robotstxt.org/robotstxt.html   But you should always remember that these things are Only suggestions that 'good' robots sometimes follow.  They are no kind of protection at all.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial