Meta robots or robots.txt

Posted on 2017-10-03
Medium Priority
Last Modified: 2017-10-06
How should you communicate with robots?

Also with access control don't worry about it right guys?
Question by:burnedfaceless
1 Comment
LVL 84

Accepted Solution

Dave Baldwin earned 2000 total points
ID: 42318026
If it requires a login to see the page, then it doesn't matter... but only for that page.  Meta robots applies also to only the page it is on.  http://www.robotstxt.org/meta.html   robots.txt applies to the whole site.   http://www.robotstxt.org/robotstxt.html   But you should always remember that these things are Only suggestions that 'good' robots sometimes follow.  They are no kind of protection at all.

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this article, you will read about the trends across the human resources departments for the upcoming year. Some of them include improving employee experience, adopting new technologies, using HR software to its full extent, and integrating artifi…
This article explains the fundamentals of industrial networking which ultimately is the backbone network which is providing communications for process devices like robots and other not so interesting stuff.
Integration Management Part 2
With just a little bit of  SQL and VBA, many doors open to cool things like synchronize a list box to display data relevant to other information on a form.  If you have never written code or looked at an SQL statement before, no problem! ...  give i…
Suggested Courses
Course of the Month13 days, 11 hours left to enroll

750 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question