[Webinar] Streamline your web hosting managementRegister Today


Robots - which type?

Posted on 2008-02-05
Medium Priority
Last Modified: 2010-04-06
Hi guys,

I was wondering if I could get an opinion on which is the best way to control robots that visit your site.
Do you recommend either adding a robots.txt file in the root directory or is it better to include coding in the meta tags of each page?

Thanks in advance
Question by:brookba
1 Comment

Accepted Solution

Oddsen84 earned 2000 total points
ID: 20822549
Even tough all robots are supposed to obey the robots.txt file and the meta tags there realy is no way to fully protect your self against them. Most search robots like google and such will obey but there are rouge ones out there... Mabye a combination of both would be the best.

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Today, the web development industry is booming, and many people consider it to be their vocation. The question you may be asking yourself is – how do I become a web developer?
Q&A with Course Creator, Mark Lassoff, on the importance of HTML5 in the career of a modern-day developer.
HTML5 has deprecated a few of the older ways of showing media as well as offering up a new way to create games and animations. Audio, video, and canvas are just a few of the adjustments made between XHTML and HTML5. As we learned in our last micr…
Video by: Mark
This lesson goes over how to construct ordered and unordered lists and how to create hyperlinks.
Suggested Courses

640 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question