Solved

How to block spiders from my site except google & yahoo

Posted on 2010-09-05
5
385 Views
Last Modified: 2013-12-09
Hi,

I want to block all crawlers/spiders who enter into my site and alter my stats and db resources.

I would like only to allow google.

How to do this?

Thank you
0
Comment
Question by:Fernanditos
  • 2
  • 2
5 Comments
 
LVL 95

Expert Comment

by:Lee W, MVP
ID: 33605979
Create a robots.txt file - if the search engine obeys it's settings, this should suffice (what, no Bing?)http://www.searchtools.com/robots/robots-txt.html
0
 
LVL 1

Expert Comment

by:martinnolan
ID: 33606033
I would suggest using Google Analytics for your website statistics. It’s free, very easy to add to your site and in answer to part of your question, because it uses javascript as the tracking mechanism spiders don’t trigger this and therefore will not show or skew your website statistics.

Is database resource a major issue...?

Robots are good but make sure you get this 100% correct as scripted incorrectly you could block the major search engines - maybe check http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

0
 

Author Comment

by:Fernanditos
ID: 33606079
Is this robots.txt syntax correct to allow access only to google, yahoo, msn, adsese and bing?

Do you consider there are others very important?

In practice I only get traffic from google.

Please review the code and tell me if syntax is correct, im afraid not.
User-agent: googlebot
User-agent: Slurp
User-agent: msnbot
User-agent: bingbot
User-agent: AdsBot-google

Disallow:
User-agent: *
Disallow: /

Open in new window

0
 

Author Comment

by:Fernanditos
ID: 33606113
martinnolan, thank you but the question was onother thing. I just want to know how to ALLOW only the mentioned agents amd exclude all others
0
 
LVL 95

Accepted Solution

by:
Lee W, MVP earned 500 total points
ID: 33607567
I've never configured anything special in a robots.txt  - I would suggest you try this validator and see what it says:
http://tool.motoricerca.info/robots-checker.phtml
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If you are a web developer, you would be aware of the <iframe> tag in HTML. The <iframe> stands for inline frame and is used to embed another document within the current HTML document. The embedded document could be even another website.
Today it’s fairly well known that high-performing websites and applications bring in more visitors, higher SEO, and ultimately more sales. By the same token, downtime is disastrous for companies and can lead to major hits on a brand, reputation, an…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
The viewer will get a basic understanding of what section 508 compliance can entail, learn about skip navigation links, alt text, transcripts, and font size controls.

730 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question