Solved

How to block spiders from my site except google & yahoo

Posted on 2010-09-05
5
384 Views
Last Modified: 2013-12-09
Hi,

I want to block all crawlers/spiders who enter into my site and alter my stats and db resources.

I would like only to allow google.

How to do this?

Thank you
0
Comment
Question by:Fernanditos
  • 2
  • 2
5 Comments
 
LVL 95

Expert Comment

by:Lee W, MVP
ID: 33605979
Create a robots.txt file - if the search engine obeys it's settings, this should suffice (what, no Bing?)http://www.searchtools.com/robots/robots-txt.html
0
 
LVL 1

Expert Comment

by:martinnolan
ID: 33606033
I would suggest using Google Analytics for your website statistics. It’s free, very easy to add to your site and in answer to part of your question, because it uses javascript as the tracking mechanism spiders don’t trigger this and therefore will not show or skew your website statistics.

Is database resource a major issue...?

Robots are good but make sure you get this 100% correct as scripted incorrectly you could block the major search engines - maybe check http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

0
 

Author Comment

by:Fernanditos
ID: 33606079
Is this robots.txt syntax correct to allow access only to google, yahoo, msn, adsese and bing?

Do you consider there are others very important?

In practice I only get traffic from google.

Please review the code and tell me if syntax is correct, im afraid not.
User-agent: googlebot
User-agent: Slurp
User-agent: msnbot
User-agent: bingbot
User-agent: AdsBot-google

Disallow:
User-agent: *
Disallow: /

Open in new window

0
 

Author Comment

by:Fernanditos
ID: 33606113
martinnolan, thank you but the question was onother thing. I just want to know how to ALLOW only the mentioned agents amd exclude all others
0
 
LVL 95

Accepted Solution

by:
Lee W, MVP earned 500 total points
ID: 33607567
I've never configured anything special in a robots.txt  - I would suggest you try this validator and see what it says:
http://tool.motoricerca.info/robots-checker.phtml
0

Featured Post

MIM Survival Guide for Service Desk Managers

Major incidents can send mastered service desk processes into disorder. Systems and tools produce the data needed to resolve these incidents, but your challenge is getting that information to the right people fast. Check out the Survival Guide and begin bringing order to chaos.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Developer portfolios can be a bit of an enigma—how do you present yourself to employers without burying them in lines of code?  A modern portfolio is more than just work samples, it’s also a statement of how you work.
If you don’t want your company's site to fail on the web, you’d do well to observe these best web design practices and make sure you implement them when applicable.
This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details. Login to your Google Account, then search for "Google Mapmaker…
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…

829 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question