Link to home
Start Free TrialLog in
Avatar of Fernanditos
Fernanditos

asked on

How to block spiders from my site except google & yahoo

Hi,

I want to block all crawlers/spiders who enter into my site and alter my stats and db resources.

I would like only to allow google.

How to do this?

Thank you
Avatar of Lee W, MVP
Lee W, MVP
Flag of United States of America image

Create a robots.txt file - if the search engine obeys it's settings, this should suffice (what, no Bing?)http://www.searchtools.com/robots/robots-txt.html
Avatar of martinnolan
martinnolan

I would suggest using Google Analytics for your website statistics. It’s free, very easy to add to your site and in answer to part of your question, because it uses javascript as the tracking mechanism spiders don’t trigger this and therefore will not show or skew your website statistics.

Is database resource a major issue...?

Robots are good but make sure you get this 100% correct as scripted incorrectly you could block the major search engines - maybe check http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

Avatar of Fernanditos

ASKER

Is this robots.txt syntax correct to allow access only to google, yahoo, msn, adsese and bing?

Do you consider there are others very important?

In practice I only get traffic from google.

Please review the code and tell me if syntax is correct, im afraid not.
User-agent: googlebot
User-agent: Slurp
User-agent: msnbot
User-agent: bingbot
User-agent: AdsBot-google

Disallow:
User-agent: *
Disallow: /

Open in new window

martinnolan, thank you but the question was onother thing. I just want to know how to ALLOW only the mentioned agents amd exclude all others
ASKER CERTIFIED SOLUTION
Avatar of Lee W, MVP
Lee W, MVP
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial