Solved

How to block spiders from my site except google & yahoo

Posted on 2010-09-05
5
382 Views
Last Modified: 2013-12-09
Hi,

I want to block all crawlers/spiders who enter into my site and alter my stats and db resources.

I would like only to allow google.

How to do this?

Thank you
0
Comment
Question by:Fernanditos
  • 2
  • 2
5 Comments
 
LVL 95

Expert Comment

by:Lee W, MVP
ID: 33605979
Create a robots.txt file - if the search engine obeys it's settings, this should suffice (what, no Bing?)http://www.searchtools.com/robots/robots-txt.html
0
 
LVL 1

Expert Comment

by:martinnolan
ID: 33606033
I would suggest using Google Analytics for your website statistics. It’s free, very easy to add to your site and in answer to part of your question, because it uses javascript as the tracking mechanism spiders don’t trigger this and therefore will not show or skew your website statistics.

Is database resource a major issue...?

Robots are good but make sure you get this 100% correct as scripted incorrectly you could block the major search engines - maybe check http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

0
 

Author Comment

by:Fernanditos
ID: 33606079
Is this robots.txt syntax correct to allow access only to google, yahoo, msn, adsese and bing?

Do you consider there are others very important?

In practice I only get traffic from google.

Please review the code and tell me if syntax is correct, im afraid not.
User-agent: googlebot
User-agent: Slurp
User-agent: msnbot
User-agent: bingbot
User-agent: AdsBot-google

Disallow:
User-agent: *
Disallow: /

Open in new window

0
 

Author Comment

by:Fernanditos
ID: 33606113
martinnolan, thank you but the question was onother thing. I just want to know how to ALLOW only the mentioned agents amd exclude all others
0
 
LVL 95

Accepted Solution

by:
Lee W, MVP earned 500 total points
ID: 33607567
I've never configured anything special in a robots.txt  - I would suggest you try this validator and see what it says:
http://tool.motoricerca.info/robots-checker.phtml
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Foreword (July, 2015) Since I first wrote this article, years ago, a great many more people have begun using the internet.  They are coming online from every part of the globe, learning, reading, shopping and spending money at an ever-increasing ra…
Using SQL Scripts we can save all the SQL queries as files that we use very frequently on our database later point of time. This is one of the feature present under SQL Workshop in Oracle Application Express.
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now