Robots.txt just to allow google and sharepoint crawls

I have a robots.txt file in one of my sharepoint sites.
I want it to allow only google and sharepoint itself to crawl this site.

I have a
User-agent  google

at the top and
 
User-agent: *
Disallow: /

at the bottom.
The last part supposedly stops the sharepoint crawl as well as the other
crawlers.
Does anybody know how to allow only
Google and sharepoint in your robots.txt file.

kind regards
WTFISTHISAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrei TeodorescuBusiness OwnerCommented:
first of all, you need to update some registry entries:
http://sharepoint.microsoft.com/blogs/LKuhn/Lists/Posts/Post.aspx?List=29310d0a%2D1eda%2D4834%2Dbb4c%2D06ee575a40c3&ID=49

then us the newly user agent string creted in your robotx.txt file.
0
WTFISTHISAuthor Commented:
Thanks for your response.

I just want to know what the user agent for sharepoint 2007 is.
is it "MS Search 5.0 Robot".

Because it looks like Sharepoint is not searching the sites when I
am using my current Robot.txt file (Mentioned in my question).
So all I want to do is allow Sharepoint and google and disallow everything else.

0
Andrei TeodorescuBusiness OwnerCommented:
try this way

User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot)
Allow: /
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SharePoint

From novice to tech pro — start learning today.