?
Solved

Robots.txt just to allow google and sharepoint crawls

Posted on 2009-07-05
4
Medium Priority
?
2,156 Views
Last Modified: 2012-05-07
I have a robots.txt file in one of my sharepoint sites.
I want it to allow only google and sharepoint itself to crawl this site.

I have a
User-agent  google

at the top and
 
User-agent: *
Disallow: /

at the bottom.
The last part supposedly stops the sharepoint crawl as well as the other
crawlers.
Does anybody know how to allow only
Google and sharepoint in your robots.txt file.

kind regards
0
Comment
Question by:WTFISTHIS
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
4 Comments
 
LVL 11

Expert Comment

by:Andrei Teodorescu
ID: 24793725
first of all, you need to update some registry entries:
http://sharepoint.microsoft.com/blogs/LKuhn/Lists/Posts/Post.aspx?List=29310d0a%2D1eda%2D4834%2Dbb4c%2D06ee575a40c3&ID=49

then us the newly user agent string creted in your robotx.txt file.
0
 

Author Comment

by:WTFISTHIS
ID: 24799026
Thanks for your response.

I just want to know what the user agent for sharepoint 2007 is.
is it "MS Search 5.0 Robot".

Because it looks like Sharepoint is not searching the sites when I
am using my current Robot.txt file (Mentioned in my question).
So all I want to do is allow Sharepoint and google and disallow everything else.

0
 
LVL 11

Accepted Solution

by:
Andrei Teodorescu earned 1000 total points
ID: 24800931
try this way

User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot)
Allow: /
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

For SharePoint sites, particularly public-facing ones, there are times when adding JavaScript, Meta Tags, CSS Styles or other content to the page <head> section is more practical than modifying master pages.  For instance, you could add the jQuery l…
SharePoint Designer 2010 has tools and commands to do everything that can be done with web parts in the browser, and then some – except uploading a web part straight into a page that is edited in SPD. So, can it be done? Scenario For a recent pr…
NetCrunch network monitor is a highly extensive platform for network monitoring and alert generation. In this video you'll see a live demo of NetCrunch with most notable features explained in a walk-through manner. You'll also get to know the philos…
In this video you will find out how to export Office 365 mailboxes using the built in eDiscovery tool. Bear in mind that although this method might be useful in some cases, using PST files as Office 365 backup is troublesome in a long run (more on t…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question