We have a site as site as http://www.xxx.com and also have the secured URL https://secure.xxx.com which serves the same contents of http://www.xxx.com. We already have a robots.txt file in the root directory http://www.xxx.com which also serves for https://secure.xxx.com. What we need is to block Google(and other se's) from crawling https versions. So we are planning to create a new robots file with code for not to crawl https versions.
We got a chance to see this link, http://www.kleenecode.net/2007/11/17/dynamic-robotstxt-with-aspnet-20/
, it is very helpful because they are telling we can create different robots.txt file for http and https. And also you can see a code for Robots.txt to prevent from crawling to https versions. So to implement this concept, we have done the following steps as the instructions in the post.
1) Create a Robots.txt with the given code in the root of our web project.
After that for checking, i browse robots.txt file, then i can see the full code as i typed.
2)To get the path to the ASPX engine,
a) Open IIS and right click on our website and bring up the properties screen
b) Go to Home Directory > Configuration. then got Mappings Tab
c) Locate the ASPX item and click Edit - Copy the path in the Executable Field and cancel out of that window
3) Create the ISAPI entry for .txt .
a) Still on "Mappings Tab"
b) Click "Add"
c) Populate the Executable path with the value I copied in the last section
d) Enter GET in the Limit To field
e) Enter ".txt" in the "Extension" field(they are not specifying this one)
f) Press "OK" to save all my changes.
Next for checking the result , they are telling when we browse robots.txt file , will get a blank page. But we are not getting blank page, we got a page with the full code as i typed.
Could anyone help me to resolve this? If anyone t-ry this dynamic creation of robots.txt, just share their experience also.
Thanks in advance for any help