• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 762
  • Last Modified:

view a hidden robots.txt file

When I go to http://www.shutterstock.com/robots.txt

the robots.txt file is hidden

how can a person hide a robots.txt file

how can a person read a hidden robots.txt file
0
jerseyguy0
Asked:
jerseyguy0
2 Solutions
 
thehagmanCommented:
Sure it's hidden and not just empty?
0
 
jerseyguy0Author Commented:
its hidden
0
 
RobinSoftware EngineerCommented:
How do you know it really is a robots.txt ?
Hiding it makes its existence useless.

Maybe it a script that only shows the contect on certain browser version, search engine, etc.
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
thehagmanCommented:
Of course they might evaluate the user agent string and send the full robots.txt only if the agent is among the list of known-to-thems.
This will cause any new robot to see an empty robots.txt and hence it will not preventd from indexing even if it obeys the robots protocol.
0
 
jazzIIIloveCommented:
Hi here;

how can a person hide a robots.txt file:

A very nice tutorial on this...

http://forums.ukwebmasterworld.com/programming/7256-hide-your-robots-txt-visitors-show-only-validated-robots.html

Best regards...
0
 
gwkgCommented:
Here is another article, with a more advanced script with more user agents

http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/


0

Featured Post

Cyber Threats to Small Businesses (Part 1)

This past May, Webroot surveyed more than 600 IT decision-makers at medium-sized companies to see how these small businesses perceived new threats facing their organizations.  Read what Webroot CISO, Gary Hayslip, has to say about the survey in part 1 of this 2-part blog series.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now