Solved

Is it possible to post a file on a website in a non-protected folder that bots can't get to?

Posted on 2009-05-20
6
216 Views
Last Modified: 2013-12-09
I have a number of PDFs in a password protected folder.  In that folder I have an HTML file with links to the files.  Unfortunately some people cannot access them due to filtering software on their end that gives the following error message:

   You are explicitly denied access to this website "http://www.thedomain.com/Presentations/Index.aspx" per local base policy and AFI-33-129

   Your request was denied because of its content categorization: "none"

I can put together a password protected page easily enough that only gives them the links to the files if they enter a correct password, but the files themselves would then be in a non-protected folder.  Will search engine bots be able to find those files without the password since they are just sitting in the folder?  

When I put in a URL for a folder on my web server with no default page I get the error
   Directory Listing Denied
   This Virtual Directory does not allow contents to be listed.

This leads me to believe that they would be safe, but I don't want to find out afterward they aren't as the data is sensitive.

Thanks.
0
Comment
Question by:dtburdick
6 Comments
 
LVL 8

Expert Comment

by:Pearl_export_ben
Comment Utility
stick this into a file called robots.txt at the root of your drive:

User-agent: *
Disallow: /presentations/
0
 
LVL 8

Expert Comment

by:Pearl_export_ben
Comment Utility
*by drive i mean website root
0
 

Author Comment

by:dtburdick
Comment Utility
Thanks.  Unfortunately I can't use that because it seems that is only a request, not a requirement, so I can't put the files there as they need to be truly secure.

This is the page that discusses it.

http://www.robotstxt.org/orig.html

The second paragraph says "It is not an official standard backed by a standards body, or owned by any commercial organisation. It is not enforced by anybody, and there no guarantee that all current and future robots will use it. Consider it a common facility the majority of robot authors offer the WWW community to protect WWW server against unwanted accesses by their robots."

0
What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

 
LVL 9

Accepted Solution

by:
djpazza earned 500 total points
Comment Utility
I think the ultimate way is to stream the file so it does not actually existing until it is request then a script will put it together.

http://www.builderau.com.au/webdev/soa/Protect-your-site-against-Web-scrapes/0,339024680,339129927,00.htm

2) Can you not give the users the password they need to access the folder.
3) You could setup a folder for ftp access with a password on.  So users would goto e.g
ftp.thedomain.com/Presentations/
4) You could put the files into a password protect zip file.

5) The easiest way might be an online storage facility.

Box.net allows you to setup shared folders, invite users to get notifications of new files etc.
0
 
LVL 23

Expert Comment

by:Tiggerito
Comment Utility
Maybe ASP.Net forms authentication would work for you:

http://msdn.microsoft.com/en-us/library/aa480476.aspx

You can control access to files and folders based on the user that is logged in. All at a server level.
0
 

Author Closing Comment

by:dtburdick
Comment Utility
Box.net is the perfect solution for me.  Especially the notification piece.

However, for anyone else reading this thread I already figured out a solution.  My files were in a password protected folder, accessed from an ASPX file sitting in an iframe in a COTS app with a crazy URL (that they wanted to keep).  I sent them a direct link to the ASPX file taking the URL and iframe out of the picture and it worked.  

I'm not sure if it was the combination of craziness or one of those two that caused the problem, but it did resolve it.  Even still, I'll probably end up going with box.net as it is a more elegant solution.
0

Featured Post

Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

Join & Write a Comment

Author Note: Since this E-E article was originally written, years ago, formal testing has come into common use in the world of PHP.  PHPUnit (http://en.wikipedia.org/wiki/PHPUnit) and similar technologies have enjoyed wide adoption, making it possib…
"In order to have an organized way for empathy mapping, we rely on a psychological model and trying to model it in a simple way, so we will split the board to three section for each persona and a scenario and try to see what those personas would Do,…
This video teaches users how to migrate an existing Wordpress website to a new domain.
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now