Solved

Is it possible to post a file on a website in a non-protected folder that bots can't get to?

Posted on 2009-05-20
6
228 Views
Last Modified: 2013-12-09
I have a number of PDFs in a password protected folder.  In that folder I have an HTML file with links to the files.  Unfortunately some people cannot access them due to filtering software on their end that gives the following error message:

   You are explicitly denied access to this website "http://www.thedomain.com/Presentations/Index.aspx" per local base policy and AFI-33-129

   Your request was denied because of its content categorization: "none"

I can put together a password protected page easily enough that only gives them the links to the files if they enter a correct password, but the files themselves would then be in a non-protected folder.  Will search engine bots be able to find those files without the password since they are just sitting in the folder?  

When I put in a URL for a folder on my web server with no default page I get the error
   Directory Listing Denied
   This Virtual Directory does not allow contents to be listed.

This leads me to believe that they would be safe, but I don't want to find out afterward they aren't as the data is sensitive.

Thanks.
0
Comment
Question by:dtburdick
6 Comments
 
LVL 8

Expert Comment

by:Pearl_export_ben
ID: 24432494
stick this into a file called robots.txt at the root of your drive:

User-agent: *
Disallow: /presentations/
0
 
LVL 8

Expert Comment

by:Pearl_export_ben
ID: 24432503
*by drive i mean website root
0
 

Author Comment

by:dtburdick
ID: 24433537
Thanks.  Unfortunately I can't use that because it seems that is only a request, not a requirement, so I can't put the files there as they need to be truly secure.

This is the page that discusses it.

http://www.robotstxt.org/orig.html

The second paragraph says "It is not an official standard backed by a standards body, or owned by any commercial organisation. It is not enforced by anybody, and there no guarantee that all current and future robots will use it. Consider it a common facility the majority of robot authors offer the WWW community to protect WWW server against unwanted accesses by their robots."

0
Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

 
LVL 9

Accepted Solution

by:
djpazza earned 500 total points
ID: 24433853
I think the ultimate way is to stream the file so it does not actually existing until it is request then a script will put it together.

http://www.builderau.com.au/webdev/soa/Protect-your-site-against-Web-scrapes/0,339024680,339129927,00.htm

2) Can you not give the users the password they need to access the folder.
3) You could setup a folder for ftp access with a password on.  So users would goto e.g
ftp.thedomain.com/Presentations/
4) You could put the files into a password protect zip file.

5) The easiest way might be an online storage facility.

Box.net allows you to setup shared folders, invite users to get notifications of new files etc.
0
 
LVL 23

Expert Comment

by:Tony McCreath
ID: 24439905
Maybe ASP.Net forms authentication would work for you:

http://msdn.microsoft.com/en-us/library/aa480476.aspx

You can control access to files and folders based on the user that is logged in. All at a server level.
0
 

Author Closing Comment

by:dtburdick
ID: 31583543
Box.net is the perfect solution for me.  Especially the notification piece.

However, for anyone else reading this thread I already figured out a solution.  My files were in a password protected folder, accessed from an ASPX file sitting in an iframe in a COTS app with a crazy URL (that they wanted to keep).  I sent them a direct link to the ASPX file taking the URL and iframe out of the picture and it worked.  

I'm not sure if it was the combination of craziness or one of those two that caused the problem, but it did resolve it.  Even still, I'll probably end up going with box.net as it is a more elegant solution.
0

Featured Post

Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

"In order to have an organized way for empathy mapping, we rely on a psychological model and trying to model it in a simple way, so we will split the board to three section for each persona and a scenario and try to see what those personas would Do,…
Boost your ability to deliver ambitious and competitive web apps by choosing the right JavaScript framework to best suit your project’s needs.
The viewer will learn how to dynamically set the form action using jQuery.
The viewer will learn how to count occurrences of each item in an array.

860 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question