Solved

Is it possible to post a file on a website in a non-protected folder that bots can't get to?

Posted on 2009-05-20
6
239 Views
Last Modified: 2013-12-09
I have a number of PDFs in a password protected folder.  In that folder I have an HTML file with links to the files.  Unfortunately some people cannot access them due to filtering software on their end that gives the following error message:

   You are explicitly denied access to this website "http://www.thedomain.com/Presentations/Index.aspx" per local base policy and AFI-33-129

   Your request was denied because of its content categorization: "none"

I can put together a password protected page easily enough that only gives them the links to the files if they enter a correct password, but the files themselves would then be in a non-protected folder.  Will search engine bots be able to find those files without the password since they are just sitting in the folder?  

When I put in a URL for a folder on my web server with no default page I get the error
   Directory Listing Denied
   This Virtual Directory does not allow contents to be listed.

This leads me to believe that they would be safe, but I don't want to find out afterward they aren't as the data is sensitive.

Thanks.
0
Comment
Question by:dtburdick
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
6 Comments
 
LVL 8

Expert Comment

by:Pearl_export_ben
ID: 24432494
stick this into a file called robots.txt at the root of your drive:

User-agent: *
Disallow: /presentations/
0
 
LVL 8

Expert Comment

by:Pearl_export_ben
ID: 24432503
*by drive i mean website root
0
 

Author Comment

by:dtburdick
ID: 24433537
Thanks.  Unfortunately I can't use that because it seems that is only a request, not a requirement, so I can't put the files there as they need to be truly secure.

This is the page that discusses it.

http://www.robotstxt.org/orig.html

The second paragraph says "It is not an official standard backed by a standards body, or owned by any commercial organisation. It is not enforced by anybody, and there no guarantee that all current and future robots will use it. Consider it a common facility the majority of robot authors offer the WWW community to protect WWW server against unwanted accesses by their robots."

0
DevOps Toolchain Recommendations

Read this Gartner Research Note and discover how your IT organization can automate and optimize DevOps processes using a toolchain architecture.

 
LVL 9

Accepted Solution

by:
djpazza earned 500 total points
ID: 24433853
I think the ultimate way is to stream the file so it does not actually existing until it is request then a script will put it together.

http://www.builderau.com.au/webdev/soa/Protect-your-site-against-Web-scrapes/0,339024680,339129927,00.htm

2) Can you not give the users the password they need to access the folder.
3) You could setup a folder for ftp access with a password on.  So users would goto e.g
ftp.thedomain.com/Presentations/
4) You could put the files into a password protect zip file.

5) The easiest way might be an online storage facility.

Box.net allows you to setup shared folders, invite users to get notifications of new files etc.
0
 
LVL 23

Expert Comment

by:Tony McCreath
ID: 24439905
Maybe ASP.Net forms authentication would work for you:

http://msdn.microsoft.com/en-us/library/aa480476.aspx

You can control access to files and folders based on the user that is logged in. All at a server level.
0
 

Author Closing Comment

by:dtburdick
ID: 31583543
Box.net is the perfect solution for me.  Especially the notification piece.

However, for anyone else reading this thread I already figured out a solution.  My files were in a password protected folder, accessed from an ASPX file sitting in an iframe in a COTS app with a crazy URL (that they wanted to keep).  I sent them a direct link to the ASPX file taking the URL and iframe out of the picture and it worked.  

I'm not sure if it was the combination of craziness or one of those two that caused the problem, but it did resolve it.  Even still, I'll probably end up going with box.net as it is a more elegant solution.
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article will inform Clients about common and important expectations from the freelancers (Experts) who are looking at your Gig.
Color can increase conversions, create feelings of warmth or even incite people to get behind a cause. If you want your website to really impact site visitors, then it is vital to consider the impact color has on them.
The viewer will learn how to dynamically set the form action using jQuery.
The is a quite short video tutorial. In this video, I'm going to show you how to create self-host WordPress blog with free hosting service.

734 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question