Solved

Is it possible to post a file on a website in a non-protected folder that bots can't get to?

Posted on 2009-05-20
6
233 Views
Last Modified: 2013-12-09
I have a number of PDFs in a password protected folder.  In that folder I have an HTML file with links to the files.  Unfortunately some people cannot access them due to filtering software on their end that gives the following error message:

   You are explicitly denied access to this website "http://www.thedomain.com/Presentations/Index.aspx" per local base policy and AFI-33-129

   Your request was denied because of its content categorization: "none"

I can put together a password protected page easily enough that only gives them the links to the files if they enter a correct password, but the files themselves would then be in a non-protected folder.  Will search engine bots be able to find those files without the password since they are just sitting in the folder?  

When I put in a URL for a folder on my web server with no default page I get the error
   Directory Listing Denied
   This Virtual Directory does not allow contents to be listed.

This leads me to believe that they would be safe, but I don't want to find out afterward they aren't as the data is sensitive.

Thanks.
0
Comment
Question by:dtburdick
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
6 Comments
 
LVL 8

Expert Comment

by:Pearl_export_ben
ID: 24432494
stick this into a file called robots.txt at the root of your drive:

User-agent: *
Disallow: /presentations/
0
 
LVL 8

Expert Comment

by:Pearl_export_ben
ID: 24432503
*by drive i mean website root
0
 

Author Comment

by:dtburdick
ID: 24433537
Thanks.  Unfortunately I can't use that because it seems that is only a request, not a requirement, so I can't put the files there as they need to be truly secure.

This is the page that discusses it.

http://www.robotstxt.org/orig.html

The second paragraph says "It is not an official standard backed by a standards body, or owned by any commercial organisation. It is not enforced by anybody, and there no guarantee that all current and future robots will use it. Consider it a common facility the majority of robot authors offer the WWW community to protect WWW server against unwanted accesses by their robots."

0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 9

Accepted Solution

by:
djpazza earned 500 total points
ID: 24433853
I think the ultimate way is to stream the file so it does not actually existing until it is request then a script will put it together.

http://www.builderau.com.au/webdev/soa/Protect-your-site-against-Web-scrapes/0,339024680,339129927,00.htm

2) Can you not give the users the password they need to access the folder.
3) You could setup a folder for ftp access with a password on.  So users would goto e.g
ftp.thedomain.com/Presentations/
4) You could put the files into a password protect zip file.

5) The easiest way might be an online storage facility.

Box.net allows you to setup shared folders, invite users to get notifications of new files etc.
0
 
LVL 23

Expert Comment

by:Tony McCreath
ID: 24439905
Maybe ASP.Net forms authentication would work for you:

http://msdn.microsoft.com/en-us/library/aa480476.aspx

You can control access to files and folders based on the user that is logged in. All at a server level.
0
 

Author Closing Comment

by:dtburdick
ID: 31583543
Box.net is the perfect solution for me.  Especially the notification piece.

However, for anyone else reading this thread I already figured out a solution.  My files were in a password protected folder, accessed from an ASPX file sitting in an iframe in a COTS app with a crazy URL (that they wanted to keep).  I sent them a direct link to the ASPX file taking the URL and iframe out of the picture and it worked.  

I'm not sure if it was the combination of craziness or one of those two that caused the problem, but it did resolve it.  Even still, I'll probably end up going with box.net as it is a more elegant solution.
0

Featured Post

Space-Age Communications Transitions to DevOps

ViaSat, a global provider of satellite and wireless communications, securely connects businesses, governments, and organizations to the Internet. Learn how ViaSat’s Network Solutions Engineer, drove the transition from a traditional network support to a DevOps-centric model.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Boost your ability to deliver ambitious and competitive web apps by choosing the right JavaScript framework to best suit your project’s needs.
Today, the web development industry is booming, and many people consider it to be their vocation. The question you may be asking yourself is – how do I become a web developer?
This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details. Login to your Google Account, then search for "Google Mapmaker…
Learn how to create flexible layouts using relative units in CSS.  New relative units added in CSS3 include vw(viewports width), vh(viewports height), vmin(minimum of viewports height and width), and vmax (maximum of viewports height and width).

696 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question