?
Solved

Is it possible to post a file on a website in a non-protected folder that bots can't get to?

Posted on 2009-05-20
6
Medium Priority
?
246 Views
Last Modified: 2013-12-09
I have a number of PDFs in a password protected folder.  In that folder I have an HTML file with links to the files.  Unfortunately some people cannot access them due to filtering software on their end that gives the following error message:

   You are explicitly denied access to this website "http://www.thedomain.com/Presentations/Index.aspx" per local base policy and AFI-33-129

   Your request was denied because of its content categorization: "none"

I can put together a password protected page easily enough that only gives them the links to the files if they enter a correct password, but the files themselves would then be in a non-protected folder.  Will search engine bots be able to find those files without the password since they are just sitting in the folder?  

When I put in a URL for a folder on my web server with no default page I get the error
   Directory Listing Denied
   This Virtual Directory does not allow contents to be listed.

This leads me to believe that they would be safe, but I don't want to find out afterward they aren't as the data is sensitive.

Thanks.
0
Comment
Question by:dtburdick
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
6 Comments
 
LVL 8

Expert Comment

by:Pearl_export_ben
ID: 24432494
stick this into a file called robots.txt at the root of your drive:

User-agent: *
Disallow: /presentations/
0
 
LVL 8

Expert Comment

by:Pearl_export_ben
ID: 24432503
*by drive i mean website root
0
 

Author Comment

by:dtburdick
ID: 24433537
Thanks.  Unfortunately I can't use that because it seems that is only a request, not a requirement, so I can't put the files there as they need to be truly secure.

This is the page that discusses it.

http://www.robotstxt.org/orig.html

The second paragraph says "It is not an official standard backed by a standards body, or owned by any commercial organisation. It is not enforced by anybody, and there no guarantee that all current and future robots will use it. Consider it a common facility the majority of robot authors offer the WWW community to protect WWW server against unwanted accesses by their robots."

0
Free Backup Tool for VMware and Hyper-V

Restore full virtual machine or individual guest files from 19 common file systems directly from the backup file. Schedule VM backups with PowerShell scripts. Set desired time, lean back and let the script to notify you via email upon completion.  

 
LVL 9

Accepted Solution

by:
djpazza earned 2000 total points
ID: 24433853
I think the ultimate way is to stream the file so it does not actually existing until it is request then a script will put it together.

http://www.builderau.com.au/webdev/soa/Protect-your-site-against-Web-scrapes/0,339024680,339129927,00.htm

2) Can you not give the users the password they need to access the folder.
3) You could setup a folder for ftp access with a password on.  So users would goto e.g
ftp.thedomain.com/Presentations/
4) You could put the files into a password protect zip file.

5) The easiest way might be an online storage facility.

Box.net allows you to setup shared folders, invite users to get notifications of new files etc.
0
 
LVL 23

Expert Comment

by:Tony McCreath
ID: 24439905
Maybe ASP.Net forms authentication would work for you:

http://msdn.microsoft.com/en-us/library/aa480476.aspx

You can control access to files and folders based on the user that is logged in. All at a server level.
0
 

Author Closing Comment

by:dtburdick
ID: 31583543
Box.net is the perfect solution for me.  Especially the notification piece.

However, for anyone else reading this thread I already figured out a solution.  My files were in a password protected folder, accessed from an ASPX file sitting in an iframe in a COTS app with a crazy URL (that they wanted to keep).  I sent them a direct link to the ASPX file taking the URL and iframe out of the picture and it worked.  

I'm not sure if it was the combination of craziness or one of those two that caused the problem, but it did resolve it.  Even still, I'll probably end up going with box.net as it is a more elegant solution.
0

Featured Post

Plesk WordPress Toolkit

Plesk's WordPress Toolkit allows server administrators, resellers and customers to manage their WordPress instances, enabling a variety of development workflows for WordPress admins of all skill levels, from beginners to pros.

See why 2/3 of Plesk servers use it.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article will inform Clients about common and important expectations from the freelancers (Experts) who are looking at your Gig.
Without even knowing it, most of us are using web applications on a daily basis.  In fact, Gmail and Yahoo email, Twitter, Facebook, and eBay are used by most of us daily—and they are web applications. We generally confuse these web applications to…
The viewer will learn how to dynamically set the form action using jQuery.
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…

719 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question