Hide some files from robots

Posted on 2013-10-12
Medium Priority
Last Modified: 2013-10-13
How to hide .sql files on a website from robots.
Shared hosting.  I can not store files on a lower directory.

.sql file must stay in folder.

Please do not say to delete .sql file
Question by:rgb192
LVL 49

Accepted Solution

PortletPaul earned 2000 total points
ID: 39568897
What facilities do you have access to in the shared hosting (e.g. cPanel)
and: Do you know what type of web server is being used? (e.g. Apache)

if Apache; a .htaccess file could be to deny access like this:

<FilesMatch "\.(htaccess|htpasswd|sql|ini|log|sh|inc|bak)$">
Order Allow,Deny
Deny from all

note: a spider (robot) should honour robots.txt but there is no guarantee they all will
LVL 84

Expert Comment

by:Dave Baldwin
ID: 39568905
Why must the .sql file stay in the folder?  No database that I know of can directly use a *.sql file.  Such files are used for import and export, not for normal operations.

One option is to put the files in a password protected directory.  That will prevent external access while allowing access to files on the web server.

Expert Comment

ID: 39569093
Add one line to Robots.txt file ,

User-agent: *
Disallow: /Foldername/

eg:- Your folder name is "junk". then edit  to.

User-agent: *
Disallow: /junk/
LVL 111

Expert Comment

by:Ray Paseur
ID: 39569219
If you do not have any anchor tags (links) to the files, you will have hidden them effectively from the robots.  Robots can only find the files that are linked; they cannot make up names and guess.

Author Closing Comment

ID: 39570044
shared hosting has an .htaccess file

I like this answer the best

Featured Post

Never miss a deadline with monday.com

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Since pre-biblical times, humans have sought ways to keep secrets, and share the secrets selectively.  This article explores the ways PHP can be used to hide and encrypt information.
The title says it all. Writing any type of PHP Application or API code that provides high throughput, while under a heavy load, seems to be an arcane art form (Black Magic). This article aims to provide some general guidelines for producing this typ…
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.

597 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question