Solved

robots.txt explained

Posted on 2012-03-12
2
463 Views
Last Modified: 2012-03-13
1) Does every website have a robots.txt file?
2) Is every websites robots.txt file publicly accessible/downloadable - if so from where?
3) What is the point in them, for example if you have an entry in robots.txt for /admin - then if the file is publicly accessible then what has it actually solved? I.e. how are you any better off in hiding /admin from google if someone can download your robots.txt and then see you actually have /admin directory on the server?

I cant see the logic or what the point in adding entries in it are?
0
Comment
Question by:pma111
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
2 Comments
 
LVL 53

Accepted Solution

by:
COBOLdinosaur earned 250 total points
ID: 37710248
No every site does not have one.  

It needs to be accessible so the 'bots can find it. The purpose of robot.txt is to tell the spiders not to index things.  Without direction they will index everything they find.  If you don't want the directory name exposed put it in a higher level folder and deny at the higher level

You prevent the public from accessing the admin or anything else with .htaccess

If you have something that is sensitive it should not be on the web server, because a hacker will always find a way to see it.


Cd&
0
 
LVL 15

Assisted Solution

by:Ess Kay
Ess Kay earned 250 total points
ID: 37710267
1> no
2> yes typically-->   website.com/robots.txt
3> entries entered here are for the webcrawling robots. it will allow/disallow to crawl through certain sections of your website and index them into the search engines such as google, yahoo, bing


if you have certain pages which you dont want to be added to the search engine such as your admin login page, you might want to add it here, so that common folk will not see it when they search for you site



as far as true hackers, the lack of a robots.txt file will not stop them



you dont have to add all of the admin pages, only the first portal

once the robots stops there it will not go further through that page's links
0

Featured Post

Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When the s#!t hits the fan, you don’t have time to look up who’s on call, draft emails, call collaborators, or send text messages. An instant chat window is definitely the way to go, especially one like HipChat. HipChat is a true business app. An…
When it comes to security, close monitoring is a must. According to WhiteHat Security annual report, a substantial number of all web applications are vulnerable always. Monitis offers a new product - fully-featured Website security monitoring and pr…
HTML5 has deprecated a few of the older ways of showing media as well as offering up a new way to create games and animations. Audio, video, and canvas are just a few of the adjustments made between XHTML and HTML5. As we learned in our last micr…
The viewer will learn the basics of jQuery, including how to invoke it on a web page. Reference your jQuery libraries: (CODE) Include your new external js/jQuery file: (CODE) Write your first lines of code to setup your site for jQuery.: (CODE)

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question