Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

robots.txt explained

Posted on 2012-03-12
2
Medium Priority
?
477 Views
Last Modified: 2012-03-13
1) Does every website have a robots.txt file?
2) Is every websites robots.txt file publicly accessible/downloadable - if so from where?
3) What is the point in them, for example if you have an entry in robots.txt for /admin - then if the file is publicly accessible then what has it actually solved? I.e. how are you any better off in hiding /admin from google if someone can download your robots.txt and then see you actually have /admin directory on the server?

I cant see the logic or what the point in adding entries in it are?
0
Comment
Question by:pma111
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
2 Comments
 
LVL 53

Accepted Solution

by:
COBOLdinosaur earned 1000 total points
ID: 37710248
No every site does not have one.  

It needs to be accessible so the 'bots can find it. The purpose of robot.txt is to tell the spiders not to index things.  Without direction they will index everything they find.  If you don't want the directory name exposed put it in a higher level folder and deny at the higher level

You prevent the public from accessing the admin or anything else with .htaccess

If you have something that is sensitive it should not be on the web server, because a hacker will always find a way to see it.


Cd&
0
 
LVL 15

Assisted Solution

by:Ess Kay
Ess Kay earned 1000 total points
ID: 37710267
1> no
2> yes typically-->   website.com/robots.txt
3> entries entered here are for the webcrawling robots. it will allow/disallow to crawl through certain sections of your website and index them into the search engines such as google, yahoo, bing


if you have certain pages which you dont want to be added to the search engine such as your admin login page, you might want to add it here, so that common folk will not see it when they search for you site



as far as true hackers, the lack of a robots.txt file will not stop them



you dont have to add all of the admin pages, only the first portal

once the robots stops there it will not go further through that page's links
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Australian government abolished Visa 457 earlier this April and this article describes how this decision might affect Australian IT scene and IT experts.
When the s#!t hits the fan, you don’t have time to look up who’s on call, draft emails, call collaborators, or send text messages. An instant chat window is definitely the way to go, especially one like HipChat. HipChat is a true business app. An…
This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details. Login to your Google Account, then search for "Google Mapmaker…
Any person in technology especially those working for big companies should at least know about the basics of web accessibility. Believe it or not there are even laws in place that require businesses to provide such means for the disabled and aging p…

636 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question