Solved

robots.txt file

Posted on 2006-11-28
2
601 Views
Last Modified: 2011-10-03
Is it a good idea to include a robots.txt file within an ASP.NET web application?

I created the following one for a site I'm working on:

# robots.txt file

User-agent: *
Disallow: /App_Browsers/            # Contains browser specific declarations
Disallow: /App_Code/                  # Contains uncompilied class files
Disallow: /App_Themes/            # Contains skins and stylesheets for each theme folder
Disallow: /Bin/                  # Contains compiled assemblies
Disallow: /CSS/                  # Contains generic global stylesheet
Disallow: /files/                  # Contains all downloadable and demo files
Disallow: /images/                  # Contains all the sitewide images
Disallow: /Javascript/                  # Contains javascript files
Disallow: /Templates/                  # Contains the master pages
Disallow: /UserControls/            # Contains the user controls
Disallow: /Breadcrumb.sitemap            # breadcrumb xml file
Disallow: /earlynews.aspx            # protected early press release page - login required
Disallow: /employmentdetails.aspx               # page displays career opportunity details
Disallow: /odyssey.aspx            # page redirects to external odyssey domain
Disallow: /popup.aspx                  # default popup page used to display flash demos
Disallow: /TopNavigationBar.sitemap      # top navigation xml file
Disallow: /Web.Config                  # web configuration file

these were the directories and files I didn't deem necessary for spiders to crawl.

This is the first .net application I've built (previously classic ASP developer).  So, I wasn't sure the necessity or usefulness of a robots.txt file in an asp.net application.

Thanks for any input.
0
Comment
Question by:-Dman100-
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
2 Comments
 
LVL 25

Accepted Solution

by:
Lee Savidge earned 500 total points
ID: 18029567
Hi,

It is a good idea to try and limit where robots can go because they would be confused by the code the the ASP.NET engine creates. The HTML produced by ASP.NET can be more difficult to parse than that created using classic ASP. This is particularly so if the content the robots are interested in can only be displayed after a form is posted back. The .NET Framework gives form fields names such as _ctl10__ctl1_DropDownListPrice which can often be inconsistent if the page contains different numbers of controls each time it is viewed or it contains controls with many subcontrols within them, such as DataGrids and GridViews.

That said, robots don't have to honour the robots.txt file because the protocol isn't an industry standard. You might want to invest some time in using the meta tag to prevent access by certain robots. See http://www.robotstxt.org/wc/meta-user.html.

Regards,

Lee
0
 
LVL 25

Expert Comment

by:Lee Savidge
ID: 18029571
Hi,

This site might help you as well: http://www.seochat.com/seo-tools/robots-generator/

Regards,

Lee
0

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this Article, I will provide a few tips in problem and solution manner. Opening an ASPX page in Visual studio 2003 is very slow. To make it fast, please do follow below steps:   Open the Solution/Project. Right click the ASPX file to b…
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
There's a multitude of different network monitoring solutions out there, and you're probably wondering what makes NetCrunch so special. It's completely agentless, but does let you create an agent, if you desire. It offers powerful scalability …
Monitoring a network: why having a policy is the best policy? Michael Kulchisky, MCSE, MCSA, MCP, VTSP, VSP, CCSP outlines the enormous benefits of having a policy-based approach when monitoring medium and large networks. Software utilized in this v…

724 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question