Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

robots.txt file

Posted on 2006-11-28
2
Medium Priority
?
605 Views
Last Modified: 2011-10-03
Is it a good idea to include a robots.txt file within an ASP.NET web application?

I created the following one for a site I'm working on:

# robots.txt file

User-agent: *
Disallow: /App_Browsers/            # Contains browser specific declarations
Disallow: /App_Code/                  # Contains uncompilied class files
Disallow: /App_Themes/            # Contains skins and stylesheets for each theme folder
Disallow: /Bin/                  # Contains compiled assemblies
Disallow: /CSS/                  # Contains generic global stylesheet
Disallow: /files/                  # Contains all downloadable and demo files
Disallow: /images/                  # Contains all the sitewide images
Disallow: /Javascript/                  # Contains javascript files
Disallow: /Templates/                  # Contains the master pages
Disallow: /UserControls/            # Contains the user controls
Disallow: /Breadcrumb.sitemap            # breadcrumb xml file
Disallow: /earlynews.aspx            # protected early press release page - login required
Disallow: /employmentdetails.aspx               # page displays career opportunity details
Disallow: /odyssey.aspx            # page redirects to external odyssey domain
Disallow: /popup.aspx                  # default popup page used to display flash demos
Disallow: /TopNavigationBar.sitemap      # top navigation xml file
Disallow: /Web.Config                  # web configuration file

these were the directories and files I didn't deem necessary for spiders to crawl.

This is the first .net application I've built (previously classic ASP developer).  So, I wasn't sure the necessity or usefulness of a robots.txt file in an asp.net application.

Thanks for any input.
0
Comment
Question by:-Dman100-
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
2 Comments
 
LVL 25

Accepted Solution

by:
Lee Savidge earned 2000 total points
ID: 18029567
Hi,

It is a good idea to try and limit where robots can go because they would be confused by the code the the ASP.NET engine creates. The HTML produced by ASP.NET can be more difficult to parse than that created using classic ASP. This is particularly so if the content the robots are interested in can only be displayed after a form is posted back. The .NET Framework gives form fields names such as _ctl10__ctl1_DropDownListPrice which can often be inconsistent if the page contains different numbers of controls each time it is viewed or it contains controls with many subcontrols within them, such as DataGrids and GridViews.

That said, robots don't have to honour the robots.txt file because the protocol isn't an industry standard. You might want to invest some time in using the meta tag to prevent access by certain robots. See http://www.robotstxt.org/wc/meta-user.html.

Regards,

Lee
0
 
LVL 25

Expert Comment

by:Lee Savidge
ID: 18029571
Hi,

This site might help you as well: http://www.seochat.com/seo-tools/robots-generator/

Regards,

Lee
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I recently went through the process of creating a Calendar Control of events with the basis of using a database to keep track of the dates that are selectable, one requirement was to have the selected date pop-up in a simple lightbox.  At first this…
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
This course is ideal for IT System Administrators working with VMware vSphere and its associated products in their company infrastructure. This course teaches you how to install and maintain this virtualization technology to store data, prevent vuln…
This tutorial will teach you the special effect of super speed similar to the fictional character Wally West aka "The Flash" After Shake : http://www.videocopilot.net/presets/after_shake/ All lightning effects with instructions : http://www.mediaf…
Suggested Courses

618 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question