Solved

robots.txt file

Posted on 2006-11-28
2
584 Views
Last Modified: 2011-10-03
Is it a good idea to include a robots.txt file within an ASP.NET web application?

I created the following one for a site I'm working on:

# robots.txt file

User-agent: *
Disallow: /App_Browsers/            # Contains browser specific declarations
Disallow: /App_Code/                  # Contains uncompilied class files
Disallow: /App_Themes/            # Contains skins and stylesheets for each theme folder
Disallow: /Bin/                  # Contains compiled assemblies
Disallow: /CSS/                  # Contains generic global stylesheet
Disallow: /files/                  # Contains all downloadable and demo files
Disallow: /images/                  # Contains all the sitewide images
Disallow: /Javascript/                  # Contains javascript files
Disallow: /Templates/                  # Contains the master pages
Disallow: /UserControls/            # Contains the user controls
Disallow: /Breadcrumb.sitemap            # breadcrumb xml file
Disallow: /earlynews.aspx            # protected early press release page - login required
Disallow: /employmentdetails.aspx               # page displays career opportunity details
Disallow: /odyssey.aspx            # page redirects to external odyssey domain
Disallow: /popup.aspx                  # default popup page used to display flash demos
Disallow: /TopNavigationBar.sitemap      # top navigation xml file
Disallow: /Web.Config                  # web configuration file

these were the directories and files I didn't deem necessary for spiders to crawl.

This is the first .net application I've built (previously classic ASP developer).  So, I wasn't sure the necessity or usefulness of a robots.txt file in an asp.net application.

Thanks for any input.
0
Comment
Question by:-Dman100-
  • 2
2 Comments
 
LVL 25

Accepted Solution

by:
Lee Savidge earned 500 total points
ID: 18029567
Hi,

It is a good idea to try and limit where robots can go because they would be confused by the code the the ASP.NET engine creates. The HTML produced by ASP.NET can be more difficult to parse than that created using classic ASP. This is particularly so if the content the robots are interested in can only be displayed after a form is posted back. The .NET Framework gives form fields names such as _ctl10__ctl1_DropDownListPrice which can often be inconsistent if the page contains different numbers of controls each time it is viewed or it contains controls with many subcontrols within them, such as DataGrids and GridViews.

That said, robots don't have to honour the robots.txt file because the protocol isn't an industry standard. You might want to invest some time in using the meta tag to prevent access by certain robots. See http://www.robotstxt.org/wc/meta-user.html.

Regards,

Lee
0
 
LVL 25

Expert Comment

by:Lee Savidge
ID: 18029571
Hi,

This site might help you as well: http://www.seochat.com/seo-tools/robots-generator/

Regards,

Lee
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

Just a quick little trick I learned recently.  Now that I'm using jQuery with abandon in my asp.net applications, I have grown tired of the following syntax:      (CODE) I suppose it just offends my sense of decency to put inline VBScript on a…
It was really hard time for me to get the understanding of Delegates in C#. I went through many websites and articles but I found them very clumsy. After going through those sites, I noted down the points in a easy way so here I am sharing that unde…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now