Solved

robots.txt file

Posted on 2006-11-28
2
588 Views
Last Modified: 2011-10-03
Is it a good idea to include a robots.txt file within an ASP.NET web application?

I created the following one for a site I'm working on:

# robots.txt file

User-agent: *
Disallow: /App_Browsers/            # Contains browser specific declarations
Disallow: /App_Code/                  # Contains uncompilied class files
Disallow: /App_Themes/            # Contains skins and stylesheets for each theme folder
Disallow: /Bin/                  # Contains compiled assemblies
Disallow: /CSS/                  # Contains generic global stylesheet
Disallow: /files/                  # Contains all downloadable and demo files
Disallow: /images/                  # Contains all the sitewide images
Disallow: /Javascript/                  # Contains javascript files
Disallow: /Templates/                  # Contains the master pages
Disallow: /UserControls/            # Contains the user controls
Disallow: /Breadcrumb.sitemap            # breadcrumb xml file
Disallow: /earlynews.aspx            # protected early press release page - login required
Disallow: /employmentdetails.aspx               # page displays career opportunity details
Disallow: /odyssey.aspx            # page redirects to external odyssey domain
Disallow: /popup.aspx                  # default popup page used to display flash demos
Disallow: /TopNavigationBar.sitemap      # top navigation xml file
Disallow: /Web.Config                  # web configuration file

these were the directories and files I didn't deem necessary for spiders to crawl.

This is the first .net application I've built (previously classic ASP developer).  So, I wasn't sure the necessity or usefulness of a robots.txt file in an asp.net application.

Thanks for any input.
0
Comment
Question by:-Dman100-
  • 2
2 Comments
 
LVL 25

Accepted Solution

by:
Lee Savidge earned 500 total points
ID: 18029567
Hi,

It is a good idea to try and limit where robots can go because they would be confused by the code the the ASP.NET engine creates. The HTML produced by ASP.NET can be more difficult to parse than that created using classic ASP. This is particularly so if the content the robots are interested in can only be displayed after a form is posted back. The .NET Framework gives form fields names such as _ctl10__ctl1_DropDownListPrice which can often be inconsistent if the page contains different numbers of controls each time it is viewed or it contains controls with many subcontrols within them, such as DataGrids and GridViews.

That said, robots don't have to honour the robots.txt file because the protocol isn't an industry standard. You might want to invest some time in using the meta tag to prevent access by certain robots. See http://www.robotstxt.org/wc/meta-user.html.

Regards,

Lee
0
 
LVL 25

Expert Comment

by:Lee Savidge
ID: 18029571
Hi,

This site might help you as well: http://www.seochat.com/seo-tools/robots-generator/

Regards,

Lee
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Re-position the objects 7 96
Jquey and footrable 2 34
Need a complete list of ASP.NET page load events 3 26
Help with C#, MVC, razor. 6 21
I have developed many web applications with asp & asp.net and to add and use a dropdownlist was always a very simple task, but with the new asp.net, setting the value is a bit tricky and its not similar to the old traditional method. So in this a…
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, just open a new email message. In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…
With the power of JIRA, there's an unlimited number of ways you can customize it, use it and benefit from it. With that in mind, there's bound to be things that I wasn't able to cover in this course. With this summary we'll look at some places to go…

911 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now