Solved

robots.txt file

Posted on 2006-11-28
2
591 Views
Last Modified: 2011-10-03
Is it a good idea to include a robots.txt file within an ASP.NET web application?

I created the following one for a site I'm working on:

# robots.txt file

User-agent: *
Disallow: /App_Browsers/            # Contains browser specific declarations
Disallow: /App_Code/                  # Contains uncompilied class files
Disallow: /App_Themes/            # Contains skins and stylesheets for each theme folder
Disallow: /Bin/                  # Contains compiled assemblies
Disallow: /CSS/                  # Contains generic global stylesheet
Disallow: /files/                  # Contains all downloadable and demo files
Disallow: /images/                  # Contains all the sitewide images
Disallow: /Javascript/                  # Contains javascript files
Disallow: /Templates/                  # Contains the master pages
Disallow: /UserControls/            # Contains the user controls
Disallow: /Breadcrumb.sitemap            # breadcrumb xml file
Disallow: /earlynews.aspx            # protected early press release page - login required
Disallow: /employmentdetails.aspx               # page displays career opportunity details
Disallow: /odyssey.aspx            # page redirects to external odyssey domain
Disallow: /popup.aspx                  # default popup page used to display flash demos
Disallow: /TopNavigationBar.sitemap      # top navigation xml file
Disallow: /Web.Config                  # web configuration file

these were the directories and files I didn't deem necessary for spiders to crawl.

This is the first .net application I've built (previously classic ASP developer).  So, I wasn't sure the necessity or usefulness of a robots.txt file in an asp.net application.

Thanks for any input.
0
Comment
Question by:-Dman100-
  • 2
2 Comments
 
LVL 25

Accepted Solution

by:
Lee Savidge earned 500 total points
ID: 18029567
Hi,

It is a good idea to try and limit where robots can go because they would be confused by the code the the ASP.NET engine creates. The HTML produced by ASP.NET can be more difficult to parse than that created using classic ASP. This is particularly so if the content the robots are interested in can only be displayed after a form is posted back. The .NET Framework gives form fields names such as _ctl10__ctl1_DropDownListPrice which can often be inconsistent if the page contains different numbers of controls each time it is viewed or it contains controls with many subcontrols within them, such as DataGrids and GridViews.

That said, robots don't have to honour the robots.txt file because the protocol isn't an industry standard. You might want to invest some time in using the meta tag to prevent access by certain robots. See http://www.robotstxt.org/wc/meta-user.html.

Regards,

Lee
0
 
LVL 25

Expert Comment

by:Lee Savidge
ID: 18029571
Hi,

This site might help you as well: http://www.seochat.com/seo-tools/robots-generator/

Regards,

Lee
0

Featured Post

ScreenConnect 6.0 Free Trial

Discover new time-saving features in one game-changing release, ScreenConnect 6.0, based on partner feedback. New features include a redesigned UI, app configurations and chat acknowledgement to improve customer engagement!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

User art_snob (http://www.experts-exchange.com/M_6114203.html) encountered strange behavior of Android Web browser on his Mobile Web site. It took a while to find the true cause. It happens so, that the Android Web browser (at least up to OS ver. 2.…
Introduction This article shows how to use the open source plupload control to upload multiple images. The images are resized on the client side before uploading and the upload is done in chunks. Background I had to provide a way for user…
Nobody understands Phishing better than an anti-spam company. That’s why we are providing Phishing Awareness Training to our customers. According to a report by Verizon, only 3% of targeted users report malicious emails to management. With compan…
Finds all prime numbers in a range requested and places them in a public primes() array. I've demostrated a template size of 30 (2 * 3 * 5) but larger templates can be built such 210  (2 * 3 * 5 * 7) or 2310  (2 * 3 * 5 * 7 * 11). The larger templa…

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question