MVC 4 and Robots.txt Usage?

Posted on 2013-09-04
Medium Priority
Last Modified: 2013-09-10
I've been using a Robots.txt file for years using ASPX web sites. Just curious if its needed with MVC 4?  Also curious is Ignore Routes is useful for Robots?
Question by:WorknHardr
LVL 96

Accepted Solution

Bob Learned earned 1600 total points
ID: 39472718
Robots.txt is what it is, a file to help search engines know what to index.  IMHO it is needed for all public-facing web sites.

Ignored routes are useful to reduce the "noise" from some requests, or to avoid routing when you have special HTTP handlers in place.

On my internal web sites, I always ignore a long list of stuff, including Robots.txt requests.

Author Closing Comment

ID: 39478949
Thx again

Featured Post

The 14th Annual Expert Award Winners

The results are in! Meet the top members of our 2017 Expert Awards. Congratulations to all who qualified!

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

I recently went through the process of creating a Calendar Control of events with the basis of using a database to keep track of the dates that are selectable, one requirement was to have the selected date pop-up in a simple lightbox.  At first this…
The article shows the basic steps of integrating an HTML theme template into an ASP.NET MVC project
The video provides a quick and easy steps to migrate MBOX file to well known Outlook PST and Office 365. Besides this, it also supports and migrates more than 20 email clients of MBOX which include AppleMail, Opera, Thunderbird and SeaMonkey effortl…
Through the video, you can check the migration process of Outlook PST file to PDF. Kernel for Outlook to PDF tool can convert Outlook emails with all attributes like Subject, To, From, Cc, Bcc and other folders such as Inbox, Outbox, Sent Items, Jun…
Suggested Courses

624 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question