asked on

Sitemap contains urls which are blocked by robots.txt

Hi, It looks like our sitemap lists pages which have a canonical tag, like www dot magickitchen.com/menu/get-well/7601.html.

Is that something not allowed by Google? Will it affect search results? Because the site map generator just adds them in, and I update it every few months. It will take some extra work to list all my pages with canonical tags and remove them from the sitemap each time.

For instance, this is listed five times in different categories, the highlighted one is the canonical page.

Thanks, looking forward to your advice!

Scott Fell

Set your canonical links in your site map only and 301 redirect the rest to the canonical.

https://support.google.com/webmasters/answer/139066?hl=en

Melody Scott

ASKER

That's not practical, as these are product pages and we change content, products and categories frequently. Right now all I do is add a product to all its categories, click "canonical" next to one, and publish.

From the article you posted, it would seem it's not that big a deal to have these urls in the sitemap?

ASKER CERTIFIED SOLUTION

Scott Fell

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Melody Scott

ASKER

Thanks, Scott.

Scott Fell

Your welcome!

I am a big believer in content. Great content with bad coding will trump great coding with bad content. If the content is there and people are using it, I think the search engines will recognize that and give you a lift.

Bryr de Gray

It is okay to have all the pages included on the sitemap since it's a database of all the pages of your site. However, you need to make sure that in the header area of each of the page, it should contain the rel="canonical" and rel="alternate".